arXiv preprint ·
Structure is not mechanism: high-gain gated-FFN rows across text and genomic foundation models
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
A small number of unusually high-gain parameters can exert disproportionate effects in transformer language models, but whether analogous structures recur in genomic foundation models and whether structural geometry determines functional importance remains unknown. We analyzed high-gain rows in gated feed-forward networks across text and genomic foundation models, including a frozen 22-model causal census. Computing an associated bilinear weight operator exactly, without a diagonal approximation, we tested whether structural extremeness is a transferable mechanism. Activation-derived candidates were functionally enriched relative to random and top-norm same-layer controls, yet neither spectral concentration nor operator magnitude predicted causal effect size, and these associations vanished within the endpoint-homogeneous text-decoder subset. A within-layer sweep of 36 rows in one genomic and one text decoder resolved this into two regimes: below the detector's acceptance threshold the ratio carried no positive information about causal damage, whereas above it the ratio ordered rows strongly but did not grade severity as a dose-response. The same sweep revealed a second individually catastrophic row invisible to a one-candidate-per-model census, and non-additive damage among co-located critical rows. Case studies showed divergent causal organizations: a robust super-additive pair interaction in DNABERT-2, and in GENERator a sharply position-localized dependence in which preserving or restoring the row's beginning-of-sequence contribution rescued essentially all native-loss damage. High-gain gated-FFN rows are therefore a recurrent architectural phenotype whose structural prominence acts as an enrichment signal, not a calibrated measure of functional criticality or a specification of causal organization. Enrichment is general, but the mechanism is model-specific.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Genomics
Towards a knowledge-enhanced single-cell foundation model
Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens
Democratizing Clinical Tumor Whole Genome Sequencing: 18-hour End-to-end Analysis via Trillion-parameter Large Language Models Locally Deployed on Consumer-grade Hardware
A Transformer-Based Delta Expression Encoder for Psilocybin Transcriptional Response: Architecture, Representations, and Biological Validation
HyCoSeq: Contextual Hyperbolic Representation Learning for Genomic Sequences
Get the few papers worth your time in Genomics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.