arXiv preprint ·
ISBC: Interpretable Sparse Binary Coding of Pretrained Text Embeddings
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Pretrained text embeddings encode semantic structure in high-dimensional continuous spaces whose coordinates are difficult to interpret. We introduce Interpretable Sparse Binary Coding (ISBC), which learns a small set of binary dimensions from frozen embeddings without taxonomy labels or downstream outcomes. A trainable encoder and affine decoder combine reconstruction with controls on activity, factor utilization, dependence, and activation uncertainty. Individual bits and their conjunctions select overlapping document groups; complete activation profiles provide finer contexts for conditional comparisons. We evaluate these representations across corpora, backbones, and code configurations through reconstruction, geometry, predictive probes, and document inspection. Empirical cases identify both distinctions within existing categories and shared content across their boundaries. In a mixed-domain corpus, the most frequent profiles predominantly identify source collections, while selected within-source comparisons reveal sentiment and migration contrasts. Selected associations persist in additional documents encoded without refitting. In an engagement application, bits alone improve prediction of log-transformed outcomes over a constant reference, while their combinations identify within-event content contrasts associated with engagement. The results support using binary coordinates and their combinations for empirical interpretation, with complete profiles providing a fine, partly initialization-dependent description rather than a unique discrete taxonomy.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Ensemble Classification and Cancer-Specific Gene Signatures from Quality-Aware Hybrid Gene Selection in TCGA RNA-Seq data
State-Augmented Learning for Post-Deployment Adaptation of Coexisting O-RAN Applications
Where Causal Discovery Breaks on Real Data: Measuring the Assumptions One Property at a Time
Information-Theoretic Design of Sparse Autoencoder-Based Steering
Integer-Exact Microscaling: Running MXFP4 Language Models on INT8 Matrix Units
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.