arXiv preprint ·
Do Knowledge and Reasoning Oriented Benchmarks Induce Stable Representation Signatures in LLMs?
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Internal signatures could support monitoring, adaptive computation, and mechanistic analysis, but benchmark prompts can be separable without any shared computational distinction. We test whether a prespecified knowledge/reasoning label defines a reusable internal signature in five contemporary 7B to 31B models. We treat the \textbf{task} as the unit of inference probes hold out complete tasks, labels are permuted across tasks, and the best layer and stream are reselected for every permutation. Raw probes recover an eight-versus-eight-subject MMLU partition in all five models (macro-F1 $0.78$--$0.81$), but prompt text predicts it nearly as well: under an identical probe configuration, the activation advantage is only $0.017$--$0.070$ macro-F1 and is not significant. At the prespecified $0.01$ level, lexical adjustment removes significance while the adjusted states still identify individual subjects, and the selected site changes across task panels and selection rules. Additionally, two released base-training trajectories show no common direction. These pooled layerwise measures recover the task label but identify no transferable knowledge/reasoning signature at the tested sensitivity, which cannot exclude moderate effects. The resulting protocol gives future internal monitors and intervention studies a sequence of tests to pass. Project page: \url{https://parsa-ra.github.io/lmbehave/}.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Shape-Aware Neural Time Fields for Rigid-Body Motion and Manipulation Planning
Power-Law Correlations Without Apparent Temperature-Tuned Criticality in Fixed-Tree Hierarchical Languages
Acquisition Function-Guided Generation for Bayesian Optimization over Discrete Spaces
Uniform convergence at optimal rate and scale
One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.