Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic

Abstract

Lexical representations can encode both abstract information and information specific to particular contexts. We investigate how these forms of information are organized in language models by decomposing contextual word representations into item, class, and word-class interaction components. Across six pretrained transformers and two linguistic categories, we find that the interaction is consistently present, grows across layers, and contains information associated with both item and class. Unlike the nearly orthogonal item and class effects, the interaction overlaps with both and encodes class information beyond the shared class effect. An interaction also emerges in artificial-language learners even when the input contains only additive structure. Together, these findings show that lexical representations combine abstract information with structure specific to particular word-class combinations.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Computation & Language / NLP

• arXiv preprint · 14 October 2026

  TopicMerge: A Divide and Conquer Approach to Topic Modeling

  Matthew Raffel, Lizhong Chen, Sai Munikoti and 1 other
• arXiv preprint · 13 October 2026

  Many Steps into One: Reasoning Contraction and the Limits of Safe Memory Reuse for LLM Agents

  Jianghong Zhou, Hao Rong, Chen Lin and 1 other
• arXiv preprint · 14 October 2026

  LLM-jp-4 Technical Report

  LLM-jp, :, Chaoran Liu and 16 others
• arXiv preprint · 13 October 2026

  Reference Wording Shapes Retrieval of Robot Task Instructions

  Jahnavi Yelamanchi
• arXiv preprint · 14 October 2026

  Counterfactual Gate-Support Decomposition: Separating Selection from Redistribution in Training-Data Filtering

  Hyojung Han, Jongmin Kim and Seung-Hun Jeon

All new papers in Computation & Language / NLP