Rethinking Length-Based Training: Batch Composition and Loss Normalization in Speech Token Language Models

Abstract

Short-to-long training is a simple curriculum for speech models, but its gains can be difficult to interpret. In speech token language models, length-based training can change the shuffle policy, batch composition, token retention, and token weights under batch-mean loss. We disentangle these factors through matched comparisons. In the tested settings, short-to-long ordering shows no independent benefit when batch composition and token exposure are fixed. First-epoch grouping lowers perplexity for Mimi under batch-mean loss, but this gain is not observed under token-balanced loss. The cross-tokenizer results are consistent with a link between chunk-length variation and token weighting. This work provides a systematic analysis protocol for studying length-based training in variable-length speech models.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Computation & Language / NLP

• arXiv preprint · 22 September 2026

  Matryoshka attribution: Learning to attribute language model outputs to representations and weights

  Aryaman Arora, Kirill Acharya, Nathan Hu and 4 others
• arXiv preprint · 23 September 2026

  Realize What Matters: Principled Context Representation for Large-Scale Reasoning

  Michael Theologitis, Dean Light, Shuyue Stella Li and 3 others
• arXiv preprint · 22 September 2026

  Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

  Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia and 2 others
• arXiv preprint · 23 September 2026

  Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure

  Divyansh Singh
• arXiv preprint · 22 September 2026

  Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs

  Shubham Santosh Pandere, Gautam Ranka, Ritika Varshney and 3 others

All new papers in Computation & Language / NLP