arXiv preprint ·
Rectified Language Flow
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
How To Build A Modern Diffusion Language Model
Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation
Data Optimization for LLMs: A Survey
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.