Direct Preference Density Alignment for Conversational Audio Equalization

Abstract

Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Sound

• arXiv preprint · 11 September 2026

  TokenMapper: A Step Toward Interoperable Speech Token Translation

  Tal Kozakov, Tal Rosenwein and Eliya Nachmani
• arXiv preprint · 12 September 2026

  UniqueShip: Mitigating Data Leakage in Acoustic Ship Classification Benchmark Datasets

  Connor Hashemi, Trevor Stout, Anthony Hoogs and 1 other
• arXiv preprint · 12 September 2026

  CRAF: Cross-View Residual-Aware Fusion for Deepfake Speech Detection

  Minh-Xuan Phan, Khalid Zaman, Candy Olivia Mawalim and 1 other
• arXiv preprint · 11 September 2026

  What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability

  Lucia Cascone, Valeria Fraenza, Michele Nappi and 2 others
• arXiv preprint · 12 September 2026

  DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis

  Ziyu Zhang, Tianlun Zuo, Hanzhao Li and 2 others

All new papers in Sound