Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search

Abstract

Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Sound

• arXiv preprint · 8 October 2026

  Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

  Shiao Zhu, Lianbo Liu, Sizhen Lyu and 3 others
• arXiv preprint · 9 October 2026

  AV-FlowEdit: Training-Free Prompt-Driven Audio-Visual Editing with Event-Aware Spectral Trajectory Preservation

  Shaohua Pan and Xiao Liu
• arXiv preprint · 8 October 2026

  SteerablePlex: Can We Steer Full-Duplex Models?

  Haolong Zheng, Maike Züfle, Dominik Macháček and 6 others
• arXiv preprint · 9 October 2026

  From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

  Yaroslav Getman, Tamás Grósz, Tommi Lehtonen and 1 other
• arXiv preprint · 8 October 2026

  Structural and Representational Agency in LLM-Driven Timbre Generation and Interpolation

  Joseph Cameron and Alan Blackwell

All new papers in Sound