Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation

Abstract

Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of \textbf{Dynamic Audio-Visual Navigation (D-AVN)}. Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel \textbf{Two-Stage Multi-modal Learning} framework for D-AVN. In the Physical Alignment stage, the proposed \textbf{Intensity-Calibrated Geometry Encoder (ICGE)} performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based \textbf{Cross-Modal Scene Aggregator (CMSA)} dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Sound

• arXiv preprint · 8 October 2026

  Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

  Shiao Zhu, Lianbo Liu, Sizhen Lyu and 3 others
• arXiv preprint · 9 October 2026

  AV-FlowEdit: Training-Free Prompt-Driven Audio-Visual Editing with Event-Aware Spectral Trajectory Preservation

  Shaohua Pan and Xiao Liu
• arXiv preprint · 8 October 2026

  SteerablePlex: Can We Steer Full-Duplex Models?

  Haolong Zheng, Maike Züfle, Dominik Macháček and 6 others
• arXiv preprint · 9 October 2026

  From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

  Yaroslav Getman, Tamás Grósz, Tommi Lehtonen and 1 other
• arXiv preprint · 8 October 2026

  Structural and Representational Agency in LLM-Driven Timbre Generation and Interpolation

  Joseph Cameron and Alan Blackwell

All new papers in Sound