arXiv preprint ·
I'll Keep an Ear Out: Teaching AudioLLMs Proactive Audio Assistance
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: \texttt{ } and \texttt{ }, capturing four states: onset detection, sustained-relevance triggering, irrelevance suppression, and de-duplication. Applied to Qwen2-Audio-7B, ISM achieves 99.6\% interrupt F1 and perfect de-duplication recall on ESC-50. On noisy Epic-Sounds kitchen audio, ISM achieves the highest interrupt F1 without domain-specific training, the only method maintaining strong onset detection without over-triggering or over-suppression. Streaming evaluation confirms real-time viability with 3.5-second average latency.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Sound
COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction
ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding
The Spoken Wikipedia Presentation Corpus
Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs
Get the few papers worth your time in Sound every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.