arXiv preprint ·
MoSAT: Human Motion Generation from Spatial Audio and Textual Description
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Human motion is shaped by both external acoustic events and behavioral intent: spatial audio conveys environmental cues that elicit or guide a response, while text specifies the desired action and how it should be performed. In this paper, we study the novel task of human motion synthesis jointly conditioned on spatial audio and natural language, a problem that has been largely overlooked in previous research. To support this task, We introduce STAM, a dataset of motion sequences paired with spatial audio and detailed textual annotations whose rich vocabulary affords precise and nuanced specification of human motions. We further introduce MoSAT, a latent flow-matching framework for full-body motion generation jointly conditioned on natural-language intent and directional spatial-audio cues through hierarchical cross-attention before generating motion. Such a hierarchical design enhances temporally coherent and semantically aligned motion sequences. We also develop tri-modal evaluators for comprehensive evaluation on this novel task. Extensive experiments show that MoSAT achieves the SOTA performance by leveraging spatial audio's intrinsic motion-shaping properties alongside textual semantics, enabling precise and diverse motion in various scenarios.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Graphics
ProxyBuild: Text-Guided Structured 3D Building Generation with Mesh-Anchored Procedural Proxies
Estimating Accurate Hand Pose in Camera Space with Vision Transformer
Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction
When is a closed-form RGB->S/P ratio adequate? A hyperspectral characterization on natural scenes for mesopic display
WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Get the few papers worth your time in Graphics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.