OpenAlex ·
Puppeteer: Object-Grounded Posture-Aware Co-Speech Gesture Generation
- Published
- Source
- OpenAlex
- DOI
- 10.48550/arxiv.2609.00369
- Original
- Read it at the publisher
- Topics
Abstract
Generating co-speech gestures that are temporally coherent, semantically aligned with speech, and grounded with surrounding objects remains challenging. Prior speech-driven gesture models emphasize audio-gesture alignment but do not explicitly account for posture constraints or surrounding objects, failing to capture the inherent correlation between body gestures and the physical space. We present Puppeteer, a posture-aware, object-grounded co-speech gesture diffusion model operating in a causal latent space. We decompose long gestures into structured primitives and learn a causal variational autoencoder that encodes them into temporally ordered latent tokens, each depending only on the past. We then perform conditional diffusion directly in the causal latent space, conditioning on speech signals, motion history, an initial posture reference, and object geometry to synthesize physically consistent gestures. This temporally ordered latent formulation enables explicit temporal control and supports tasks such as gesture in-betweening and gesture completion. To better assess co-speech gesture synthesis beyond existing measures, we introduce new evaluation metrics tailored to this task. We also created SceneGes, the first curated synthetic 3D dataset of embodied co-speech gestures and corresponding 3D objects, enabling object-grounded gesture generation. Experiments show that Puppeteer generates more diverse and temporally synchronized gestures than prior methods, while enabling object-grounded gesture synthesis.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Human-Computer Interaction
Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling
Projection-based low-rank assembly in IgA
Compressed Single-Tone Frequency Estimation With Unknown Complex Gain: Singular Rates and Global Identifiability
EvoGS: Modeling Deformation Evolution for Dynamic Gaussian Splatting
Intercoupling of Segregation and Rheology in Spatially Developing Granular Chute Flows
Get the few papers worth your time in Human-Computer Interaction every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.