arXiv preprint ·
PSM-WSM: Predictively Sufficient Multimodal World Social Modeling
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Learning latent states that support future evolution remains an open problem for multimodal interactive systems. Existing approaches typically optimize representations for current semantic understanding or direct prediction, without explicitly defining what information a latent state should preserve to remain predictive of future dynamics. In this work, we investigate this problem through the principle of predictive sufficiency, which defines a predictive state as a latent variable that retains future-relevant information from historical multimodal observations while reducing redundant dependence on the observed history. Based on this principle, we propose Predictively Sufficient Multimodal Social World Modeling (PSM-WM), a framework that jointly learns predictive latent states and their stochastic temporal evolution through probabilistic state inference, event-conditioned transitions, and multi-step consistency optimization. Beyond forecasting performance, we introduce state-centric evaluations to measure whether learned representations satisfy predictive state properties, including predictive closure, residual history gain, and future semantic alignment. Experiments on Hi-EF, IEMOCAP, MELD, and Ego4D show that PSM-WM improves future multimodal interaction modeling and learns latent states with stronger predictive properties and robustness. These results indicate that predictive sufficiency provides an effective principle for constructing future-oriented multimodal states and highlight the importance of state construction in multimodal world modeling.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Multimedia
VideoTrail: Interactive Video Foraging Through Hierarchical Visualization
Passing NPCR Without Diffusion: Exact Signatures, a Detection Benchmark and an Audit of Published Image Ciphers
Sonic Palette: Predicting Dynamic Color Themes from Music
Learning Martial Arts Motion Skills for Animating Opponents in VR Sparring Games
VideoACE: Benchmarking Agentic Video Understanding through High-Frame-Rate Actions
Get the few papers worth your time in Multimedia every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.