arXiv preprint ·
AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Dense temporal audio-visual captioning is important for multimodal video understanding and generation. However, prior work is constrained by three main limitations: (1) the lack of a dataset combining high temporal resolution, longer videos, and bidirectional alignment between time and events; (2) the reliance of most existing RL methods on checklists; and (3) the lack of a benchmark that bidirectionally evaluates event semantic correctness and temporal accuracy. To address these challenges, we propose: (1) AVTime-50K, a high-quality 50K-scale dataset with temporally precise, semantically accurate captions and longer videos; (2) AVTime, a state-of-the-art model optimized with a novel Bidirectional Temporal-Semantic Alignment (BiTSA) reward that imposes explicit bidirectional constraints between time and events; and (3) AVTime-Bench, a specialized benchmark that introduces a novel bidirectional evaluation of temporal accuracy and semantic correctness. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avtime.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings
Test-time Adaptation for Point Cloud Denoising under Unseen Noise
Recurrent Vision Transformers for Event-Based Spacecraft Pose Estimation Under Slow Relative Motions
KPHarness: Co-Evolving Knowledge and Probing for Long-Horizon Gaming Harnesses
Where Do the Gaussians Go? Foreground Attraction for Camera-Based Weakly Supervised 3D Occupancy
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.