PianoBench: Evaluating Musical Perception and Reasoning in Language Models

Abstract

Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Sound

• arXiv preprint · 2 November 2026

  Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing

  Jaeyeon Kim, Sebastian Braun and Hannes Gamper
• arXiv preprint · 3 November 2026

  ReListen: Training-Free Evidence Localization in Hour-Long Audio with Frozen Audio Language Models

  Ming-Hao Hsu, Shujie Liu, Yuxuan Hu and 4 others
• arXiv preprint · 2 November 2026

  Deployment-constrained key estimation with a DSP-guided transformer reranker

  Pardis Noorzad
• arXiv preprint · 3 November 2026

  Hybrid acoustic and pretrained embedding feature fusion for unmanned aerial vehicle detection using stacking ensemble learning under a leakage-aware evaluation protocol

  Zahid Iqbal
• arXiv preprint · 2 November 2026

  InteractionBench: Measuring Timing, Task Accuracy, Whisper and Volume, and Grounding in Real-Time Voice Systems

  Richard Yucheng He, Chen Xu, Yihang Liu and 2 others

All new papers in Sound