arXiv preprint ·
Omni2Web: Benchmarking Audiovisual Website Development
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Screen-recorded web editing requests contain weak deictic expressions such as ``this'' and ``there,'' whose referents depend on speech, cursor trajectories, page state, and edit history. Such requests require intent recovery beyond the explicit specifications assumed by many existing web-editing benchmarks. We introduce Omni2Web, a bilingual benchmark of 918 instances spanning 13,907 edit steps. It defines three complementary tracks: Direct Editing evaluates webpage editing from recordings, Instruction Recovery measures explicit intent recovery, and Instruction Utility tests whether recovered instructions can drive a fixed code executor. We evaluate 17 open- and closed-source models. The best models attain 51.17 on the Edit Fidelity Score (EFS) for Direct Editing and 49.14 on the Instruction Recovery Score (IRS); under the fixed executor, the strongest recovered instructions reach 51.08 EFS, still far below the 89.69 EFS obtained with oracle instructions. Step-level analyses show that correct grounding does not guarantee successful edits, while some Omni models recover instructions that the fixed coding model executes substantially better than their direct edits. Controlled ablations further demonstrate the value of temporally aligned audiovisual evidence, while alternative judges preserve the leader and broad ordering. Together, these findings reveal substantial headroom in multimodal intent recovery and code execution and highlight the promise of pairing Omni rewriters with coding models.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks
Djinnlang: Higher-Level Programming by Unambiguous Specification with an LLM in the Compiler
One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents
A Carbon-Aware Quantum Computing Framework for LCA-Driven Sustainability in Quantum Cloud Services
Fault-Class-Matched Test Oracles for Output-Invisible Quantum Transpiler Regressions
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.