arXiv preprint ·
NepTTS-Bench: Native-Speaker Benchmarking and NepaliMOS for Nepali Text-to-Speech
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Nepali TTS has reached a point where multiple systems are available, but there is no shared evaluation to compare them. NepTTS-Bench was created to address what existing automated and manual evaluation systems lack for Nepali TTS. A controlled benchmark with phonological grounding which contains 205 sentences (77 minimal pairs targeting aspiration, retroflex/dental contrasts, nasalization, gemination, and schwa deletion) which is rated by 180 speakers. There are 6,962 ratings for 10 TTS configurations and two human references. Along with this 7 ASR systems were evaluated for round-trip CER. SCOREQ was also run as an automated baseline. The key finding is that automated evaluation like SCOREQ prefers proprietary systems like ElevenLabs but native listeners prefer Piper. This is contributed by the fact that most of the systems not properly tuned for Nepali tend to show Hindi phonological behaviour and our minimal pairs are designed to surface such contrasts. We also present NepaliMOS predictor which is fine tuned on collected ratings. NepaliMOS achieves system-level rho = 0.90 on the dense panel but drops to utterance-level rho = 0.43 under leave-one-system-out evaluation. This is statistically significant and confirms that NepaliMOS captures genuine signal but cross-provider transfer is limited. A human calibration is recommended for new systems for which we release a Python evaluation package for practitioners to run the benchmark against new systems.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Audio & Speech Processing
Spatial DJ: Spatial Audio Control for Arbitrary Loudspeaker Layouts
Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models
Learning Timeline-Preserving Speech-to-Speech Translation
Wave Physics as Representation: Tensor-Structured Neural Reconstruction for Ocean Acoustic Fields
ResMamba: A Dual-Path Convolution-Mamba Model for Music Genre Classification
Get the few papers worth your time in Audio & Speech Processing every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.