arXiv preprint ·
Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.18653/v1/2025.findings-emnlp.997
- Original
- Read it at arXiv
- Topics
Abstract
Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/test. This raises concerns about whether models truly generalise or instead rely on signer-specific regularities. We conduct signer-fold cross-validation on GFSLT-VLP, GASLT, and SignCL, three leading, publicly available, gloss-free SLT models, on CSL-Daily and PHOENIX14T. Under signer-independent evaluation, performance drops sharply: on PHOENIX14T, GFSLT-VLP falls from BLEU-4 21.44 to 3.59 and ROUGE-L 42.49 to 11.89; GASLT from 15.74 to 8.26; and SignCL from 22.74 to 3.66. We also observe that in CSL-Daily many target sentences are performed by multiple signers, so common splits can place identical sentences in both training and test, inflating absolute scores by rewarding recall of recurring sentences rather than genuine generalisation. These findings indicate that signer-dependent evaluation can substantially overestimate SLT capability. We recommend: (1) adopting signer-independent protocols to ensure generalisation to unseen signers; (2) restructuring datasets to include explicit signer-independent, sentence-disjoint splits for consistent benchmarking; and (3) reporting both signer-dependent and signer-independent results together with train-test sentence overlap to improve transparency and comparability.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models
Popular Knowledge Propagates More Errors in LLM Knowledge Updating
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
Vectorizer: Vectorizing NumPy Programs with Shape-Guided Rewrite
Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruning
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.