arXiv preprint ·
SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Machine translation (MT) is increasingly utilized in multilingual crisis communication, yet conventional evaluation does not directly assess whether translated emergency messages preserve their actionability. We introduce SIREN (Situation-aware Index for Response and Emergency ActioNability), a human evaluation framework assessing whether translation choices compromise protective action, directive force, and urgency of the source text. We evaluate SIREN in two complementary settings. At the error level, we apply it to HOPE-annotated Italian--English translations produced by ChatGPT-4o and Google Translate: 31.63\% and 48.73\% of errors, respectively, compromise actionability, predominantly by affecting the protective action. At the message level, we evaluate English--French translations produced by TranslateGemma-4B and Google Translate, with actionability preserved in 85.34\% and 92.67\% of translations, respectively. TranslateGemma-4B failures predominantly weaken directive force, whereas Google Translate failures more often compromise protective actions and receive Major impact ratings. These findings show that translation systems can differ not only in how often actionability is lost, but also in the nature and functional impact of these failures. Alongside SIREN, we contribute an English--French crisis-translation resource with human actionability annotations and post-edited reference translations, supporting further research on function-oriented evaluation for crisis communication. Our experimental data is available at https://github.com/mcstaiano/SIREN.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion
Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic
Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals
Many Steps into One: Reasoning Contraction and the Limits of Safe Memory Reuse for LLM Agents
Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.