arXiv preprint ·
Language Non-Invariance in the Ideological Evaluation of Large Language Models
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Recent multilingual studies show that political evaluations of large language models (LLMs) can vary across languages. This paper examines how robust this non-invariance is to questionnaire translation, surface wording, stochastic model variation, and instrument choice, and how much it matters for cross-model comparison. Using the Political Compass, the analysis compares the positions of five LLMs, GPT-3.5, GPT-4o, GPT-5.2, Qwen-2.5, and Mistral Large, on its economic and social axes in English, Spanish, and French, with repeated stochastic draws. Additional exercises use independent Spanish and French translations, paraphrased versions in all three languages, and an alternative ideological instrument. Measured ideological positions are not language-invariant: relative to English, between four and five of the ten pairwise model rankings reverse in Spanish or French, depending on the language and axis. The broad cross-language pattern persists under independent translations, indicating that it is not solely an artifact of the official questionnaire versions. Yet the magnitude of particular shifts, and in some cases their sign, remains sensitive to wording and instrument choice. The evidence therefore supports a measurement interpretation: ideological audits recover language- and instrument-conditioned outcomes rather than a language-free political attribute. Because a single provider serves distinct linguistic constituencies through a common product, without users observing what other constituencies receive, this lack of comparability also bears on political accountability and cross-model comparisons.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computers & Society
Do Social Influence Pressures Move LLM Agents? Evidence from Delegated Audit Judgment
GESIS Surf: An Open-Source Infrastructure for Privacy-Preserving Longitudinal Web Browsing Data Collection
Dual-Axis Policy Entailment: A Benchmark for Policy-Change Reasoning
The Ineliminable Anchor: Human Judgment as a Structural Requirement of Autonomous Artificial Intelligence Systems
ROD-Dataset: A Pedestrian-Perspective Benchmark of Urban Obstacle Classes for YOLO-Based Detection
Get the few papers worth your time in Computers & Society every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.