arXiv preprint ·
Social Bias or Social Desirability? Comparing Factorial Survey Experiments for Humans and Large Language Models
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Factorial survey experiments are a survey method from the social sciences that uses descriptions of socially complex situation (``vignettes'') to elicit judgments such as social bias. They provide two major benefits: (a), they reduce the effect of social desirability inherent in self-reported social biases, and (b), they make it possible to separate the effects of multiple sociodemographic attributes. These aspects are also desiderata for the evaluation of social bias in LLMs, but are often ignored in social bias evaluations. In our study, we evaluate social bias in humans and eleven LLMs on the basis of four different English and German factorial survey experiments. Our results show that the LLMs mirror the human biases only partially and sometimes even invert them. We provide evidence that this is due to a fundamental difference in how humans and LLMs react to socially complex situations. While such vignettes reduce social desirability bias in humans, this is not the case for LLMs, where the alignment-induced tendency towards socially desirable responses carries over directly. We therefore caution against applying bias measurement techniques from the social sciences that were developed for human to measure attitudes and bias in LLMs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computers & Society
A Unified Response Framework for Questions and Tutoring Dialogues
Workplace Ambiguity and Technology Support for Autistic and IDD Workers
Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs
What do AI chatbots do after users decline therapeutic redirection for suicide thoughts?
HACHIMI-II: From Role-Play to Learner Behavior Modeling for Faithful Student Simulation
Get the few papers worth your time in Computers & Society every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.