arXiv preprint ·
Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Human-Computer Interaction
Beyond Initial Creation: Two Decades of Community-based Chart Revisions from Wikimedia Commons
XARA: A Multimodal Agentic System for eXtended Reality Experiment Generation
"The Rose Garden Is a Cemetery": Spatial Dissonance and Divergent Public Experiences of an Indigenous Augmented Reality Tour
Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows
Measuring and Reducing Unusable Replies of LLM Agents
Get the few papers worth your time in Human-Computer Interaction every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.