arXiv preprint ·
Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
In brief
Standardizing AI Agents for Better Research
This paper offers a comprehensive survey of AI agents by organizing them around five key dimensions of agenticness, along with a public digital resource called the Agent Compendium. This structured approach aims to enhance reproducibility and clarity in AI agent research.
What they found: This paper successfully outlines five dimensions of agenticness that can be used to evaluate AI agents. It synthesizes over 20 existing evaluation frameworks, providing a clearer structure for future research. The introduction of the Agent Compendium further supports the accessibility of these evaluation methods, promoting reproducibility and systematic study in the field.
AI-written summary. Check anything you rely on against the original paper.
Abstract
The term agent in artificial intelligence lacks a standard definition, complicating the evaluation, comparison, and reproducibility of AI agent research. We address this ambiguity through a survey organized around five dimensions of agenticness: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. For each dimension, we examine how the underlying capability has been conceptualized across prior work and synthesize the metrics, benchmarks, and evaluation frameworks used to assess it. This review provides a structured account of the current landscape of agent evaluation, highlighting both established approaches and areas where evaluation remains limited or inconsistent. We additionally introduce the Agent Compendium, a public-facing digital resource that organizes and extends the evaluation methods identified through this review. Together, the survey and compendium provide a common structure for evaluating and comparing agent capabilities across AI systems, supporting more reproducible research, clearer communication, and more systematic study of artificial agents.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training
Hierarchical Belief Modeling for Zero-Shot Opponent Adaptation in Partially Observable Multi-Agent Navigation
What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework
But How Would AI Agents Run a Town's Economy?
The Mechanics of a Swarm: A Reproducible External Reconstruction of an Unintended Agent-Coordination Episode on a Third-Party Wiki
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.