arXiv preprint ·
Astro2Method: Benchmarking LLMs on Machine Learning Method Selection in Astronomy
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Machine learning (ML) is increasingly used in astronomy, but choosing a suitable method for a scientific problem often requires expertise in both astronomy and ML. Large language models (LLMs) could help astronomers make such choices, yet this ability remains underexplored in existing benchmarks. We introduce Astro2Method, a benchmark of 500 expert-reviewed binary questions from 346 astronomy papers (2024/01-2026/06), based on clear performance differences and complete (Easy) or partial (Hard) explanations from the authors. Across five LLMs, direct prompting achieves 87.50-92.55% accuracy on Easy questions but only 58.87-64.52% on Hard. Using historical papers, neither retrieval-augmented generation nor most agent methods with additional reasoning or retrieval (including ReAct, Self-Ask, and CRITIC) bring consistent improvement. Agents required to search the web improve accuracy mainly when they find the papers the questions are based on, with no consistent improvement otherwise. Providing the source paper's pipeline description with method names masked improves Hard accuracy, matching or exceeding web search. Together, these results suggest that relevant experimental information helps distinguish ML methods, but inferring it from related studies and domain knowledge remains challenging for LLMs in astronomy.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Instrumentation & Methods for Astrophysics
B-fields In STar-forming Region Observations (BISTRO): Technical Foundations for Quantitative 450/850 $μ$m JCMT SCUBA-2/POL-2 Polarimetry
Lomb-Scargle Recovery of Periodic Binary Signals across Black Hole Mass and Cadence
B-fields In STar-forming Region Observations (BISTRO): Dual-Wavelength Constraints on Size-Dependent Grain Alignment in Star-Forming Regions
A cosmic filament catalog for COSMOS-Web Data Release 1 across 0.5 < z < 3.5
simesh: A Python Framework for Quantitative Analysis of Native AMR Data from MPI-AMRVAC
All new papers in Instrumentation & Methods for Astrophysics
Get the few papers worth your time in Instrumentation & Methods for Astrophysics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.