arXiv preprint ·
Extraction of clinical findings from mammography and breast ultrasound reports: a comparison between specialists and Artificial Intelligence
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- arXiv
- DOI
- 10.48550/arxiv.2609.31974
- Original
- Read it at arXiv
- Cited by
- 0 works, as OpenAlex counts them
- Topics
In brief
AI vs. Experts: Who Extracts Better?
This study showcases the Gemini 2.5 Flash model's impressive performance in extracting clinical findings, outperforming human experts in some areas. It highlights the potential of AI as a valuable tool in medical settings, particularly for enhancing report accuracy.
What they found: The Gemini 2.5 Flash model achieved a Macro F1 score of 0.91 and a Micro F1 score of 0.98, outperforming human extraction, which scored 0.72. Additionally, the subjective validation showed a 93.1% agreement among health researchers, indicating the model's reliability. These results suggest that AI can serve as a powerful tool to enhance the accuracy of clinical report extraction.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Breast cancer is the leading cause of cancer-related death among women in Brazil, and the time between the request and the release of mammography reports directly influences adherence to screening, making the agility in processing these reports a critical factor for early diagnosis. In this context, this study compares the performance of a Large Language Model (LLM) with manual extraction performed by a team of health researchers in identifying clinical findings from mammography and breast ultrasound reports written in Brazilian Portuguese. Named Entity Recognition (NER) was applied through Prompt Engineering using a few-shot strategy, employing the Gemini 2.5 Flash model, selected from preliminary exploratory tests with four candidate models. The Gemini 2.5 Flash model demonstrated the best performance, achieving a Macro F1 of 0.91 and a Micro F1 of 0.98. The subjective validation, in which 29 exams of different formats were evaluated by health researchers using a Likert scale, yielded an agreement index of 93.1%. The model outperformed human extraction in overall Macro F1 (0.91 vs. 0.72), as in four reports the model correctly identified information that had been omitted or incorrectly recorded during manual extraction, demonstrating its potential as a complementary verification tool alongside specialists. The results confirm the hypothesis that LLMs, when instructed through Prompt Engineering, can achieve performance comparable to or superior to manual extraction by health professionals.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension
Prompt Injection Detection for Email Agents Through Attack Chain Modeling
Checking Leakage Witnesses versus Certifying Bounded Non-Leakage
TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.