arXiv preprint ·
Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
How the Classical Baseline Shapes Machine-Learning Speed Claims in Topology Optimization
Delegating Earth science model calibration to knowledge-guided LLM agents
Constructed regional photovoltaic generation fields improve ultra-short-term distributed solar forecasting
Beyond Time Induction: Structured Conditional Neural Solvers for Parametric Stochastic-Volatility Pricing
Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.