arXiv preprint ·
BoardSafety: AI-Ready Data from Manufacturer Documentation for Development-Board Safety
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
The development of large language models (LLMs) are capable to assist with hardware experiments. In this scenario, this raises a practical question: Can LLMs determine whether a proposed operation is safe for a specific development board? Documented operating limits and safety requirements are board-specific, even across products from the same manufacturer. This variation makes board-specific safety judgments by an LLM difficult to evaluate. To address this problem, we introduce the BoardSafety Benchmark. The benchmark is built from vendor datasheets and hardware guides. For data transformation, these documents are converted into a board-specific safety rule base and are then used to create labeled operating scenarios. Each data item pairs a board and a proposed operation to an applicable safety rule with ground-truth label, indicating the level of safety. The categorized level for each operation includes safe, unsafe, or insufficient-info. In addition, the benchmark covers diverse operations, board-specific cases, boundary conditions, and missing-information cases. We evaluate a series of models by varying their access to board-specific rules and the use of structured-rationale prompting. Our results demonstrate that providing the safety rules generally improves accuracy, whereas structured-rationale prompting produces gains that vary across models and rule-access conditions. Overall, the evaluated models cannot yet reliably determine whether an operation is safe, which implies that critical exploration is required for this field.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
PMU Foundation Model Bridge: A Tool for Connecting Synchrophasor Streams to Time Series Foundation Models
How Do Small Coding Agents Use Code Graphs and Embeddings?
OneCodeReviewBench: Typed Decisions for Code Review and Search
Securing Agent-Generated Code with Its Own Development History
Bug Severity Classification with Fine-Tuned Open LLMs: A Paired Comparison with TF-IDF and CodeBERT Baselines on Eclipse Reports
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.