arXiv preprint ·
Validation and Simulation Catch Different Errors: Four Levels of Evaluation for LLM-Generated Circuits
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Simulation success is not equivalent to structural correctness for LLM-generated circuits. We define and measure four evaluation levels -- schema validity, topological validity, backend executability, and component-set agreement -- on a 150-circuit trilingual benchmark, through a deployed pipeline built on a typed circuit interchange representation. The levels are not nested. On gpt-4o-mini, 16 of 150 circuits (10.7%, 95% CI 6.7-16.6) were rejected by the topological validator but executed in ngspice with no error or warning; 12 of these contained exactly the requested components, with one terminal disconnected. Conversely, 7 circuits (4.7%) passed the validator and ngspice refused them. Ten failed both checks and 117 passed both, so each check detects a class the other misses. A minimal three-component divider shows the cost: a dangling resistor reports 5.00 V instead of 2.50 V while ngspice stays silent. A paired ablation, in which every arm is evaluated from the same model sample rather than a fresh one, separates each repair stage from sampling noise. On a stratified 45-circuit subsample, model repair raised topological validity from 40.0% to 84.4% (+20 circuits, no regressions) while moving executability by a net 6 (+7, -1), an effect this sample size does not resolve, and component agreement by 2. One circuit moved in opposite directions at two levels in a single repair step. Against a direct-netlist baseline the pipeline executed 88.7% against 47.3%, or 62.7% under an accounting that credits the baseline with every failure we cannot confidently attribute to the netlist. These results support a narrow methodological conclusion: structural validation and simulation should be reported as distinct evaluation stages for LLM-generated circuits. A circuit that runs is not necessarily structurally valid, and a structurally valid circuit is not necessarily executable.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Hardware Architecture
ScaleMPA: Rethinking Scalable RRT* Acceleration With a Grid-Native Representation
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
NPU Accelerator: Quantized Real-Time Vehicle Detection on PYNQ-Z1 Using FINN
AgenticSizing: A Large Language Model-based Multi-Agent Framework for Analog Circuit Sizing
SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture
Get the few papers worth your time in Hardware Architecture every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.