arXiv preprint ·
The Testability Gap: A Pre-Registered Census of Naming versus Implementation in 59 Cognitive-Architecture LLM Systems
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
A companion line of work audited individual cognitive-architecture LLM systems in depth and found that module names and module computations are empirically independent claims. This paper asks how far that gap extends across the system class. We pre-registered a census protocol (sample frame, metrics, and acceptance criteria frozen before any sample was downloaded) and surveyed the 59 most-starred Python repositories declaring cognitive architecture on GitHub, plus five classic architectures as a historical control. We measure a testability gap. First, an auditability gap: 22% of repositories document no code reference whatsoever. Second, an integrity gap: 42% of repositories have at least one dangling reference, and within that subgroup the median fraction is 8%, the top quartile 19.6%, and the maximum 79.5%; one 148-star system documents paths of which a majority do not exist. Three of four pre-registered hypotheses were falsified: the integrity gap is not universal (58% of repositories either offer no checkable references or offer references that all resolve); claim density does not predict dangling rate (rho = -0.10); and the classical-architecture control does not significantly outperform (p = 0.059 with five controls, all at zero dangling). Exactly one repository in 59 (1.7%) ships executable falsification criteria for its own claims. A second pre-registered round classified every dangling path by git lifetime: 85.5% of dangling references (159 of 186) are ghosts - paths that never existed at any commit - against 14.5% that were once real and later removed.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
DERED: Enhancing Third-Party Library Detection in Binaries via Deceptive Reuse Mitigation
A Self-Exploring and Self-Evolving Paradigm for Adaptive Edge AI Development
AgentPRO: Coordinating Compiler Optimizations for LLM Agent Workflows
Toward Executable Multi-framework Front-end Code Generation with Self-Correction
CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.