arXiv preprint ·
Program Bindings Inside Code Models
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Code language models complete, review, and repair programs without exposing the program structure encoded in their hidden states. In this paper, we test whether the hidden states of a code model can be described and edited using the information a compiler front end computes. For each token of a Python program, standard analyzers provide its position in the abstract syntax tree, its scope, its definition-use links, and its syntactic category. We fit a decomposition that expresses every hidden state of a frozen model as a sum of one term per token, each combining the token with this structural label, and we replace the model's own states with the decomposition at every layer while it predicts the output of a small program. If the model still answers correctly, the decomposition preserves the information the model uses. We apply this test to 14 open models, seven of them trained for code, and compare against a baseline decomposition of the same size built from token positions. The static-analysis decomposition preserves the model's answers better than the positional one in 10 of the 14 models, including six of the seven code models, and deleting the definition-use links from it removes most of that advantage on the two models tested. Editing the decomposition, for instance rebinding the argument of a call, makes StarCoder2-15B produce the answer of the edited program in 90 of 100 cases, compared with 19 for the positional baseline. The Qwen2.5 family is the exception on several measures. For most of the models tested, standard static analysis therefore provides both a description of what the hidden states encode about a program and a way to change it without retraining.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Do Open-Source Video Ad Clients Follow VAST 4.3? An Empirical Study of Player Conformance
AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
PyReprism: A Shared Lexical Foundation for Polyglot Code Analysis
RepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and Verification
Omission-Safe Conditional Contracts for Evidence-Closed CI Qualification
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.