arXiv preprint ·
When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Pasting a legacy file into an LLM and asking it to "modernize this" is an easy shortcut. But is the result better than leaving the code alone? In this paper, we study this single-file setting with five LLMs, each asked to modernize 100 WordPress 4.0 PHP files to PHP 8.3 while preserving behavior, with no project context or execution feedback. We evaluate every output with 630 hidden behavioral checks and 511 Rector-detected modernization findings. We compare against two baselines: the unchanged originals and the rule-based tool Rector. Often, the answer is no. Untouched, 83 files already pass every check; the best LLM leaves only 59. Each model breaks 32-51 working files while repairing at most 10, whereas Rector breaks 7 and repairs 6. Most of this damage is unnecessary. Only 37 findings (7 percent) require a change under PHP 8.3, yet in compatible files with no required change, the LLMs break 36-60 percent (Rector: 8 percent). The model with the highest modernization score is no more compatible than its closest competitors. Most regressed checks come from files that no longer load at all, often because of type declarations PHP itself rejects. These are errors that need no project context to avoid. Evaluations of LLM-based modernization should therefore measure against a no-change baseline and report modernization and behavioral preservation separately.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Do Open-Source Video Ad Clients Follow VAST 4.3? An Empirical Study of Player Conformance
AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
PyReprism: A Shared Lexical Foundation for Polyglot Code Analysis
RepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and Verification
Omission-Safe Conditional Contracts for Evidence-Closed CI Qualification
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.