arXiv preprint ·
PyReprism: A Shared Lexical Foundation for Polyglot Code Analysis
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Empirical studies of polyglot repositories apply the same analyses to every language of a corpus, such as separating code from comments, computing metrics, normalizing identifiers, and screening for duplicated code. Assembling them from specialized tools introduces differences in language coverage, measurement conventions, and workflows. We present PYREPRISM, an open-source Python library and command-line tool that separates language-specific lexical processing from reusable analyses: each language declares its lexical surface, which yields a lossless stream of typed tokens, and every analysis is implemented once on that stream. Across 187 built-in languages, language-specific declarations have a median length of 16 lines, and no language overrides a shared analysis. Against a reference parser, comment and string classification reach F1 scores of 0.994 and 0.958 on 1,412 files in 12 languages that were never used to develop the tokenizer. In a study of a polyglot repository, one tool replaces three; with matched inputs and analysis operations, it takes 2.9s instead of 8.7s on one core. We also characterize where lexical analysis falls short, in complexity estimation and clone screening. Tool: https://github.com/unlv-evol/PyReprism Video: https://youtu.be/IXc-NP5t4GU
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Do Open-Source Video Ad Clients Follow VAST 4.3? An Empirical Study of Player Conformance
AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
RepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and Verification
Omission-Safe Conditional Contracts for Evidence-Closed CI Qualification
One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.