arXiv preprint ·
Distinguishing among reasoning levels of GPT 5.5 Pro via a grammar lens benchmark
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Does the level of test-time processing effort affect a language model's computational capability? A formal grammar-lens benchmark method, derived from the theory of computation, generates novel data at specified levels, each successively more computationally powerful than the levels below. Using this data, we tested GPT-5.5, and GPT-5.5-Pro-Medium, -High, and -XHigh, showing incrementally increased acceptance of test data: 16, 40, 43, and 52\%, respectively, illustrating the benchmark's ability to discern fine-grained differences among models' added test-time reasoning levels. Additionally, as test-time compute effort increases, model responses diverge from human responses to the same data. This increasing disagreement may turn out to be broadly characteristic of more capable language models, introducing the potential of a general capability-versus-alignment conflict.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs
XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages
Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models
Information-Theoretic Design of Sparse Autoencoder-Based Steering
Robustness Starts with the Objective: Designing and Certifying Robust LLM-Generated Heuristics
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.