arXiv preprint ·
ConformalGuard: False-Positive-Controlled Action Gating for LM Classifiers
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Language model classifiers are increasingly deployed for high-stakes decisions such as phishing screening, content moderation, and fraud triage, yet validation-set thresholds provide no finite-sample control of deployment-time false positives. We present ConformalGuard, a model-agnostic calibration layer that wraps a fixed scoring function with finite-sample control over the rate at which legitimate inputs trigger an intervention tier. Our contribution is empirical and operational, not a new conformal theorem: zero-shot LM API scores can be coarse enough (10--11 unique values here) to make temperature scaling and isotonic regression operationally identical to raw thresholding for low-FPR action gating. We compare these methods with conformal calibration for fine-tuned BERT, Claude Sonnet 4, and GPT-4o on public email corpora. For Claude, conformal calibration reduces FPR from 0.040 to 0.008 while recall falls from 0.938 to 0.875. For deterministic GPT-4o scores, the 1% operating point is too conservative to deploy (recall 0.564); three-run stochastic averaging raises recall to 0.732 at FPR 0.005. At 2% prevalence with BERT, conformal calibration removes 25 false-positive reviews per 10,000 emails (95% CI [10, 43]). The guarantee is legitimate-event-rate control under exchangeability; discrimination, workload effects, and robustness to shift remain empirical.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Inverse Synthesis of Shock Time Series from Shock Response Spectra Using Ensembles of Conditional Variational Autoencoders
PotentialEP: Backprop-Free Training of CIFAR-Scale Generative Models via Equilibrium Propagation
TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents
Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes
Scalable Learning of Determinantal Point Processes with Kernel-Structured Interactions
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.