arXiv preprint ·
Active Learning for Finding a Needle in a Haystack
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus. Active learning reduces this cost, but it must start from an initial labelled set that contains examples of the target class. Random sampling cannot supply these at reasonable cost, so they must be selected in some other way, such as keyword search, and any such selection biases the initial set towards part of the target class. We evaluate whether active learning can correct this bias. On AG News and Measuring Hate Speech, at a prevalence of around 1%, we compare random sampling with eight active learning strategies, starting from initial sets that cover or omit known subgroups of the target class, and measure recall on the omitted subgroups as well as F1. On AG News, almost every strategy recovers the omitted subgroup. On Measuring Hate Speech, uncertainty-based strategies recover less of the omitted subgroups than random sampling, whereas discriminative active learning recovers 55% of them without a significant loss of F1. Aggregate metrics conceal these differences.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Scaling Laws for Multimodal Data Mixtures
Sequent-Prover: Training Agents for Formal, Checkable SMT-based Reasoning
MemForm: Benchmarking Memory Formation Poisoning in LLM Agents
Do correct answers have stable latent knowledge support?
Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.