arXiv preprint ·
Robust Reward Machine Learning and Execution in Noisy Environments
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1613/jair.1.22539
- Original
- Read it at arXiv
- Topics
In brief
Reward Machines That Handle Noisy Signals
The authors introduce Prob-IRM, which learns reward machines from noisy proposition histories and uses them to guide policy learning. They evaluate it in both discrete and continuous environments, finding performance comparable to handcrafted reward machines and better than recurrent-network history encoders.
What they found: The authors evaluated Prob-IRM in environments with discrete and continuous state spaces and varied the amount of labelling noise. It performed comparably to baselines that used handcrafted reward machines. It outperformed approaches that encoded history with recurrent neural networks, even though Prob-IRM also had to contend with noisy labels.
AI-written summary. Check anything you rely on against the original paper.
Abstract
Reward machines (RMs) represent non-Markovian reward functions as finite-state machines over high-level propositional events. Traditionally, these events are generated by a deterministic labelling function that maps environment states to proposition sets. However, real-world labelling functions are inherently noisy. Objectives: Design and implement a method that learns RMs in the presence of noisy labelling functions and exploits them for policy learning in both discrete and continuous environments. Methods: We propose Prob-IRM, a method that learns RMs from noisy proposition histories using inductive logic programming (ILP) and exploits them for policy learning. Noisy proposition histories are represented as training examples for ILP, which learns an RM that maximises coverage of low-uncertainty histories. RM learning is interleaved with policy optimisation. Prob-IRM uses a belief over RM states to determine when to relearn the RM and to provide additional guiding rewards. Results: Prob-IRM is evaluated on environments with discrete and continuous state spaces under varying levels of labelling noise. Prob-IRM performs comparably to baselines using handcrafted RMs and outperforms approaches that use recurrent neural networks to encode history, despite operating with noisy labels. Conclusions: Prob-IRM represents a step towards deploying reward machines in noisy, real-world environments.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation
A Self-Exploring and Self-Evolving Paradigm for Adaptive Edge AI Development
Threat Modeling Intimate Partner Abuse in Agentic Systems
Toward Executable Multi-framework Front-end Code Generation with Self-Correction
Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.