arXiv preprint ·
Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
External tools can extend the capabilities of small language models (SLMs), enabling their deployment in resource constrained environments. However, their limited capabilities make it difficult for SLMs to discover effective tool use behaviors through reinforcement learning alone, while direct teacher supervision suffers from teacher student behavioral distribution mismatch and may propagate erroneous preferences. To address these challenges, we propose Selective Teacher guided Policy Optimization (SeTPO), which coordinates student exploration with selective teacher guidance and persistent demonstration supervision. Specifically, SeTPO incorporates reverse KL derived token level teacher feedback into policy advantages on low reward trajectories, while auxiliary forward KL supervision preserves teacher demonstrated behaviors throughout training. Under an idealized bidirectional KL objective, we theoretically establish that the optimal policy achieves expected reward no lower than the teacher's while retaining a guaranteed fraction of the teacher's probability mass. Extensive experiments on BFCL v4 and API Bank demonstrate that SeTPO achieves superior overall performance over supervised fine tuning, reinforcement learning, and distillation baselines across all three student scales.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Everyone Was Right: Lexical Representations in Language Models Are Both Abstract and Holistic
TopicMerge: A Divide and Conquer Approach to Topic Modeling
Many Steps into One: Reasoning Contraction and the Limits of Safe Memory Reuse for LLM Agents
LLM-jp-4 Technical Report
Reference Wording Shapes Retrieval of Robot Task Instructions
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.