SIGMA: Self-Improving Alignment Generalization from a Model Spec

Abstract

LLM agents are increasingly capable of executing complex tasks and of recursively improving themselves on easy-to-verify objectives such as software engineering and mathematics. Since alignment is much harder to verify, this creates a growing risk of capabilities increasing without appropriate safety alignment, especially as capabilities expand to auto-research and cybersecurity. Existing approaches focus on capability self-improvement using verifiable feedback or on alignment training with supervision from stronger models or curated data, creating an external supervision bottleneck for alignment. We ask whether current models can improve their own safety alignment, and propose SIGMA, a data generation and training pipeline enabling alignment self-improvement that generalizes to out-of-distribution settings. Given only a "Model Spec" stating the model's desired behavior, SIGMA leverages a model's reasoning capabilities to strengthen its own safety reasoning. SIGMA first performs spec-guided task synthesis, using the candidate model as a task designer agent to generate diverse alignment dilemma scenarios and convert them into training tasks that stress-test its understanding of the Model Spec. Next, SIGMA conducts self-judged alignment training through supervised fine-tuning and rubric-based reinforcement learning with the model itself as the reward model. Despite training only on single-turn chat data, SIGMA improves safety alignment in multi-turn agentic environments (AgentHarm harmfulness decreases from 22.6 to 14.8; Agentic Misalignment decreases from 79.1 to 3.8), outperforms Deliberative Alignment and Constitutional AI baselines, and retains general capability. Analyses show that a Model Spec balancing harmlessness and helpfulness, test-time reasoning for safety deliberation, and high-quality rubrics from SIGMA's task designer agent are crucial for effective self-improvement.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Artificial Intelligence

• arXiv preprint · 6 October 2026

  Disentangling Models from Personas in Heterogeneous LLM Simulations

  Dani Roytburg and Daphne Ippolito
• arXiv preprint · 7 October 2026

  The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules

  Litao Hu and Yutong Tang
• arXiv preprint · 6 October 2026

  A Systematic Investigation of Bias in Large Language Models for Advertising Relevance

  Weiwei Wang, Yinchuan Xu, Jialu Gao and 2 others
• arXiv preprint · 7 October 2026

  Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

  Wanjing Han, Levi Taiji Li, Mu Zhang and 2 others
• arXiv preprint · 6 October 2026

  Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization

  Myeung Suk Oh, Zhiyao Zhang, Alvaro Velasquez and 2 others

All new papers in Artificial Intelligence