Self-CTRL: Self-Consistency Training with Reinforcement Learning

• URL: http://arxiv.org/abs/2606.18327v1
• Date: Tue, 16 Jun 2026 17:59:40 GMT
• Title: Self-CTRL: Self-Consistency Training with Reinforcement Learning
• Abstract summary: Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users.<n>This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimize for consistency between a LM's self-explanations and behavior on related inputs.
• Score: 45.9704217495657
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic reasoning task in which LMs must learn to imitate a family of biased samplers and evaluated on their ability to report the associated biases. We find that consistency training improves the correlation between self-reported and behaviorally-measured latent biases from $R^2=0.24$ to $R^2=0.64$ on a set of held-out distributions, matching the generalization of direct ground-truth supervision. Second, we study a constitutional AI domain in which LMs must describe when they will refuse or comply with user requests. Here, Self-CTRL produces rules that faithfully describe the model's behavior on held-out requests, improving the refusal predictions of a third-party auditor model from $36\%$ to $92\%$. In the other direction, behavior updates improve alignment, reducing HarmBench failure rate from $15.0\%$ to $0.5\%$ without substantially increasing refusal on harmless prompts. By aligning explanations and behavior, our work provides a general recipe for training AI models to be safer, more transparent, and more controllable.

Related papers

• Training Large Language Models for Self-Explanation Faithfulness [10.933553638236036]
  We propose a Reinforcement Learning (RL) method to directly optimize the faithfulness of self-explanations.<n>We investigate whether models can be trained to accurately detect factors that affect their decisions.<n>We show that models can be trained to implicitly identify influential factors and disclose them.
  arXiv  Detail & Related papers  (2026-07-23T09:20:38Z)
• Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision [49.12474373883331]
  We show that LMs trained on fixed counterfactual explanations produce explanations more faithful to their own current behaviors than to those of their training targets.<n>This "introspective" coupling occurs when training explanations remain sufficiently correlated with current behaviors over the course of training.<n>We also show that introspective coupling tracks behavior shifts: when explanation training is provided concurrently with other post-training objectives, explanations track those shifts without requiring updated supervision.
  arXiv  Detail & Related papers  (2026-06-30T17:59:32Z)
• CLARity: Reasoning Consistency Alone Can Teach Reinforced Experts [20.606939295163752]
  Training expert LLMs in domains with scarce data is difficult, often relying on multiple-choice questions (MCQs)<n>Existing solutions to supervise reasoning, such as large-scale Process Reward Models (PRMs), are prohibitively expensive.<n>We propose CLARity, a cost-effective RL framework that enhances reasoning quality using only a small, general-purpose LLM.
  arXiv  Detail & Related papers  (2025-10-10T11:21:09Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models [56.055015597319674]
  Reinforcement learning with verifiable rewards (RLVR) is effective to improve the reasoning ability of large language models (LLMs)<n>Recent self-rewarding methods investigate a label-free alternative to unlock the reasoning capabilities of LLMs.<n>We propose textitCo-rewarding, a novel self-supervised RL framework that improves training stability by seeking complementary supervision from another views.
  arXiv  Detail & Related papers  (2025-08-01T08:09:14Z)
• Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models [3.207886496235499]
  We study the process through which reasoning models trained with reinforcement learning on verifiable rewards (RLVR) can learn to solve new problems.<n>We find that RLVR drives performance in two main ways: (1) by compressing pass@$k$ into pass@1 and (2) via "capability gain" in which models learn to solve new problems that they previously could not solve even at high $k$.
  arXiv  Detail & Related papers  (2025-06-16T19:03:06Z)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning [37.13807960501503]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training language models (LMs)<n>We decompose the learning signal into reinforcing correct responses and penalizing incorrect ones, referred to as Positive and Negative Sample Reinforcement (PSR and NSR)<n>We show that NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs.
  arXiv  Detail & Related papers  (2025-06-02T06:10:54Z)
• Can Large Reasoning Models Self-Train? [51.0277533541394]
  We use majority voting as a simple self-feedback mechanism to study whether self-training can be sustained within reinforcement learning.<n>We find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration.<n>Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking, resulting in sudden and complete performance collapse.
  arXiv  Detail & Related papers  (2025-05-27T17:16:00Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Training Language Models to Self-Correct via Reinforcement Learning [98.35197671595343]
  Self-correction has been found to be largely ineffective in modern large language models (LLMs) We develop a multi-turn online reinforcement learning approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. We find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.
  arXiv  Detail & Related papers  (2024-09-19T17:16:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.