Removing Sandbagging in LLMs by Training with Weak Supervision

• URL: http://arxiv.org/abs/2604.22082v1
• Date: Thu, 23 Apr 2026 21:26:31 GMT
• Title: Removing Sandbagging in LLMs by Training with Weak Supervision
• Abstract summary: Can training elicit a model's best work even without reliable verification?<n>We find that training with weak supervision can reliably elicit sandbagging models when supervised fine-tuning (SFT) and reinforcement learning (RL) are combined.
• Score: 2.522724840981509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI systems begin to automate complex tasks, supervision increasingly relies on weaker models or limited human oversight that cannot fully verify output quality. A model more capable than its supervisors could exploit this gap through sandbagging, producing work that appears acceptable but falls short of its true abilities. Can training elicit a model's best work even without reliable verification? We study this using model organisms trained to sandbag, testing elicitation techniques on problem-solving math, graduate-level science, and competitive coding tasks. We find that training with weak supervision can reliably elicit sandbagging models when supervised fine-tuning (SFT) and reinforcement learning (RL) are combined: SFT on weak demonstrations breaks the sandbagging behavior, enabling RL to then fully elicit performance. Neither method succeeds reliably alone-RL without SFT almost always leads to reward hacking rather than genuine improvement. Critically, this relies on training being indistinguishable from deployment; when models can distinguish between training and deployment, they can perform well during training while continuing to sandbag afterward. Our results provide initial evidence that training is a viable mitigation against sandbagging, while highlighting the importance of making training indistinguishable from deployment.

Related papers

• SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents [52.758563996979596]
  Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use.<n>We propose SENTINEL, a failure-driven reinforcement learning framework that turns rollout failures into targeted training tasks.<n>On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Passk metrics.
  arXiv  Detail & Related papers  (2026-06-11T05:06:50Z)
• Exploration Hacking: Can LLMs Learn to Resist RL Training? [10.715721930948504]
  Reinforcement learning (RL) has become essential to the post-training of large language models (LLMs)<n>In this paper we study this behavior, called exploration hacking.<n>First, we create model organisms of selective RL resistance by fine-tuning LLMs to follow specific underperformance strategies.<n>These models can successfully resist our RL-based capability elicitation in agentic biosecurity and AI R&D environments while maintaining performance on related tasks.
  arXiv  Detail & Related papers  (2026-04-30T17:58:39Z)
• RefineRL: Advancing Competitive Programming with Self-Refinement Reinforcement Learning [63.432969627395686]
  RefineRL is a novel approach designed to unleash the self-refinement capabilities of large language models for competitive programming problems.<n>Skeptical-Agent is an iterative self-refinement agent equipped with local execution tools to validate generated solutions against public test cases of CP problems.<n>A reinforcement learning solution incentivizes LLMs to self-refine with only standard RLVR data.
  arXiv  Detail & Related papers  (2026-04-01T11:54:57Z)
• PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards [16.15363954575401]
  Current techniques for post-training Large Language Models rely on costly human supervision or on external verifiers.<n>We propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence.<n>We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance.
  arXiv  Detail & Related papers  (2026-01-08T08:09:29Z)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [73.10315509190623]
  Recent reinforcement learning techniques have yielded impressive reasoning improvements in language models.<n>It remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training.<n>We develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training.
  arXiv  Detail & Related papers  (2025-12-08T18:12:10Z)
• Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions [22.83151273022573]
  counterintuitive phenomena have been reported in large language models (LLMs)<n>We identify a key factor that differentiates RL observations: whether the pretrained model already exhibits strong Model-Task Alignment.<n>Our findings show that while standard RL training remains consistently robust across settings, many of these counterintuitive results arise only when the model and task already exhibit strong model-task alignment.
  arXiv  Detail & Related papers  (2025-08-28T20:02:10Z)
• Can Large Reasoning Models Self-Train? [51.0277533541394]
  We use majority voting as a simple self-feedback mechanism to study whether self-training can be sustained within reinforcement learning.<n>We find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration.<n>Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking, resulting in sudden and complete performance collapse.
  arXiv  Detail & Related papers  (2025-05-27T17:16:00Z)
• RLSR: Reinforcement Learning from Self Reward [0.0]
  We show that large language models can effectively self-improve through self-judging without reference solutions.<n>Our experiments show that models can provide reliable reward signals without ground truth answers.<n>This work represents a significant step toward autonomous AI systems that continuously improve through self-directed learning.
  arXiv  Detail & Related papers  (2025-05-12T23:51:04Z)
• Making Pre-trained Language Models both Task-solvers and Self-calibrators [52.98858650625623]
  Pre-trained language models (PLMs) serve as backbones for various real-world systems. Previous work shows that introducing an extra calibration task can mitigate this issue. We propose a training algorithm LM-TOAST to tackle the challenges.
  arXiv  Detail & Related papers  (2023-07-21T02:51:41Z)
• Simplifying Deep Reinforcement Learning via Self-Supervision [51.2400839966489]
  Self-Supervised Reinforcement Learning (SSRL) is a simple algorithm that optimize policies with purely supervised losses. We show that SSRL is surprisingly competitive to contemporary algorithms with more stable performance and less running time.
  arXiv  Detail & Related papers  (2021-06-10T06:29:59Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.