Unsupervised Elicitation of Language Models

• URL: http://arxiv.org/abs/2506.10139v1
• Date: Wed, 11 Jun 2025 19:40:08 GMT
• Title: Unsupervised Elicitation of Language Models
• Abstract summary: We introduce a new unsupervised algorithm, Internal Coherence Maximization (ICM), to fine-tune pretrained language models on their own generated labels.<n>On GSM8k-verification, TruthfulQA, and Alpaca reward modeling tasks, our method matches the performance of training on golden supervision.<n>On tasks where LMs' capabilities are strongly superhuman, our method can elicit those capabilities significantly better than training on human labels.
• Score: 37.00621880299302
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To steer pretrained language models for downstream tasks, today's post-training paradigm relies on humans to specify desired behaviors. However, for models with superhuman capabilities, it is difficult or impossible to get high-quality human supervision. To address this challenge, we introduce a new unsupervised algorithm, Internal Coherence Maximization (ICM), to fine-tune pretrained language models on their own generated labels, \emph{without external supervision}. On GSM8k-verification, TruthfulQA, and Alpaca reward modeling tasks, our method matches the performance of training on golden supervision and outperforms training on crowdsourced human supervision. On tasks where LMs' capabilities are strongly superhuman, our method can elicit those capabilities significantly better than training on human labels. Finally, we show that our method can improve the training of frontier LMs: we use our method to train an unsupervised reward model and use reinforcement learning to train a Claude 3.5 Haiku-based assistant. Both the reward model and the assistant outperform their human-supervised counterparts.

Related papers

• Accelerating Robotic Reinforcement Learning with Agent Guidance [17.75786047468276]
  Reinforcement Learning (RL) offers a powerful paradigm for autonomous robots to master generalist manipulation skills through trial-and-error.<n>Recent Human-in-the-Loop (HIL) methods accelerate training by using human corrections, yet this approach faces a scalability barrier.<n>We present Agent-guided Policy Search (AGPS), a framework that automates the training pipeline by replacing human supervisors with a multimodal agent.
  arXiv  Detail & Related papers  (2026-02-12T14:09:32Z)
• PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards [16.15363954575401]
  Current techniques for post-training Large Language Models rely on costly human supervision or on external verifiers.<n>We propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence.<n>We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance.
  arXiv  Detail & Related papers  (2026-01-08T08:09:29Z)
• Consistent Zero-Shot Imitation with Contrastive Goal Inference [30.726311787096435]
  A prerequisite for embodied agents deployed in real world interactions ought to be training with interaction.<n>Key contribution of our paper is a method for pre-training interactive agents in a self-supervised fashion.
  arXiv  Detail & Related papers  (2025-10-20T00:28:03Z)
• Training LLM Agents to Empower Humans [67.80021254324294]
  We propose a new approach to tuning assistive language models based on maximizing the human's empowerment.<n>Our empowerment-maximizing method, Empower, only requires offline text data.<n>We show that agents trained with Empower increase the success rate of a simulated human programmer on challenging coding questions by an average of 192%.
  arXiv  Detail & Related papers  (2025-10-15T16:09:33Z)
• Can Language Models Learn to Skip Steps? [59.84848399905409]
  We study the ability to skip steps in reasoning. Unlike humans, who may skip steps to enhance efficiency or to reduce cognitive load, models do not possess such motivations. Our work presents the first exploration into human-like step-skipping ability.
  arXiv  Detail & Related papers  (2024-11-04T07:10:24Z)
• Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision [98.97575836717931]
  Current AI alignment methodologies rely on human-provided demonstrations or judgments.<n>This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans?
  arXiv  Detail & Related papers  (2024-03-14T15:12:38Z)
• Self-Rewarding Language Models [100.63658475137287]
  We study Self-Rewarding Language Models, where the language model itself is used via LLM-as-a-Judge prompting to provide its own rewards during training.<n>We show that during Iterative DPO training that not only does instruction following ability improve, but also the ability to provide high-quality rewards to itself.
  arXiv  Detail & Related papers  (2024-01-18T14:43:47Z)
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision [55.196139002977525]
  Superhuman models will behave in complex ways too difficult for humans to reliably evaluate. Can weak model supervision elicit the full capabilities of a much stronger model? We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors.
  arXiv  Detail & Related papers  (2023-12-14T23:07:33Z)
• Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback [16.268581985382433]
  An important goal in artificial intelligence is to create agents that can both interact naturally with humans and learn from their feedback. Here we demonstrate how to use reinforcement learning from human feedback to improve upon simulated, embodied agents.
  arXiv  Detail & Related papers  (2022-11-21T16:00:31Z)
• Human Decision Makings on Curriculum Reinforcement Learning with Difficulty Adjustment [52.07473934146584]
  We guide the curriculum reinforcement learning results towards a preferred performance level that is neither too hard nor too easy via learning from the human decision process. Our system is highly parallelizable, making it possible for a human to train large-scale reinforcement learning applications. It shows reinforcement learning performance can successfully adjust in sync with the human desired difficulty level.
  arXiv  Detail & Related papers  (2022-08-04T23:53:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.