Tandem Training for Language Models

• URL: http://arxiv.org/abs/2510.13551v1
• Date: Wed, 15 Oct 2025 13:48:16 GMT
• Title: Tandem Training for Language Models
• Authors: Robert West, Ashton Anderson, Ece Kamar, Eric Horvitz,
• Abstract summary: We develop methods that encourage models to produce solutions that remain intelligible to weaker collaborators.<n>We introduce tandem training for language models, in which rollout tokens are intermittently and randomly sampled from a frozen weak model.<n>In the GSM8K math reasoning task, tandem training reliably teaches models to abandon jargon and adapt their language to weaker partners.
• Score: 27.21963074798485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As language models continue to rapidly improve, we can expect their actions and reasoning to become difficult or impossible for weaker agents and humans to follow, undermining interpretability and oversight. With an eye on long-term futures, we pursue methods that encourage models to produce solutions that remain intelligible to weaker collaborators. We formalize intelligibility as handoff robustness: a strong model's solution is intelligible to a weaker model if randomly handing off control to the weaker model along the solution path does not cause failure. Building on this criterion, we introduce tandem training for language models, a reinforcement learning (RL) paradigm in which rollout tokens are intermittently and randomly sampled from a frozen weak model rather than the strong model being trained. Because rollouts succeed only when the strong model's actions and reasoning process can be continued by the weak model -- when the two can co-construct a successful solution -- optimizing standard RL objectives with tandem training implicitly incentivizes both correctness and intelligibility. In the GSM8K math reasoning task, tandem training reliably teaches models to abandon jargon and adapt their language to weaker partners while keeping task accuracy high. Our results demonstrate a promising route to building AI systems that remain auditable by weaker agents, with implications for human--AI collaboration and multi-agent communication.

Related papers

• PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards [16.15363954575401]
  Current techniques for post-training Large Language Models rely on costly human supervision or on external verifiers.<n>We propose PRISM, a unified training framework that uses a Process Reward Model (PRM) to guide learning alongside model's internal confidence.<n>We show that effectively combining PRM with self-certainty can lead to both stable training and better test-time performance.
  arXiv  Detail & Related papers  (2026-01-08T08:09:29Z)
• Self-supervised Analogical Learning using Language Models [59.64260218737556]
  We propose SAL, a self-supervised analogical learning framework.<n> SAL mimics the human analogy process and trains models to explicitly transfer high-quality symbolic solutions.<n>We show that the resulting models outperform base language models on a wide range of reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-02-03T02:31:26Z)
• Debate Helps Weak-to-Strong Generalization [68.70065254564642]
  We investigate ways of improving human supervision with a strong pretrained model and then supervise the strong model with enhanced weak human supervision.<n>We find that debate can assist a weak model in extracting trustworthy information from an untrustworthy strong model.<n>Experiments on the OpenAI weak-to-strong NLP benchmarks show that the combination approach leads to better alignment.
  arXiv  Detail & Related papers  (2025-01-21T05:36:13Z)
• Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization [0.6629765271909505]
  This paper introduces a novel approach to model alignment through weak-to-strong generalization in the context of language models. Our results suggest that this facilitation-based approach not only enhances model performance but also provides insights into the nature of model alignment.
  arXiv  Detail & Related papers  (2024-09-11T15:16:25Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• Few-shot Prompting Towards Controllable Response Generation [49.479958672988566]
  We first explored the combination of prompting and reinforcement learning (RL) to steer models' generation without accessing any of the models' parameters. We apply multi-task learning to make the model learn to generalize to new tasks better. Experiment results show that our proposed method can successfully control several state-of-the-art (SOTA) dialogue models without accessing their parameters.
  arXiv  Detail & Related papers  (2022-06-08T14:48:06Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.