Demonstrating Generalization Failures via Mixtures of Conditional Policies

• URL: http://arxiv.org/abs/2607.03478v1
• Date: Fri, 03 Jul 2026 16:43:59 GMT
• Title: Demonstrating Generalization Failures via Mixtures of Conditional Policies
• Abstract summary: We propose a simple and flexible way to construct language models which fail to generalize in controllable ways when subsequently trained with Reinforcement Learning (RL)<n>We observe that RL training then selects for policies that obtain the highest reward on the training distribution.<n>We also use our construction to illustrate two novel ways in which generalization may fail in future language models.
• Score: 5.640047328020258
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training of frontier language models is conducted on curated task suites, and inevitably leaves a distribution shift between training and deployment environments. This exposes developers to generalization failures, which are relatively poorly understood. To better understand such generalization failures, we believe the community should construct clean demonstrations under simplified conditions. To facilitate this, we propose a simple and flexible way to construct language models which fail to generalize in controllable ways when subsequently trained with Reinforcement Learning (RL) on a given distribution of training tasks. Our construction uses Supervised Fine-Tuning on a dataset of a mixture of transcripts corresponding to a collection of 'conditional policies', which can each independently be assigned certain behaviors on each different task distribution, to obtain a model that is then well approximated as a 'mixture of conditional policies.' We observe that RL training then selects for policies that obtain the highest reward on the training distribution. This can produce striking behaviors: in a controlled setting with two distributions containing identical questions prepended with two different 'trigger strings', RL training on either distribution actively degrades performance on the other to zero, even though the underlying task is identical. We also use our construction to illustrate two novel ways in which generalization may fail in future language models, corresponding to distribution shifts of task coverage and temporal context respectively. While our construction is deliberately simple and may not closely resemble 'natural' generalization failures, the resulting 'model organisms' are of interest for alignment stress-testing and generalization science, and can be used as existence proofs that training success and generalization can come apart in structured ways.

Related papers

• Multi-Objective Learning for Diffusion Models: A Statistical Theory under Semi-Supervised Learning [83.65494573724904]
  We develop a principled MOL framework for diffusion models with limited data.<n>We propose a two-stage training procedure that first fits lightweight specialist models from limited paired data.<n>We establish bounds generalization showing that the required number of paired samples only depends on the complexity of the specialist model classes.
  arXiv  Detail & Related papers  (2026-05-24T18:19:40Z)
• Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging [53.41119829581115]
  Generalist robot policies, trained on large and diverse datasets, have demonstrated the ability to generalize.<n>They still fall short on new tasks not covered in the training data.<n>We develop a method that preserves the generalization capabilities of the generalist policy during finetuning.
  arXiv  Detail & Related papers  (2025-12-09T08:02:11Z)
• Mind the Gap: Bridging Prior Shift in Realistic Few-Shot Crop-Type Classification [0.3823356975862005]
  Real-world agricultural distributions often suffer from severe class imbalance, typically following a long-tailed distribution.<n>We propose Dirichlet Prior Augmentation (DirPA), a novel method that simulates an unknown label distribution skew of the target domain proactively during model training.<n>Our experiments show that DirPA successfully shifts the decision boundary and stabilizes the training process by acting as a dynamic feature regularizer.
  arXiv  Detail & Related papers  (2025-11-20T10:39:25Z)
• Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning [20.1753113722028]
  We show how long-range temporal consistency via successor representations can facilitate generalization.<n>We then propose a simple yet effective representation learning objective, $textBYOL-gamma$ for GCBC.
  arXiv  Detail & Related papers  (2025-06-11T19:32:41Z)
• Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing [55.791818510796645]
  We aim to develop models that generalize well to any diverse test distribution, even if the latter deviates significantly from the training data. Various approaches like domain adaptation, domain generalization, and robust optimization attempt to address the out-of-distribution challenge. We adopt a more conservative perspective by accounting for the worst-case error across all sufficiently diverse test distributions within a known domain.
  arXiv  Detail & Related papers  (2024-10-08T12:26:48Z)
• Any-Shift Prompting for Generalization over Distributions [66.29237565901734]
  We propose any-shift prompting: a general probabilistic inference framework that considers the relationship between training and test distributions during prompt learning. Within this framework, the test prompt exploits the distribution relationships to guide the generalization of the CLIP image-language model from training to any test distribution. The network generates the tailored test prompt with both training and test information in a feedforward pass, avoiding extra training costs at test time.
  arXiv  Detail & Related papers  (2024-02-15T16:53:42Z)
• Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations [3.8980564330208662]
  We propose and analyze a model that assumes that the attribute responsible for the shift is unknown in advance.<n>We show that our algorithm improves generalization to diverse class distributions in both simulations and experiments on real-world datasets.
  arXiv  Detail & Related papers  (2023-11-30T14:14:31Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• Generalization Guarantees for Imitation Learning [6.542289202349586]
  Control policies from imitation learning can often fail to generalize to novel environments. We present rigorous generalization guarantees for imitation learning by leveraging the Probably Approximately Correct (PAC)-Bayes framework.
  arXiv  Detail & Related papers  (2020-08-05T03:04:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.