Weird Generalization is Weirdly Brittle

• URL: http://arxiv.org/abs/2604.10022v1
• Date: Sat, 11 Apr 2026 04:28:52 GMT
• Title: Weird Generalization is Weirdly Brittle
• Abstract summary: Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain develop surprising traits that manifest even outside that domain.<n>We show that weird generalization is exceptionally brittle and vanishes under simple training-time, prompt-based interventions.<n>Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.
• Score: 47.3491668391166
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)-a phenomenon that prior work has highlighted as a critical safety concern. Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior. However, we show that even very generic interventions that do not anticipate specific generalized traits can still be effective in mitigating weird generalization's effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.

Related papers

• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• Feature Dynamics as Implicit Data Augmentation: A Depth-Decomposed View on Deep Neural Network Generalization [18.72807692009739]
  We show that temporal consistency extends to unseen and corrupted data, but collapses when semantic structure is destroyed.<n>Together, these findings suggest a conceptual perspective that links feature dynamics to generalization.
  arXiv  Detail & Related papers  (2025-09-24T17:23:56Z)
• Prediction-powered Generalization of Causal Inferences [6.43357871718189]
  We show how the limited size of trials makes generalization a statistically infeasible task. We develop generalization algorithms that supplement the trial data with a prediction model learned from an additional observational study.
  arXiv  Detail & Related papers  (2024-06-05T02:44:14Z)
• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• Generalization in Kernel Regression Under Realistic Assumptions [41.345620270267446]
  We provide rigorous bounds for common kernels and for any amount of regularization, noise, any input dimension, and any number of samples. Our results imply benign overfitting in high input dimensions, nearly tempered overfitting in fixed dimensions, and explicit convergence rates for regularized regression. As a by-product, we obtain time-dependent bounds for neural networks trained in the kernel regime.
  arXiv  Detail & Related papers  (2023-12-26T10:55:20Z)
• Generalization Gap in Amortized Inference [17.951010274427187]
  We study the generalizations of a popular class of probabilistic models - the Variational Auto-Encoder (VAE) We show that the over-fitting phenomenon is usually dominated by the amortized inference network. We propose a new training objective, inspired by the classic wake-sleep algorithm, to improve the generalizations properties of amortized inference.
  arXiv  Detail & Related papers  (2022-05-23T21:28:47Z)
• Interpolation can hurt robust generalization even when there is no noise [76.3492338989419]
  We show that avoiding generalization through ridge regularization can significantly improve generalization even in the absence of noise. We prove this phenomenon for the robust risk of both linear regression and classification and hence provide the first theoretical result on robust overfitting.
  arXiv  Detail & Related papers  (2021-08-05T23:04:15Z)
• An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization [53.592597682854944]
  We recast generalization over sub-groups as an online game between a player minimizing risk and an adversary presenting new test. We show that ERM is provably minimax-optimal for both tasks.
  arXiv  Detail & Related papers  (2021-02-25T19:06:48Z)
• In Search of Robust Measures of Generalization [79.75709926309703]
  We develop bounds on generalization error, optimization error, and excess risk. When evaluated empirically, most of these bounds are numerically vacuous. We argue that generalization measures should instead be evaluated within the framework of distributional robustness.
  arXiv  Detail & Related papers  (2020-10-22T17:54:25Z)
• Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study [43.586269524826854]
  implicit regularization refers to the tendency of the optimization generalization towards a certain structured solution that often generalizes well. We provide a simple construction that rules out the existence of a emphdistribution-independent implicit regularizer that governs the generalization ability of Gradient Descent (SGD) We then demonstrate a learning problem that rules out a very general class of emphdistribution-dependent implicit regularizers from explaining generalization, which includes strongly convex regularizers as well as non-degenerate norm-based regularizations.
  arXiv  Detail & Related papers  (2020-03-13T08:40:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.