In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution

• URL: http://arxiv.org/abs/2602.11079v2
• Date: Fri, 13 Feb 2026 21:06:32 GMT
• Title: In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution
• Abstract summary: We propose activation-based data attribution.<n>We identify datapoints that cause specific behaviors and validate these attributions causally by retraining with modified data.<n>Applying this to OLMo 2's production DPO training, we surfaced distractor-triggered compliance.
• Score: 1.518298096221251
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose activation-based data attribution, a method that traces behavioral changes in post-trained language models to responsible training datapoints. By computing activation-difference vectors for both test prompts and preference pairs and ranking by cosine similarity, we identify datapoints that cause specific behaviors and validate these attributions causally by retraining with modified data. Clustering behavior-datapoint similarity matrices also enables unsupervised discovery of emergent behaviors. Applying this to OLMo 2's production DPO training, we surfaced distractor-triggered compliance: a harmful behavior where the model complies with dangerous requests when benign formatting instructions are appended. Filtering top-ranked datapoints reduces this behavior by 63% while switching their labels achieves 78%. Our method outperforms gradient-based attribution and LLM-judge baselines while being over 10 times cheaper than both. This in-the-wild model organism - emerging from contaminated preference data rather than deliberate injection - provides a realistic benchmark for safety techniques.

Related papers

• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations [46.963298896594814]
  Training Data Attribution seeks to trace a model's predictions back to its training data.<n>Most approaches approximate this effect in the parameter space using gradients.<n>We model the functional effect of training data in the activation space.
  arXiv  Detail & Related papers  (2026-06-03T17:59:36Z)
• Variance-Based Defense Against Blended Backdoor Attacks [0.0]
  Backdoor attacks represent a subtle yet effective class of cyberattacks targeting AI models.<n>We propose a novel defense method that trains a model on the given dataset, detects poisoned classes, and extracts the critical part of the attack trigger.
  arXiv  Detail & Related papers  (2025-06-02T09:01:35Z)
• Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs [61.04246774006429]
  We introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent.<n>We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements.<n>Our findings show that instruction-tuned models can expose pre-training data as much as their base-models, if not more so, and using instructions proposed by other LLMs can open a new avenue of automated attacks.
  arXiv  Detail & Related papers  (2024-03-05T19:32:01Z)
• Estimating Model Performance Under Covariate Shift Without Labels [0.0]
  After deployment, machine learning models often experience performance degradation due to shifts in data distribution.<n>Existing proxy methods, such as data drift detection, fail to measure the effects of these shifts adequately.<n>We introduce Probabilistic Adaptive Performance Estimation (PAPE)<n>PAPE operates independently of the original model, relying only on its predictions and probability estimates.
  arXiv  Detail & Related papers  (2024-01-16T13:29:30Z)
• Protecting Model Adaptation from Trojans in the Unlabeled Data [120.42853706967188]
  This paper explores the potential trojan attacks on model adaptation launched by well-designed poisoning target data.<n>We propose a plug-and-play method named DiffAdapt, which can be seamlessly integrated with existing adaptation algorithms.
  arXiv  Detail & Related papers  (2024-01-11T16:42:10Z)
• On the Exploitability of Instruction Tuning [103.8077787502381]
  In this work, we investigate how an adversary can exploit instruction tuning to change a model's behavior. We propose textitAutoPoison, an automated data poisoning pipeline. Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data.
  arXiv  Detail & Related papers  (2023-06-28T17:54:04Z)
• Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows [58.762959061522736]
  offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. We build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms.
  arXiv  Detail & Related papers  (2022-11-20T21:57:10Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Semi-Supervised Learning and Data Augmentation in Wearable-based Momentary Stress Detection in the Wild [14.745523471054744]
  This work investigates leveraging unlabeled wearable sensor data for stress detection in the wild. We first applied data augmentation techniques on the physiological and behavioral data to improve the robustness of supervised stress detection models. We developed a semi-supervised learning framework to leverage the unlabeled data sequences.
  arXiv  Detail & Related papers  (2022-02-22T01:10:02Z)
• Robust Variational Autoencoder for Tabular Data with Beta Divergence [0.0]
  We propose a robust variational autoencoder with mixed categorical and continuous features. Our results on the anomaly detection application for network traffic datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-06-15T08:09:34Z)
• Semi-Supervised Models via Data Augmentationfor Classifying Interactive Affective Responses [85.04362095899656]
  We present semi-supervised models with data augmentation (SMDA), a semi-supervised text classification system to classify interactive affective responses. For labeled sentences, we performed data augmentations to uniform the label distributions and computed supervised loss during training process. For unlabeled sentences, we explored self-training by regarding low-entropy predictions over unlabeled sentences as pseudo labels.
  arXiv  Detail & Related papers  (2020-04-23T05:02:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.