Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer

• URL: http://arxiv.org/abs/2509.23886v1
• Date: Sun, 28 Sep 2025 13:51:22 GMT
• Title: Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer
• Abstract summary: Language models can transfer hidden biases during distillation.<n>This phenomenon is called subliminal learning.<n>Subliminal learning can be expected under soft distillation, where the student is trained on the teacher's full next-token distribution.
• Score: 29.621788488063853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models can transfer hidden biases during distillation. For example, a teacher that "likes owls" can make its student "like owls" too, even when the training data consists only of lists of numbers. This surprising phenomenon is called subliminal learning. Subliminal learning can be expected under soft distillation, where the student is trained on the teacher's full next-token distribution. But the fact that this also occurs under hard distillation-where the student only sees sampled tokens-raises a deeper question: when and how does subliminal learning actually occur? We answer this question through controlled experiments and mechanistic analysis. Our results show that subliminal learning does not need (global) token entanglement or logit leakage. Instead, it comes down to a small set of divergence tokens-rare cases where teachers with different biases would predict different tokens. Masking out these tokens mostly removes the hidden bias transfer. Mechanistically, divergence tokens reveal that early layers are critical. Surprisingly, finetuning even a single such early layer is sufficient for subliminal learning. Finally, we find that subliminal learning is fragile. Even small changes, like paraphrasing prompts, are usually sufficient to suppress it.

Related papers

• Verbalizing Subliminal Learning Effects Using Text Optimization [48.21170843630861]
  Subliminal learning is a phenomenon in which a distillation dataset transmits traits from the teacher model that are not legibly encoded in the dataset itself.<n>In this work, we use text optimization to detect subliminal learning effects and describe them as legible prompts.
  arXiv  Detail & Related papers  (2026-09-15T10:03:31Z)
• Subliminal Learning is a LoRA Artifact [7.7398054555399085]
  Subliminal learning is a phenomenon where language models can transmit behavioral traits to other models through seemingly innocuous data.<n>We show that subliminal learning is a LoRA artifact. When subliminal learning occurs, transmission has an inverted U-shaped relationship with LoRA rank; it also disappears with full finetuning.
  arXiv  Detail & Related papers  (2026-05-30T18:05:35Z)
• Learning Through Noise: Why Subliminal Learning Works and When It Fails [0.7249400282852116]
  Sublimating student learning refers to the transfer of knowledge or unintended biases from a teacher to a student.<n>We show that subliminal learning is governed by auxiliary head-level noise signals.<n>Our results turn from a surprising transfer effect into a theoretically grounded mechanism with predictable limits.
  arXiv  Detail & Related papers  (2026-05-22T13:59:13Z)
• Subliminal Steering: Stronger Encoding of Hidden Signals [5.13724383217928]
  Subliminal learning describes a student language model inheriting a behavioral bias by fine-tuning on seemingly innocuous data.<n>We introduce subliminal steering, a variant of subliminal learning in which the teacher's bias is implemented not via a system prompt, but through a steering vector trained to maximize the likelihood of a set of target samples.<n>We show that subliminal steering transfers complex multi-word biases, whereas prior work focused on single-word preferences, demonstrating a large scope of subliminally transferrable signals.
  arXiv  Detail & Related papers  (2026-04-28T15:51:55Z)
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data [7.092398764321311]
  We study subliminal learning, a phenomenon where language models transmit behavioral traits via semantically unrelated data.<n>We prove a theoretical result showing that subliminal learning occurs in all neural networks under certain conditions.<n>We conclude that subliminal learning is a general phenomenon that presents an unexpected pitfall for AI development.
  arXiv  Detail & Related papers  (2025-07-20T03:51:13Z)
• Localizing Paragraph Memorization in Language Models [17.943637462569537]
  We show that while memorization is spread across multiple layers and model components, gradients of memorized paragraphs have a distinguishable spatial pattern. We also show that memorized continuations are not only harder to unlearn, but also to corrupt than non-memorized ones.
  arXiv  Detail & Related papers  (2024-03-28T21:53:24Z)
• The pitfalls of next-token prediction [18.68202374460919]
  We argue that the two often-conflated phases of next-token prediction must be treated distinctly.<n>The popular criticism that errors can compound during autoregressive inference, crucially assumes that teacher-forcing has learned an accurate next-token predictor.<n>We describe a general mechanism of how teacher-forcing can fail, and design a minimal planning task where both the Transformer and the Mamba architecture empirically fail.
  arXiv  Detail & Related papers  (2024-03-11T17:47:30Z)
• Random Teachers are Good Teachers [19.74244993871716]
  We investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation. When distilling a student into such a random teacher, we observe a strong improvement of the distilled student over its teacher in terms of probing accuracy.
  arXiv  Detail & Related papers  (2023-02-23T15:26:08Z)
• On student-teacher deviations in distillation: does it pay to disobey? [54.908344098305804]
  Knowledge distillation has been widely used to improve the test accuracy of a "student" network. Despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo the teacher in performance.
  arXiv  Detail & Related papers  (2023-01-30T14:25:02Z)
• Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model [18.857225397137686]
  We propose a new style of rationale for open-book question answering, called emphmarkup-and-mask. In the markup phase, the passage is augmented with free-text markup that enables each sentence to stand on its own outside the discourse context. We generate silver annotated data by sending a series of prompts to a frozen pretrained language model, which acts as a teacher. We fine-tune a smaller student model by training on the subset of rationales that led to correct answers.
  arXiv  Detail & Related papers  (2022-10-05T18:23:49Z)
• Less is More: Task-aware Layer-wise Distillation for Language Model Compression [68.30497162547766]
  Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones. We propose a novel Task-aware layEr-wise Distillation (TED) TED designs task-aware filters to align the hidden representations of the student and the teacher at each layer.
  arXiv  Detail & Related papers  (2022-10-04T03:36:53Z)
• Revisiting Self-Distillation [50.29938732233947]
  Self-distillation is the procedure of transferring "knowledge" from a large model (the teacher) to a more compact one (the student) Several works have anecdotally shown that a self-distilled student can outperform the teacher on held-out data. We conduct extensive experiments to show that self-distillation leads to flatter minima, thereby resulting in better generalization.
  arXiv  Detail & Related papers  (2022-06-17T00:18:51Z)
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision [117.15012005163322]
  AugSelf is an auxiliary self-supervised loss that learns the difference of augmentation parameters between two randomly augmented samples. Our intuition is that AugSelf encourages to preserve augmentation-aware information in learned representations, which could be beneficial for their transferability. AugSelf can easily be incorporated into recent state-of-the-art representation learning methods with a negligible additional training cost.
  arXiv  Detail & Related papers  (2021-11-18T10:43:50Z)
• Does Knowledge Distillation Really Work? [106.38447017262183]
  We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher.
  arXiv  Detail & Related papers  (2021-06-10T17:44:02Z)
• Distilling Double Descent [65.85258126760502]
  Distillation is the technique of training a "student" model based on examples that are labeled by a separate "teacher" model. We show, that, even when the teacher model is highly over parameterized, and provides emphhard labels, using a very large held-out unlabeled dataset can result in a model that outperforms more "traditional" approaches.
  arXiv  Detail & Related papers  (2021-02-13T02:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.