Learning Through Noise: Why Subliminal Learning Works and When It Fails

• URL: http://arxiv.org/abs/2605.23645v1
• Date: Fri, 22 May 2026 13:59:13 GMT
• Title: Learning Through Noise: Why Subliminal Learning Works and When It Fails
• Abstract summary: Sublimating student learning refers to the transfer of knowledge or unintended biases from a teacher to a student.<n>We show that subliminal learning is governed by auxiliary head-level noise signals.<n>Our results turn from a surprising transfer effect into a theoretically grounded mechanism with predictable limits.
• Score: 0.7249400282852116
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In the context of artificial neural networks, subliminal learning refers to the transfer of task-relevant knowledge or unintended biases from teacher to student models through distillation on task-unrelated input$\unicode{x2013}$output pairs. Prior explanations tie this effect to shared or closely matched teacher$\unicode{x2013}$student initialization. We show that a closely matched initialization is not necessary. Instead, subliminal learning is governed by compatible output heads. Using a controlled MNIST setting, we split outputs into an auxiliary head (for auxiliary, task-unrelated noise signals) and a class head (for classification) to demonstrate subliminal learning occurs$\unicode{x2014}$even when we randomly initialize hidden layers and remove layers, add new layers, or change the architecture (MLP-to-CNN). Compatible auxiliary heads enable transfer of a recoverable teacher signal, bringing the student's representations closer to the teacher's. When the class heads remain compatible as well, students trained only on task-unrelated noise can approach, and in favorable regimes match, teacher-level task performance. Our setting enables us to develop a theory that explains the mechanism of subliminal learning and to derive upper bounds on when subliminal learning fails. Together, our results turn subliminal learning from a surprising transfer effect into a theoretically grounded mechanism with predictable limits.

Related papers

• Verbalizing Subliminal Learning Effects Using Text Optimization [48.21170843630861]
  Subliminal learning is a phenomenon in which a distillation dataset transmits traits from the teacher model that are not legibly encoded in the dataset itself.<n>In this work, we use text optimization to detect subliminal learning effects and describe them as legible prompts.
  arXiv  Detail & Related papers  (2026-09-15T10:03:31Z)
• Subliminal Learning is a LoRA Artifact [7.7398054555399085]
  Subliminal learning is a phenomenon where language models can transmit behavioral traits to other models through seemingly innocuous data.<n>We show that subliminal learning is a LoRA artifact. When subliminal learning occurs, transmission has an inverted U-shaped relationship with LoRA rank; it also disappears with full finetuning.
  arXiv  Detail & Related papers  (2026-05-30T18:05:35Z)
• Subliminal Steering: Stronger Encoding of Hidden Signals [5.13724383217928]
  Subliminal learning describes a student language model inheriting a behavioral bias by fine-tuning on seemingly innocuous data.<n>We introduce subliminal steering, a variant of subliminal learning in which the teacher's bias is implemented not via a system prompt, but through a steering vector trained to maximize the likelihood of a set of target samples.<n>We show that subliminal steering transfers complex multi-word biases, whereas prior work focused on single-word preferences, demonstrating a large scope of subliminally transferrable signals.
  arXiv  Detail & Related papers  (2026-04-28T15:51:55Z)
• Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer [29.621788488063853]
  Language models can transfer hidden biases during distillation.<n>This phenomenon is called subliminal learning.<n>Subliminal learning can be expected under soft distillation, where the student is trained on the teacher's full next-token distribution.
  arXiv  Detail & Related papers  (2025-09-28T13:51:22Z)
• Subliminal Learning: Language models transmit behavioral traits via hidden signals in data [7.092398764321311]
  We study subliminal learning, a phenomenon where language models transmit behavioral traits via semantically unrelated data.<n>We prove a theoretical result showing that subliminal learning occurs in all neural networks under certain conditions.<n>We conclude that subliminal learning is a general phenomenon that presents an unexpected pitfall for AI development.
  arXiv  Detail & Related papers  (2025-07-20T03:51:13Z)
• Simple Unsupervised Knowledge Distillation With Space Similarity [15.341380611979524]
  Self-supervised learning (SSL) does not readily extend to smaller architectures. We propose a simple objective to capture the lost information due to normalisation. Our proposed loss component, termed textbfspace similarity, motivates each dimension of a student's feature space to be similar to the corresponding dimension of its teacher.
  arXiv  Detail & Related papers  (2024-09-20T22:54:39Z)
• Embedding Compression for Teacher-to-Student Knowledge Transfer [58.01044612920816]
  We propose to use an embedding compression module with a trainable teacher transformation to obtain a compact teacher embedding. Results show that adding the embedding compression module improves the classification performance. Student models trained with the guidance of embeddings show stronger generalizability.
  arXiv  Detail & Related papers  (2024-02-09T19:47:31Z)
• Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers [15.211628096103475]
  We present an initial study that evaluates intra-episodic feedback given in a collaborative setting. Our results show that intra-episodic feedback allows the follower to generalize on aspects of scene complexity.
  arXiv  Detail & Related papers  (2023-05-22T10:01:15Z)
• UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit Representation [48.49860868061573]
  Recent neural implicit representations (NIRs) have achieved great success in the tasks of 3D reconstruction and novel view synthesis. They require the images of a scene from different camera views to be available for one-time training. This is expensive especially for scenarios with large-scale scenes and limited data storage. We design a student-teacher framework to mitigate the catastrophic problem.
  arXiv  Detail & Related papers  (2022-12-21T11:43:20Z)
• Generalized Knowledge Distillation via Relationship Matching [53.69235109551099]
  Knowledge of a well-trained deep neural network (a.k.a. the "teacher") is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model. Instead of enforcing the teacher to work on the same task as the student, we borrow the knowledge from a teacher trained from a general label space.
  arXiv  Detail & Related papers  (2022-05-04T06:49:47Z)
• Iterative Teacher-Aware Learning [136.05341445369265]
  In human pedagogy, teachers and students can interact adaptively to maximize communication efficiency. We propose a gradient optimization based teacher-aware learner who can incorporate teacher's cooperative intention into the likelihood function.
  arXiv  Detail & Related papers  (2021-10-01T00:27:47Z)
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation [72.4354883997316]
  We propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student. Our method is very flexible that can be easily combined with other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-03-31T06:52:20Z)
• Bridging the Imitation Gap by Adaptive Insubordination [88.35564081175642]
  We show that when the teaching agent makes decisions with access to privileged information, this information is marginalized during imitation learning. We propose 'Adaptive Insubordination' (ADVISOR) to address this gap. ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration.
  arXiv  Detail & Related papers  (2020-07-23T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.