Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

• URL: http://arxiv.org/abs/2607.04432v1
• Date: Sun, 05 Jul 2026 17:52:09 GMT
• Title: Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation
• Abstract summary: Student model trained on pure uniform noise can still inherit its teacher's digit-classification ability.<n>Previous work proves this transfer is guaranteed when the teacher's learning rate is small enough.<n>We show these channels are not purely informational.
• Score: 0.4613981116181485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A student model trained on pure uniform noise can still inherit its teacher's digit-classification ability, provided the two share initialization. Previous work proves this transfer is guaranteed when the teacher's learning rate is small enough, but does not explain where in the network the channel lives or what sets its capacity. Working in an MLP distillation setting on MNIST, we show these channels are not purely informational: geometric alignment gates access to the information the channel carries. Shared initialization makes the output projection W_2 a common coordinate key, and KL gradients reshape the student's input projection W_0 until its hidden representations align with the teacher's. We call this covert trait propagation (CTP). Five experiments support this mechanism: channel closure tracks weight drift, not teacher accuracy; freezing W_0 destroys transfer while freezing W_2 leaves it intact; multi-teacher ensembles cancel out despite each teacher carrying comparable label information; and linear centered kernel alignment (CKA) tracks student accuracy at r=0.98 across a continuous initialization sweep. Applying the same geometric lens to cross-token behavioral entanglement (CTBE) in instruction-tuned LLMs, we find the effect appears to be activated by alignment training, acting on an inherited substrate, and that the standard log-ratio metric produces an apparent frequency bias that is largely a circularity artifact.

Related papers

• Follow the Geometry, Not the Model: Cold Start Semi-Supervised Learning [7.6344611819427035]
  We propose VAST (Veracity-Aware Semi-Supervised Training), which decouples pseudo-label generation and training.<n>VAST outperforms the strongest graph-based SSL baselines at every operating point across three datasets.
  arXiv  Detail & Related papers  (2026-09-13T11:46:26Z)
• The Distributional View of Knowledge Distillation [13.841028090842869]
  Token-level knowledge distillation matches two conditional distributions per position.<n>We develop a distributional view in which the teacher is represented not by a single softened output but by a family of multi-temperature views.
  arXiv  Detail & Related papers  (2026-08-15T13:03:31Z)
• Channel Location Constrains the Auditability of Subliminal Learning [0.0]
  Subliminal learning lets a student inherit a teacher's hidden trait from distillation data that never names it.<n>In pretrained language models, masked single-token traits instead ride convergent vocabulary geometry.<n>Channel location is necessary for deciding which audits can be sound.
  arXiv  Detail & Related papers  (2026-06-20T12:48:31Z)
• Trajectory-Refined Distillation [25.346810464266497]
  Trajectory-Refined Distillation (TRD) is a trajectory-level correction method that revises the student's rollout under the teacher guidance while within on-policy support.<n> TRD can also be applied to on-policy self-distillation (OPSD), a parameter-sharing variant that uses the student model conditioned on privileged informations as the teacher.
  arXiv  Detail & Related papers  (2026-06-07T03:17:25Z)
• Why Are DMD Students Lazy? Understanding the Copying Behavior in Few-Step Distillation [66.15462638744619]
  Distribution Matching Distillation (DMD) compresses pretrained diffusion models into efficient few-step generators by aligning their noised distributions across all scales.<n>Surprisingly, we find that in high-dimensional settings, distilled students spontaneously reproduce the original noise-data pairings of the teacher, a phenomenon we term copying.
  arXiv  Detail & Related papers  (2026-06-01T13:30:19Z)
• Graph Transfer Learning via Shared Latent Geometry: Theory and Applications [12.735610064958712]
  Inference and control in engineered physical systems pay a heavy physics cost at deployment.<n>We propose an asymmetric two-pathway architecture that resolves both issues.<n>A teacher encoder consumes privileged dense states from a high-fidelity simulator and represents the system through operator-polynomial features stable under spectral perturbation.<n>A student encoder learns the same latent geometry from sparse field data and operator descriptors.
  arXiv  Detail & Related papers  (2026-05-30T13:00:13Z)
• One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration [20.457786704615454]
  Fixed-Point Distillation (FPD) is an end-to-end framework that constructs local correction targets by partially corrupting the student's one-step draft and refining it with a single teacher step.<n>FPD achieves competitive visual fidelity and structural alignment within a single inference step, narrowing the gap to multi-step teachers.
  arXiv  Detail & Related papers  (2026-05-20T17:59:10Z)
• Belief or Circuitry? Causal Evidence for In-Context Graph Learning [3.9356042574269545]
  We show that either the model tracks global topology, or it copies local transitions.<n>We present two lines of evidence that neither account alone is sufficient.<n>Our findings are most consistent with a dual-mechanism account in which genuine structure inference and induction circuits operate in parallel.
  arXiv  Detail & Related papers  (2026-05-08T19:11:19Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)
• Visual Prompt Tuning in Null Space for Continual Learning [51.96411454304625]
  Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL) This paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient projection.
  arXiv  Detail & Related papers  (2024-06-09T05:57:40Z)
• Repainting and Imitating Learning for Lane Detection [52.5220065495956]
  Current lane detection methods are struggling with the invisibility lane issue caused by heavy shadows. We propose a novel Repainting and Imitating Learning framework containing a pair of teacher and student. Our method introduces no extra time cost during inference and can be plug-and-play in various cutting-edge lane detection networks.
  arXiv  Detail & Related papers  (2022-10-11T02:26:39Z)
• Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation [91.56643684860062]
  Inter-Channel Correlation for Knowledge Distillation(ICKD) is developed. ICKD captures intrinsic distribution of the featurespace and sufficient diversity properties of features in the teacher network. We are the first method based on knowl-edge distillation boosts ResNet18 beyond 72% Top-1 ac-curacy on ImageNet classification.
  arXiv  Detail & Related papers  (2022-02-08T07:01:56Z)
• Channel-wise Knowledge Distillation for Dense Prediction [73.99057249472735]
  We propose to align features channel-wise between the student and teacher networks. We consistently achieve superior performance on three benchmarks with various network structures.
  arXiv  Detail & Related papers  (2020-11-26T12:00:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.