Circuit Distillation

• URL: http://arxiv.org/abs/2509.25002v1
• Date: Mon, 29 Sep 2025 16:28:36 GMT
• Title: Circuit Distillation
• Abstract summary: We propose circuit distillation, which aims to align internal representations between analogous circuit components in teacher and student models.<n>We evaluate circuit distillation on entity tracking and theory of mind (ToM) tasks using models from the Llama3 family.
• Score: 18.631673704920296
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model distillation typically focuses on behavioral mimicry, where a student model is trained to replicate a teacher's output while treating its internal computations as a black box. In this work we propose an alternative approach: Distilling the underlying computational mechanisms implemented by a teacher model. Specifically, we propose circuit distillation, which introduces an objective to align internal representations between analogous circuit components in teacher and student models. We propose a method to match ``functionally correspondent'' circuit components and introduce a loss reflecting similarities between the representations that these induce. We evaluate circuit distillation on entity tracking and theory of mind (ToM) tasks using models from the Llama3 family. Our results demonstrate that circuit distillation outperforms standard distillation, successfully transferring algorithmic capabilities by adjusting only a small, targeted subset of student model parameters. This work establishes the feasibility of transferring mechanisms, which may in turn allow for efficient distillation of targeted teacher capabilities via interpretable and controllable internal student mechanisms.

Related papers

• Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillation [50.19746127327559]
  We propose a new tail-aware divergence that decouples the contribution of the teacher model's top-K predicted probabilities from that of lower-probability predictions.<n> Experimental results demonstrate that our modified distillation method yields competitive performance in both pre-training and supervised distillation of decoder models.
  arXiv  Detail & Related papers  (2026-02-24T11:54:06Z)
• Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspaces [17.627125013326175]
  'SubDistill' is a new distillation algorithm with improved numerical properties that only distills the relevant components of the teacher model at each layer.<n>Our benchmark evaluations are complemented by Explainable AI analyses showing that our distilled student models more closely match the decision structure of the original teacher model.
  arXiv  Detail & Related papers  (2026-01-09T16:28:55Z)
• Knowledge distillation through geometry-aware representational alignment [3.901188865224763]
  We show that existing feature distillation methods cannot capture the feature structure, even under zero loss.<n>We then motivate the use of Procrustes distance and the Frobenius norm of Feature Gram Matrix, distances already common in the context of measuring representational alignment.<n>We show that feature distillation through our method showcases statistically significant improvement in distillation performance across language models families.
  arXiv  Detail & Related papers  (2025-09-27T09:59:46Z)
• Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation [0.3683202928838613]
  We analyze how internal circuits, representations, and activation patterns differ between teacher and student.<n>We find that student models reorganize, compress, and discard teacher components, often resulting in stronger reliance on fewer individual components.
  arXiv  Detail & Related papers  (2025-05-16T03:37:40Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• DETRDistill: A Universal Knowledge Distillation Framework for DETR-families [11.9748352746424]
  Transformer-based detectors (DETRs) have attracted great attention due to their sparse training paradigm and the removal of post-processing operations. Knowledge distillation (KD) can be employed to compress the huge model by constructing a universal teacher-student learning framework.
  arXiv  Detail & Related papers  (2022-11-17T13:35:11Z)
• Delta Distillation for Efficient Video Processing [68.81730245303591]
  We propose a novel knowledge distillation schema coined as Delta Distillation. We demonstrate that these temporal variations can be effectively distilled due to the temporal redundancies within video frames. As a by-product, delta distillation improves the temporal consistency of the teacher model.
  arXiv  Detail & Related papers  (2022-03-17T20:13:30Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.