A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

• URL: http://arxiv.org/abs/2607.08776v1
• Date: Tue, 05 May 2026 07:24:01 GMT
• Title: A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
• Abstract summary: We propose a unified approach to explore the common mechanism of various KD methods using interactions.<n>We discover that the common mechanism underlying various KD methods is the sparsification of interactions.<n>Motivated by these insights, we propose a plug-and-play loss function called Complex Interaction Penalty (CIP) to explicitly enforce the sparsity of complex interactions.
• Score: 6.929805474293474
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the success of knowledge distillation (KD) in Large Language Models (LLMs), the underlying mechanism behind its efficacy remains unclear. In this paper, we propose a unified approach to explore the common mechanism of various KD methods using interactions. Specifically, we decompose the output score of the LLM into the sum of numerous interactions. Each interaction represents a nonlinear relationship involving a set of input variables (e.g., words). Based on the decomposed interactions, we discover that the common mechanism underlying various KD methods is the sparsification of interactions, i.e., student models retain fewer interactions for inference while suppressing other interactions to zero effects. Furthermore, we discover that the performance variance across different KD methods arises from their capabilities in handling complex interactions. A KD method typically yields better performance if it enables the student model to achieve higher sparsity of complex interactions. Motivated by these insights, we propose a plug-and-play loss function called Complex Interaction Penalty (CIP) to explicitly enforce the sparsity of complex interactions during the distillation process. Extensive experiments demonstrate that integrating CIP consistently improves the performance of diverse KD methods on both in-domain and out-of-distribution benchmarks.

Related papers

• SAILS: Surrogate-based Analysis of Interactions via Local Effect Smooths [12.435976341898135]
  Surrogate-based Analysis of Interactions via Local effect Smooths (SAILS) is a model-agnostic framework that analyzes pairwise interactions.<n>For each interval of a feature of interest, surrogate terms isolate the interaction components on derivative level.<n>We empirically validate the framework through controlled simulations and a real-world task, demonstrating its effectiveness for pairwise interactions.
  arXiv  Detail & Related papers  (2026-06-08T12:22:02Z)
• Dynamic Programming Techniques for Enhancing Cognitive Representation in Knowledge Tracing [125.75923987618977]
  We propose the Cognitive Representation Dynamic Programming based Knowledge Tracing (CRDP-KT) model.<n>It is a dynamic programming algorithm to optimize cognitive representations based on the difficulty of the questions and the performance intervals between them.<n>It provides more accurate and systematic input features for subsequent model training, thereby minimizing distortion in the simulation of cognitive states.
  arXiv  Detail & Related papers  (2025-06-03T14:44:48Z)
• iLOCO: Distribution-Free Inference for Feature Interactions [4.56754610152086]
  We develop a new model-agnostic metric for measuring the importance of pairwise feature interactions.<n>We also introduce an ensemble learning method for calculating the iLOCO metric and confidence intervals.<n>We validate our iLOCO metric and our confidence intervals on both synthetic and real data sets.
  arXiv  Detail & Related papers  (2025-02-10T16:49:46Z)
• Composable Interventions for Language Models [60.32695044723103]
  Test-time interventions for language models can enhance factual accuracy, mitigate harmful outputs, and improve model efficiency without costly retraining.<n>But despite a flood of new methods, different types of interventions are largely developing independently.<n>We introduce composable interventions, a framework to study the effects of using multiple interventions on the same language models.
  arXiv  Detail & Related papers  (2024-07-09T01:17:44Z)
• AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents [65.16893197330589]
  Large Language Models (LLMs) have demonstrated their ability to replicate human behaviors across a wide range of scenarios. However, their capability in handling complex, multi-character social interactions has yet to be fully explored. We introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods.
  arXiv  Detail & Related papers  (2024-01-12T11:18:00Z)
• MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts [92.76662894585809]
  We introduce an approach to enhance multimodal models, which we call Multimodal Mixtures of Experts (MMoE) MMoE is able to be applied to various types of models to gain improvement.
  arXiv  Detail & Related papers  (2023-11-16T05:31:21Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Learning Interaction Variables and Kernels from Observations of Agent-Based Systems [14.240266845551488]
  We propose a learning technique that, given observations of states and velocities along trajectories of agents, yields both the variables upon which the interaction kernel depends and the interaction kernel itself. This yields an effective dimension reduction which avoids the curse of dimensionality from the high-dimensional observation data. We demonstrate the learning capability of our method to a variety of first-order interacting systems.
  arXiv  Detail & Related papers  (2022-08-04T16:31:01Z)
• Coarse-to-Fine Knowledge-Enhanced Multi-Interest Learning Framework for Multi-Behavior Recommendation [52.89816309759537]
  Multi-types of behaviors (e.g., clicking, adding to cart, purchasing, etc.) widely exist in most real-world recommendation scenarios. The state-of-the-art multi-behavior models learn behavior dependencies indistinguishably with all historical interactions as input. We propose a novel Coarse-to-fine Knowledge-enhanced Multi-interest Learning framework to learn shared and behavior-specific interests for different behaviors.
  arXiv  Detail & Related papers  (2022-08-03T05:28:14Z)
• Interaction Pattern Disentangling for Multi-Agent Reinforcement Learning [39.4394389642761]
  We introduce a novel interactiOn Pattern disenTangling (OPT) method to disentangle the entity interactions into interaction prototypes. OPT facilitates filtering the noisy interactions between irrelevant entities and thus significantly improves generalizability as well as interpretability. Experiments on single-task, multi-task and zero-shot benchmarks demonstrate that the proposed method yields results superior to the state-of-the-art counterparts.
  arXiv  Detail & Related papers  (2022-07-08T13:42:54Z)
• REPID: Regional Effect Plots with implicit Interaction Detection [0.9023847175654603]
  Interpretable machine learning methods visualize marginal feature effects but may lead to misleading interpretations when feature interactions are present. We introduce implicit interaction detection, a novel framework to detect interactions between a feature of interest and other features. The framework also quantifies the strength of interactions and provides interpretable and distinct regions in which feature effects can be interpreted more reliably.
  arXiv  Detail & Related papers  (2022-02-15T08:54:00Z)
• Multi-Agent Imitation Learning with Copulas [102.27052968901894]
  Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
  arXiv  Detail & Related papers  (2021-07-10T03:49:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.