MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation

• URL: http://arxiv.org/abs/2605.01374v1
• Date: Sat, 02 May 2026 10:52:49 GMT
• Title: MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
• Abstract summary: We propose a framework that aligns teacher and student representations along their layer-wise transformation trajectory.<n>Lower layers are aligned at the word level to preserve lexical information, while higher layers operate on phrase-level spans to capture compositional semantics.
• Score: 17.375971374283385
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation is a key technique for compressing large language models (LLMs), but most existing methods align representations at fixed layers or token-level outputs, ignoring how representations evolve across depth. As a result, the student is only weakly guided to capture the teacher's internal relational structure during distillation, which limits knowledge transfer. To address this limitation, we propose Multi-Granular Trajectory Alignment (MTA), a framework that aligns teacher and student representations along their layer-wise transformation trajectory. MTA adopts a layer-adaptive strategy: lower layers are aligned at the word level to preserve lexical information, while higher layers operate on phrase-level spans (e.g., noun and verb phrases) to capture compositional semantics. We instantiate this idea through a Dynamic Structural Alignment loss that matches the relative geometry among semantic units within each layer. This design is motivated by empirical findings that Transformer representations become increasingly abstract with depth, and is also consistent with linguistic views in which higher-level meaning emerges through the composition of lower-level lexical units. We further incorporate a Hidden Representation Alignment loss to directly align selected teacher-student layers. Experiments show that MTA consistently outperforms state-of-the-art baselines on standard benchmarks, with ablations confirming the contribution of each component.

Related papers

• An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars [54.11540943172608]
  Deep neural networks are widely believed to derive their expressive power from their ability to form textbfhierarchical representations<n>In language modeling, textbftransformers have emerged as the dominant architecture, with early layers capturing local syntactic patterns and later layers encoding more complex clause-level dependencies.<n>We argue that these architectures possess the structural capacity to encode abstract grammatical states into low-dimensional, linearly separable subspaces within the residual stream.
  arXiv  Detail & Related papers  (2026-06-16T05:02:13Z)
• Visual Instruction Tuning Aligns Modalities through Abstraction [7.963871750032694]
  We show that instruction tuning serves as a bridge, embedding visual features directly into the intermediate semantic layers of the Large Language Model.<n>We show that these intermediate layers are the semantic core of vision-language processing and play a critical role in the performance on a broad set of multimodal benchmarks.
  arXiv  Detail & Related papers  (2026-06-02T16:42:21Z)
• Uncovering the Latent Potential of Deep Intermediate Representations [16.988545433223933]
  We show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by nave aggregation.<n>We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces.<n>Our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses.
  arXiv  Detail & Related papers  (2026-05-21T20:58:42Z)
• Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models [13.707653566827704]
  Transformer models achieve state-of-the-art performance across domains and tasks, yet their deeply layered representations make their predictions difficult to interpret.<n>Existing explainability methods rely on final-layer attributions, capture either local token-level attributions or global attention patterns without unification.<n>We propose a unified hierarchical attribution framework that computes layer-wise Integrated Gradients within each Transformer block and fuses these token-level attributions with class-specific attention gradients.
  arXiv  Detail & Related papers  (2026-02-18T17:03:10Z)
• DVLA-RL: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning [53.36809572236361]
  Few-shot learning aims to generalize to novel categories with only a few samples.<n>Recent approaches incorporate large language models to enrich visual representations with semantic embeddings derived from class names.<n>We propose Dual-level Vision-Language Alignment with Reinforcement Learning gating (DVLA-RL)
  arXiv  Detail & Related papers  (2026-01-31T16:09:37Z)
• Structured Agent Distillation for Large Language Model [56.38279355868093]
  We propose Structured Agent Distillation, a framework that compresses large LLM-based agents into smaller student models.<n>Our method segments trajectories into [REASON] and [ACT] spans, applying segment-specific losses to align each component with the teacher's behavior.<n>Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines.
  arXiv  Detail & Related papers  (2025-05-20T02:01:55Z)
• Learning and Evaluating Hierarchical Feature Representations [3.770103075126785]
  We propose a novel framework, Hierarchical Composition of Orthogonal Subspaces (Hier-COS)<n>Hier-COS learns to map deep feature embeddings into a vector space that is, by design, consistent with the structure of a given taxonomy tree.<n>We demonstrate that Hier-COS achieves state-of-the-art hierarchical performance across all the datasets while simultaneously beating top-1 accuracy in all but one case.
  arXiv  Detail & Related papers  (2025-03-10T20:59:41Z)
• Layer by Layer: Uncovering Hidden Representations in Language Models [28.304269706993942]
  We show that intermediate layers can encode even richer representations, often improving performance on a range of downstream tasks.<n>Our framework highlights how each layer balances information compression and signal preservation, revealing why mid-depth embeddings can exceed the last layer's performance.
  arXiv  Detail & Related papers  (2025-02-04T05:03:42Z)
• Emergent effects of scaling on the functional hierarchies within large language models [0.0]
  Analyses using a small model (Llama-3.2-3b; 28 layers)<n>Item-level semantics are most strongly represented early (layers 2-7), then two-item relations (layers 8-12), and then four-item analogies (layers 10-15)<n>Deep layers also compress information from early portions of the context window without meaningful abstraction.
  arXiv  Detail & Related papers  (2025-01-13T14:27:39Z)
• Does Representation Matter? Exploring Intermediate Layers in Large Language Models [22.704926222438456]
  We investigate the quality of intermediate representations in large language models (LLMs)<n>We find that intermediate layers often yield more informative representations for downstream tasks than the final layers.<n>Our results illuminate the internal mechanics of LLMs and guide strategies for architectural optimization and training.
  arXiv  Detail & Related papers  (2024-12-12T18:48:51Z)
• Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics [50.982315553104975]
  We investigate the bottom-up evolution of lexical semantics for a popular large language model, namely Llama2. Our experiments show that the representations in lower layers encode lexical semantics, while the higher layers, with weaker semantic induction, are responsible for prediction. This is in contrast to models with discriminative objectives, such as mask language modeling, where the higher layers obtain better lexical semantics.
  arXiv  Detail & Related papers  (2024-03-03T13:14:47Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Exploring the Hierarchy in Relation Labels for Scene Graph Generation [75.88758055269948]
  The proposed method can improve several state-of-the-art baselines by a large margin (up to $33%$ relative gain) in terms of Recall@50. Experiments show that the proposed simple yet effective method can improve several state-of-the-art baselines by a large margin.
  arXiv  Detail & Related papers  (2020-09-12T17:36:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.