Localizing Knowledge in Diffusion Transformers

• URL: http://arxiv.org/abs/2505.18832v1
• Date: Sat, 24 May 2025 19:02:20 GMT
• Title: Localizing Knowledge in Diffusion Transformers
• Authors: Arman Zarei, Samyadeep Basu, Keivan Rezaei, Zihao Lin, Sayan Nag, Soheil Feizi,
• Abstract summary: We propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the Diffusion Transformer blocks.<n>We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs.<n>Our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing.
• Score: 44.27817967554535
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding how knowledge is distributed across the layers of generative models is crucial for improving interpretability, controllability, and adaptation. While prior work has explored knowledge localization in UNet-based architectures, Diffusion Transformer (DiT)-based models remain underexplored in this context. In this paper, we propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the DiT blocks. We evaluate our method on state-of-the-art DiT-based models, including PixArt-alpha, FLUX, and SANA, across six diverse knowledge categories. We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs. Building on these insights, we apply our localization framework to two key applications: model personalization and knowledge unlearning. In both settings, our localized fine-tuning approach enables efficient and targeted updates, reducing computational cost, improving task-specific performance, and better preserving general model behavior with minimal interference to unrelated or surrounding content. Overall, our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing.

Related papers

• How Bidirectionality Helps Language Models Learn Better via Dynamic Bottleneck Estimation [4.670329628077522]
  Bidirectional language models have better context understanding and perform better than unidirectional models on natural language understanding tasks.<n>We propose FlowNIB, a dynamic and scalable method for estimating mutual information during training.<n>We show that bidirectional models retain more mutual information and exhibit higher effective dimensionality than unidirectional models.
  arXiv  Detail & Related papers  (2025-06-01T06:56:45Z)
• Empirical Evaluation of Knowledge Distillation from Transformers to Subquadratic Language Models [3.287942619833188]
  We systematically evaluate the transferability of knowledge distillation from a Transformer teacher model to eight subquadratic student architectures.<n>Our study investigates which subquadratic model can most effectively approximate the teacher model's learned representations through knowledge distillation.
  arXiv  Detail & Related papers  (2025-04-19T17:49:52Z)
• Quantized and Interpretable Learning Scheme for Deep Neural Networks in Classification Task [0.0]
  We introduce an approach that combines saliency-guided training with quantization techniques to create an interpretable and resource-efficient model.<n>Our results demonstrate that the combined use of saliency-guided training and PACT-based quantization not only maintains classification performance but also produces models that are significantly more efficient and interpretable.
  arXiv  Detail & Related papers  (2024-12-05T06:34:06Z)
• Structure-aware Domain Knowledge Injection for Large Language Models [38.08691252042949]
  StructTuning is a methodology to transform Large Language Models (LLMs) into domain specialists.<n>It significantly reduces the training corpus needs to a mere 5% while achieving an impressive 100% of traditional knowledge injection performance.
  arXiv  Detail & Related papers  (2024-07-23T12:38:48Z)
• Contextualized Structural Self-supervised Learning for Ontology Matching [0.9402105308876642]
  We introduce a novel self-supervised learning framework called LaKERMap. LaKERMap capitalizes on the contextual and structural information of concepts by integrating implicit knowledge into transformers. The findings from our innovative approach reveal that LaKERMap surpasses state-of-the-art systems in terms of alignment quality and inference time.
  arXiv  Detail & Related papers  (2023-10-05T18:51:33Z)
• Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation [68.13453771001522]
  We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings. We conduct extensive experiments and evaluate our model on large-scale real-world data.
  arXiv  Detail & Related papers  (2023-06-14T13:07:48Z)
• Plug-and-Play Knowledge Injection for Pre-trained Language Models [116.37916535076478]
  Injecting external knowledge can improve the performance of pre-trained language models (PLMs) on various downstream NLP tasks. Massive retraining is required to deploy new knowledge injection methods or knowledge bases for downstream tasks. We study how to improve the flexibility and efficiency of knowledge injection by reusing existing downstream models.
  arXiv  Detail & Related papers  (2023-05-28T10:58:00Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• Towards Interpretable Deep Learning Models for Knowledge Tracing [62.75876617721375]
  We propose to adopt the post-hoc method to tackle the interpretability issue for deep learning based knowledge tracing (DLKT) models. Specifically, we focus on applying the layer-wise relevance propagation (LRP) method to interpret RNN-based DLKT model. Experiment results show the feasibility using the LRP method for interpreting the DLKT model's predictions.
  arXiv  Detail & Related papers  (2020-05-13T04:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.