DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing

• URL: http://arxiv.org/abs/2606.15796v1
• Date: Sun, 14 Jun 2026 13:02:44 GMT
• Title: DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing
• Abstract summary: We extend transcoder-based circuit tracing to multimodal diffusion transformers.<n>We find mechanisms underlying attribute binding and cross-stream semantic propagation.<n>Our results provide causal explanations for systematic generation errors.
• Score: 8.306039957986394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits. While transcoder-based circuit tracing has recently enabled detailed causal analyses of large language models, multimodal diffusion transformers for image generation remain comparatively opaque. We still lack tools for understanding how semantic information propagates across denoising steps and how text and image representations interact within double-stream MM-DiT architectures. Existing methods provide only partial insight: attention maps expose a limited view of token interactions, while sparse autoencoders can discover interpretable features but do not directly reveal how these features are transformed and composed through nonlinear MLP layers. In this work, we extend transcoder-based circuit tracing to multimodal diffusion transformers. We train timestep-conditioned transcoders that faithfully approximate the input-output behavior of MLP sublayers in FLUX.1[schnell]. By replacing MLPs with transcoders and linearizing the remaining computation, we obtain exact feature-to-feature attribution and recover compact, interpretable circuits. Empirically, our transcoders match or slightly outperform sparse autoencoders on the sparsity-faithfulness tradeoff. The resulting circuits reveal mechanisms underlying attribute binding and cross-stream semantic propagation, and provide causal explanations for systematic generation errors. Moreover, circuit-guided interventions are substantially more precise and effective than standard SAE-based steering. Our results demonstrate that transcoder-based circuit analysis is feasible for state-of-the-art diffusion transformers and provides a powerful framework for understanding and controlling multimodal generative models. The code is available at https://github.com/Artalmaz31/DifFRACT

Related papers

• Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models [7.425292797106273]
  Generative Vision-Language Models (VLMs) perform well on multimodal reasoning, but how visual inputs are transformed to text remains poorly understood.<n>We adopt a function-centric framework based on Transcoders, sparse approximations of sublayers that act as a causal proxy for layer-wise.<n>We show that function-centric circuit decomposition yields interpretable and predictive accounts of multimodal computation inVLMs.
  arXiv  Detail & Related papers  (2026-05-21T17:34:39Z)
• Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers [54.2202951543048]
  We investigate whether useful mechanistic circuits can be identified through computational graphs in vision transformers.<n>We propose an effective method for Automatic Visual Circuit Discovery (Vi-CD) that recovers class-specific circuits for classification.<n>We find that insightful and actionable edge-based circuits can be recovered from vision transformers, adding transparency to the internal computations of these models.
  arXiv  Detail & Related papers  (2026-04-15T23:25:46Z)
• Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer [65.38883376379812]
  We propose the Discrete Transformer, an architecture engineered to bridge the gap between continuous representations and discrete symbolic logic.<n> Empirically, the Discrete Transformer not only achieves performance comparable to RNN-based baselines but crucially extends interpretability to continuous variable domains.
  arXiv  Detail & Related papers  (2026-01-09T12:49:41Z)
• A Transformer Inspired AI-based MIMO receiver [0.5039813366558306]
  The AttDet design combines model-based interpretability with data-driven flexibility.<n>We demonstrate through link-level simulations under 5G channel models and high-order, mixed QAM modulation and coding schemes.<n>AttDet can approach near-optimal BER/BLER performance while maintaining predictable, realistic complexity.
  arXiv  Detail & Related papers  (2025-10-23T09:05:10Z)
• TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation [41.909091496502704]
  Diffusion Transformers (DiTs) are a powerful yet underexplored class of generative models.<n>We propose TIDE-Temporal-aware sparse autoencoders for Interpretable Diffusion transformErs framework.
  arXiv  Detail & Related papers  (2025-03-10T08:35:51Z)
• ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer [95.80384464922147]
  ACDiT is a blockwise Conditional Diffusion Transformer.<n>It offers a flexible between token-wise autoregression and full-sequence diffusion.<n>We show that ACDiT performs best among all autoregressive baselines on image and video generation tasks.
  arXiv  Detail & Related papers  (2024-12-10T18:13:20Z)
• Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators [83.48423407316713]
  We present a novel diffusion transformer framework incorporating an additional set of mediator tokens to engage with queries and keys separately. Our model initiates the denoising process with a precise, non-ambiguous stage and gradually transitions to a phase enriched with detail. Our method achieves a state-of-the-art FID score of 2.01 when integrated with the recent work SiT.
  arXiv  Detail & Related papers  (2024-08-11T07:01:39Z)
• Transcoders Find Interpretable LLM Feature Circuits [1.4254279830438588]
  We introduce a novel method for using transcoders to perform circuit analysis through sublayers. We train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability.
  arXiv  Detail & Related papers  (2024-06-17T17:49:00Z)
• Transformers need glasses! Information over-squashing in language tasks [18.81066657470662]
  We study how information propagates in decoder-only Transformers. We show that certain sequences of inputs to the Transformer can yield arbitrarily close representations in the final token. We also show that decoder-only Transformer language models can lose sensitivity to specific tokens in the input.
  arXiv  Detail & Related papers  (2024-06-06T17:14:44Z)
• Cross-receptive Focused Inference Network for Lightweight Image Super-Resolution [64.25751738088015]
  Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks. Transformers that need to incorporate contextual information to extract features dynamically are neglected. We propose a lightweight Cross-receptive Focused Inference Network (CFIN) that consists of a cascade of CT Blocks mixed with CNN and Transformer.
  arXiv  Detail & Related papers  (2022-07-06T16:32:29Z)
• Error Correction Code Transformer [92.10654749898927]
  We propose to extend for the first time the Transformer architecture to the soft decoding of linear codes at arbitrary block lengths. We encode each channel's output dimension to high dimension for better representation of the bits information to be processed separately. The proposed approach demonstrates the extreme power and flexibility of Transformers and outperforms existing state-of-the-art neural decoders by large margins at a fraction of their time complexity.
  arXiv  Detail & Related papers  (2022-03-27T15:25:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.