Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision

• URL: http://arxiv.org/abs/2604.13304v1
• Date: Tue, 14 Apr 2026 21:18:08 GMT
• Title: Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision
• Abstract summary: We introduce Cross-Layer Transcoders (CLTs) as reliable, sparse and depth-aware proxy models for Vision Transformers (ViTs)<n>CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers.<n>We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy.
• Score: 29.67524916084184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introduce the adoption of Cross-Layer Transcoders (CLTs) as reliable, sparse, and depth-aware proxy models for MLP blocks in ViTs. CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers, yielding a linear decomposition that transforms the final representation of ViTs from an opaque embedding into an additive, layer-resolved construction that enables faithful attribution and process-level interpretability. We train CLTs on CLIP ViT-B/32 and ViT-B/16 across CIFAR-100, COCO, and ImageNet-100. We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy. In terms of interpretability, we show that the cross-layer contribution scores provide faithful attribution, revealing that the final representation is concentrated in a smaller set of dominant layer-wise terms whose removal degrades performance and whose retention largely preserves it. These results showcase the significance of adopting CLTs as an alternative interpretable proxy of ViTs in the vision domain.

Related papers

• Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding [49.540751364954666]
  Visual Token Codec separates global and patch tokens into dedicated coding paths.<n>We show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks.
  arXiv  Detail & Related papers  (2026-08-09T17:32:04Z)
• ViKANformer: Embedding Kolmogorov Arnold Networks in Vision Transformers for Pattern-Based Learning [0.0]
  Vision Transformers (ViTs) have significantly advanced image classification by applying self-attention on patch embeddings.<n>In this paper, we propose ViKANformer, a Vision Transformer where we replace the sub-layers with Kolmogorov-Arnold Network (KAN) expansions.
  arXiv  Detail & Related papers  (2025-03-03T03:10:26Z)
• BreakNet: Discontinuity-Resilient Multi-Scale Transformer Segmentation of Retinal Layers [0.8953337264557399]
  BreakNet is a Transformer-based segmentation model designed to address boundary discontinuities caused by shadow artifacts. Our findings indicate that BreakNet has the potential to significantly improve retinal quantification and analysis.
  arXiv  Detail & Related papers  (2024-08-26T19:59:20Z)
• SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers [76.13755422671822]
  This paper investigates the capability of plain Vision Transformers (ViTs) for semantic segmentation using the encoder-decoder framework. We introduce a novel Attention-to-Mask (atm) module to design a lightweight decoder effective for plain ViT. Our decoder outperforms the popular decoder UPerNet using various ViT backbones while consuming only about $5%$ of the computational cost.
  arXiv  Detail & Related papers  (2023-06-09T22:29:56Z)
• White-Box Transformers via Sparse Rate Reduction [25.51855431031564]
  We show a family of white-box transformer-like deep network architectures which are mathematically fully interpretable. Experiments show that these networks indeed learn to optimize the designed objective.
  arXiv  Detail & Related papers  (2023-06-01T20:28:44Z)
• Making Vision Transformers Efficient from A Token Sparsification View [26.42498120556985]
  We propose a novel Semantic Token ViT (STViT) for efficient global and local vision transformers. Our method can achieve competitive results compared to the original networks in object detection and instance segmentation, with over 30% FLOPs reduction for backbone. In addition, we design a STViT-R(ecover) network to restore the detailed spatial information based on the STViT, making it work for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-15T15:12:36Z)
• Unifying Global-Local Representations in Salient Object Detection with Transformer [55.23033277636774]
  We introduce a new attention-based encoder, vision transformer, into salient object detection. With the global view in very shallow layers, the transformer encoder preserves more local representations. Our method significantly outperforms other FCN-based and transformer-based methods in five benchmarks.
  arXiv  Detail & Related papers  (2021-08-05T17:51:32Z)
• Vision Transformer with Progressive Sampling [73.60630716500154]
  We propose an iterative and progressive sampling strategy to locate discriminative regions. When trained from scratch on ImageNet, PS-ViT performs 3.8% higher than the vanilla ViT in terms of top-1 accuracy.
  arXiv  Detail & Related papers  (2021-08-03T18:04:31Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)
• Scalable Visual Transformers with Hierarchical Pooling [61.05787583247392]
  We propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length. It brings a great benefit by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity. Our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2021-03-19T03:55:58Z)
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers [149.78470371525754]
  We treat semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR) SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes.
  arXiv  Detail & Related papers  (2020-12-31T18:55:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.