Collapse-Free Prototype Readout Layer for Transformer Encoders

• URL: http://arxiv.org/abs/2604.03850v1
• Date: Sat, 04 Apr 2026 20:23:21 GMT
• Title: Collapse-Free Prototype Readout Layer for Transformer Encoders
• Abstract summary: DDCL-Attention is a prototype-based readout layer for transformer encoders.<n>It produces compact token summaries at linear complexity in sequence length.<n>Same framework supports three uses: a final readout layer, a differentiable codebook extending VQ-VAE, and a hierarchical document compressor.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: DDCL-Attention is a prototype-based readout layer for transformer encoders that replaces simple pooling methods, such as mean pooling or class tokens, with a learned compression mechanism. It uses a small set of global prototype vectors and assigns tokens to them through soft probabilistic matching, producing compact token summaries at linear complexity in sequence length. The method offers three main advantages. First, it avoids prototype collapse through an exact decomposition of the training loss into a reconstruction term and a diversity term, ensuring that prototypes remain distinct. Second, its joint training with the encoder is shown to be stable under a practical timescale condition, using Tikhonov's singular perturbation theory and explicit learning-rate constraints. Third, the same framework supports three uses: a final readout layer, a differentiable codebook extending VQ-VAE, and a hierarchical document compressor. Experiments on four datasets confirm the theoretical predictions: the loss decomposition holds exactly, prototype separation grows as expected when the stability condition is met, and the codebook reaches full utilization, outperforming standard hard vector quantization. An additional study on orbital debris classification shows that the method also applies beyond standard NLP and vision tasks, including scientific tabular data.

Related papers

• The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation [0.30693357740321775]
  Concept-bottleneck controllable generation routes multi-attribute control through a low-dimensional concept code.<n>We study this problem in concept-bottleneck text generation under multi-axis compositional generalisation.
  arXiv  Detail & Related papers  (2026-08-24T08:21:10Z)
• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression? [0.0]
  We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs.<n>A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace.<n>These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression.
  arXiv  Detail & Related papers  (2026-07-19T22:19:17Z)
• Multigrain-aware Semantic Prototype Scanning and Tri-Token Prompt Learning Embraced High-Order RWKV for Pan-Sharpening [46.94213480279033]
  We propose a Multigrain-aware Semantic Prototype Scanning paradigm for pan-sharpening.<n>We use a high-order RWKV architecture and a tri-token prompting mechanism derived from semantic clustering.<n> Experimental results demonstrate the superiority of our method.
  arXiv  Detail & Related papers  (2026-04-16T05:10:03Z)
• TENSURE: Fuzzing Sparse Tensor Compilers (Registered Report) [9.083996936429465]
  We present TENSURE, the first black-box fuzzing framework specifically designed for the testing of sparse Compilers.<n>We propose a novel constraint-based generation algorithm that guarantees 100% semantic validity of synthesized kernels.<n>Our evaluation on two state-of-the-art systems, TACO and Finch, reveals widespread fragility, particularly in TACO, where TENSURE exposed crashes or silent miscompilations.
  arXiv  Detail & Related papers  (2026-03-19T00:13:14Z)
• Tail-Aware Post-Training Quantization for 3D Geometry Models [58.79500829118265]
  Post-Training Quantization (PTQ) enables efficient inference without retraining.<n>PTQ fails to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead.<n>We propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline for 3D geometric learning.
  arXiv  Detail & Related papers  (2026-02-02T07:21:15Z)
• From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers [67.02076505996284]
  We study how the choice of pretraining data distribution steers a shallow transformer toward one behavior or the other.<n>Our results shed light on the algorithmic biases of pretrained transformers and offer conceptual guidelines for data-driven control of their learned behaviors.
  arXiv  Detail & Related papers  (2025-12-21T08:10:26Z)
• Pooling Attention: Evaluating Pretrained Transformer Embeddings for Deception Classification [0.0]
  BERT embeddings combined with logistic regression outperform neural baselines on LIAR dataset splits.<n>This work positions attention-based token encoders as robust, architecture-centric foundations for veracity tasks.
  arXiv  Detail & Related papers  (2025-11-28T08:32:49Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection [24.11900895337062]
  Unsupervised image Anomaly Detection (UAD) aims to learn robust and discriminative representations of normal samples. This paper focuses on building a unified framework for multiple classes.
  arXiv  Detail & Related papers  (2023-10-22T08:20:33Z)
• Attention over pre-trained Sentence Embeddings for Long Document Classification [4.38566347001872]
  transformers are often limited to short sequences due to their quadratic attention complexity on the number of tokens. We suggest to take advantage of pre-trained sentence transformers to start from semantically meaningful embeddings of the individual sentences. We report the results obtained by this simple architecture on three standard document classification datasets.
  arXiv  Detail & Related papers  (2023-07-18T09:06:35Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• A Light-Weight Contrastive Approach for Aligning Human Pose Sequences [1.0152838128195467]
  Training samples consist of temporal windows of frames containing 3D body points such as mocap markers or skeleton joints. A light-weight, 3-layer encoder is trained using a contrastive loss function that encourages embedding vectors of augmented sample pairs to have cosine similarity 1, and similarity 0 with all other samples in a minibatch. In addition to being simple, the proposed method is fast to train, making it easy to adapt to new data using different marker sets or skeletal joint layouts.
  arXiv  Detail & Related papers  (2023-03-07T21:35:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.