Putting Registers to Work: Task Registers for Token Pruning in Vision Transformers

• URL: http://arxiv.org/abs/2608.10989v1
• Date: Tue, 11 Aug 2026 14:43:52 GMT
• Title: Putting Registers to Work: Task Registers for Token Pruning in Vision Transformers
• Abstract summary: Token-pruning policies are usually designed for a single recognition pipeline, but pretrained Vision Transformers are reused across tasks with different spatial demands.<n>We ask which parts of a pruning policy transfer across image classification, semantic segmentation, and object detection.<n>For each pipeline, controlled probes freeze the no-pruning checkpoint and apply a series of parameter-free reduction criteria at one eligible layer at a time without retraining.<n>The probes reveal three differences: segmentation and detection rank the criteria differently, classification is especially sensitive to attention-based pruning in the earliest layers, and the dense tasks prefer opposite recovery endpoints.
• Score: 5.410304374735817
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Token-pruning policies are usually designed for a single recognition pipeline, but pretrained Vision Transformers are reused across tasks with different spatial demands. We ask which parts of a pruning policy transfer across image classification, semantic segmentation, and object detection. For each pipeline, controlled probes freeze the no-pruning checkpoint and apply a series of parameter-free reduction criteria at one eligible layer at a time without retraining. The probes reveal three differences: segmentation and detection rank the criteria differently, classification is especially sensitive to attention-based pruning in the earliest layers, and the dense tasks prefer opposite recovery endpoints. These findings motivate Task-Adaptive Pruning (TAP). Existing register tokens serve as task-agnostic storage for feature artifacts. TAP instead introduces one task register per task and activates only the current one. Its evolving state ranks tokens, distributes an exact removal budget over depth, and sets the recovery scale for dense features. At a final keep rate of $ρ=0.5$, our jointly adapted model, TAP-J, reaches $47.0$ mIoU at $1.30\times$ encoder throughput on ADE20K and $53.7$ box AP at $1.32\times$ encoder throughput on COCO while remaining competitive on ImageNet-1K.

Related papers

• SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology [47.16406021922537]
  We introduce router, a lightweight GLiClass-based router that assigns a suitability score to each inference-time model label without autore generation.<n>For task generation, we construct a task structure with 23 families, 115 task types, 345 routable subtypes, 1,173 synthetic examples, and an axis of 30 domains.<n>We then train the Qwen3 decoder on these tasks, while explicitly separating learned request prediction from per-task policies for attributes such as eligibility, cost, cache reuse, safety, and sovereignty.
  arXiv  Detail & Related papers  (2026-09-02T08:40:07Z)
• Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks [0.0]
  We study the effects of different task specifications on agentic token spend with the Kimi K3 model at three thinking efforts.<n>We show that reducing a full task specification to a bare user story raises token spend by $29.7%$, while run-to-run variance remains unaffected by any prompt changes.
  arXiv  Detail & Related papers  (2026-08-26T05:58:12Z)
• More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning [54.65906330923846]
  We show that object-centric slot representations are a structured alternative to pre-trained vision models.<n>A dense patch grid with 16x the tokens performs no better than the global feature.<n>An explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7$pm$4.2%, just below a privileged 3D-oracle upper bound.
  arXiv  Detail & Related papers  (2026-07-10T10:35:24Z)
• Subtoken Vision Transformer for Fine-grained Recognition [21.033814568458897]
  SubViT is a selective image tokenization method for fine-grained visual recognition.<n>It represents discriminative patches with multiple subtokens while retaining the original token sequence for global context.<n>It improves the average novel-category accuracy of DINOv2 from $81.3%$ to $84.7%$, with only $0.50$ ms additional latency and $3.4%$ more FLOPs.
  arXiv  Detail & Related papers  (2026-07-10T04:07:23Z)
• Sequential Planning via Anchored Robotic Keypoints [3.176338968032152]
  We present Sequential Planning via Anchored Robotic Keypoints, SPARK, a training-free neurosymbolic manipulation system.<n>A single Gemini call composes the plan as a typed behavior tree (BT) of composable kinematic primitives.<n>A second Gemini call proposes three alternative text prompts per object, SAM3 evaluates each, and we keep the prompt$to$label pair with the most confident detection.<n>The alternative prompts add +27.7 points on the spatial suite and +10.0 on the object suite, with the recovery loop adding +5.0 overall.
  arXiv  Detail & Related papers  (2026-06-29T17:48:01Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• Learning Accurate Segmentation Purely from Self-Supervision [87.78965637247107]
  Selfment is a fully self-supervised framework that segments objects directly from raw images without human labels.<n>Selfment sets new state-of-the-art (SoTA) results across multiple benchmarks.
  arXiv  Detail & Related papers  (2026-02-27T07:36:32Z)
• PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective [59.24570811503256]
  We propose PIO-FVLM to reduce redundant visual tokens in vision-models (VLMs) to accelerate inference.<n>The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment.<n>On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance.
  arXiv  Detail & Related papers  (2026-02-04T15:33:10Z)
• H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers [124.11648300910444]
  We present a hierarchical plug-and-play pruning-and-$-recovering framework, called Hierarchical Hourglass Tokenizer (H$_2$OT)<n>Our method is general-purpose: it can be easily incorporated into common VPT models on both seq2seq and seq2frame pipelines.
  arXiv  Detail & Related papers  (2025-09-08T17:59:59Z)
• AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer [13.945118817568366]
  We introduce an anchor-based efficient vision transformer (AnchorFormer), which employs the anchor tokens to learn the pivotal information and accelerate the inference.<n>By representing the anchors with the neurons in a neural layer, we can differentiably learn these anchors and approximate global self-attention.<n>Experiments show the effectiveness of our AnchorFormer, achieving up to a 9.0% higher accuracy or 46.7% FLOPs reduction on ImageNet classification.
  arXiv  Detail & Related papers  (2025-05-22T09:44:44Z)
• Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens [65.4435926060951]
  We propose to significantly improve the efficiency of Transformers for ultra long sequences, by compressing the sequence into a much smaller representation at each layer. Our algorithm is not only efficient (achieving more than $3times$ efficiency gain compared to baselines on 4K and 16K lengths) but also offers competitive/better performance on a large number of tasks.
  arXiv  Detail & Related papers  (2023-05-07T10:32:18Z)
• Parameter-Efficient Transfer Learning with Diff Pruning [108.03864629388404]
  diff pruning is a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark.
  arXiv  Detail & Related papers  (2020-12-14T12:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.