LookThere! Sparse Vision by Reinforced Selection

• URL: http://arxiv.org/abs/2609.04698v1
• Date: Fri, 04 Sep 2026 04:00:43 GMT
• Title: LookThere! Sparse Vision by Reinforced Selection
• Abstract summary: LookThere is a framework that trains a shallow input selector and a deep representation extractor.<n>It excels at sparse recognition in high-resolution settings and maintains accuracy with as little as 0.2% of the input.<n>LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
• Score: 39.092804141582754
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.

Related papers

• Are All Tokens Necessary for Visual Place Recognition? An Empirical Study of Token Reduction for Efficient Inference [27.84020098999677]
  We present the first systematic benchmark of token reduction for efficient visual place recognition.<n>Our benchmark comprehensively evaluates representative token pruning, token merging, and hybrid pruning-merging methods.<n>Our work establishes a comprehensive foundation for future research on token-efficient VPR and efficient visual retrieval systems.
  arXiv  Detail & Related papers  (2026-07-17T02:13:19Z)
• TaskTok: Delving into Task Tokens for Task-driven Image Restoration [14.471297434908209]
  Task-Driven Image Restoration (TDIR) aims to maximize the performance of downstream high-level vision tasks.<n>Recent approaches leveraging generative priors have shown promise for TDIR.<n>We propose TaskTok, a novel framework that selectively restores only task-relevant tokens.
  arXiv  Detail & Related papers  (2026-06-25T05:20:01Z)
• Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers [58.582630781164305]
  Visual geometry transformers are powerful architectures for multi-view 3D reconstruction.<n>Their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models.<n>We address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention.<n>Our approach offers a superior speed-accuracy trade-off compared to existing solutions.
  arXiv  Detail & Related papers  (2026-05-22T17:55:13Z)
• ToDRE: Visual Token Pruning via Diversity and Task Awareness for Efficient Large Vision-Language Models [59.47738955960352]
  ToDRE is a two-stage and training-free token compression framework.<n>It achieves superior performance by pruning tokens based on token Diversity and token-task RElevance.
  arXiv  Detail & Related papers  (2025-05-24T15:47:49Z)
• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision [10.461453853510964]
  Vision transformers are ever larger, more accurate, and more expensive to compute.<n>We turn to adaptive computation to cope with this cost by learning to predict where to compute.<n>Our LookWhere method divides a low-resolution selector and a high-resolution extractor without ever processing the full high-resolution input.
  arXiv  Detail & Related papers  (2025-05-23T15:56:35Z)
• Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection [54.21851618853518]
  We present a concise yet effective approach called Patch Generation-to-Selection to enhance CLIP's training efficiency.<n>Our approach, CLIP-PGS, sets new state-of-the-art results in zero-shot classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-21T12:10:38Z)
• When Less is Enough: Adaptive Token Reduction for Efficient Image Representation [2.2120851074630177]
  We introduce a new method for determining feature utility based on the idea that less valuable features can be reconstructed from more valuable ones.<n>We implement this concept by integrating an autoencoder with a Gumbel-Softmax selection mechanism.<n>Our results highlight a promising direction towards adaptive and efficient multimodal pruning.
  arXiv  Detail & Related papers  (2025-03-20T19:17:08Z)
• Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers [5.825612611197359]
  Fine-grained recognition involves the classification of images from subordinate macro-categories. We propose a novel and computationally inexpensive metric to identify discriminative regions in an image. Our method achieves these results at a much lower computational cost compared to the alternatives.
  arXiv  Detail & Related papers  (2024-07-17T10:04:54Z)
• Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention [36.90363317158731]
  We propose an adaptive sparse token pruning framework with a minimal cost. Our method improves the throughput of DeiT-S by 50% and brings only 0.2% drop in top-1 accuracy.
  arXiv  Detail & Related papers  (2022-09-28T03:07:32Z)
• CloudAttention: Efficient Multi-Scale Attention Scheme For 3D Point Cloud Learning [81.85951026033787]
  We set transformers in this work and incorporate them into a hierarchical framework for shape classification and part and scene segmentation. We also compute efficient and dynamic global cross attentions by leveraging sampling and grouping at each iteration. The proposed hierarchical model achieves state-of-the-art shape classification in mean accuracy and yields results on par with the previous segmentation methods.
  arXiv  Detail & Related papers  (2022-07-31T21:39:15Z)
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition [52.36728157779307]
  We propose a unified distribution alignment strategy for long-tail visual recognition. We then introduce a generalized re-weight method in the two-stage learning to balance the class prior. Our approach achieves the state-of-the-art results across all four recognition tasks with a simple and unified framework.
  arXiv  Detail & Related papers  (2021-03-30T14:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.