Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation

• URL: http://arxiv.org/abs/2606.04046v1
• Date: Tue, 02 Jun 2026 07:50:56 GMT
• Title: Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
• Abstract summary: Vision-Language and Vision-Language-Action Models (VLMs & VLAs) are powerful tools with different benefits.<n>VLMs are better at long-term planning, while VLAs are better at reactive control.<n>Visual hallucinations arise due to the models' inability to distinguish task-relevant objects from distractors.<n>In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation.
• Score: 26.39892022038146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models (VLMs & VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual hallucinations arise due to the models' inability to distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation. A straightforward solution is one-step focus: directly attending to essential objects. However, this approach proves ineffective because effective focus inherently requires deep scene understanding. To this end, we propose SceneDiver, a coarse-to-fine focus plan generation method for VLMs leveraging their long-term planning abilities, that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through an iterative cycle of recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter for distilling the deliberate focus ability into VLAs. Evaluations on standard embodied AI benchmarks confirm that our method substantially reduces visual hallucinations for both VLMs and VLAs, while preserving computational efficiency in tasks requiring fast execution. Our code and data are released at: https://future-item.github.io/SceneDiver.

Related papers

• FocusVLA: Focused Visual Utilization for Vision-Language-Action Models [12.859683124954339]
  Vision-Language-Action (VLA) models improve action generation by conditioning policies on rich vision-language information.<n>We introduce FocusVLA, a novel paradigm that directs the model's attention to task-relevant visual regions to effectively bridge vision to action.
  arXiv  Detail & Related papers  (2026-03-30T17:50:54Z)
• Vision-Language Models Unlock Task-Centric Latent Actions [75.53481518882275]
  We propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations.<n>We show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld.
  arXiv  Detail & Related papers  (2026-01-30T08:38:59Z)
• V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs [66.81402538540458]
  We propose V-Attack, a novel method for precise local semantic attacks.<n>V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-11-25T11:51:17Z)
• VLA-Pruner: Temporal-Aware Dual-Level Visual Token Pruning for Efficient Vision-Language-Action Inference [17.901428758295307]
  Vision-Language-Action (VLA) models have shown great promise for embodied AI, yet the heavy computational cost limits their real-time deployment.<n>We propose VLA-Pruner, a versatile plug-and-play VLA-specific token prune method that aligns with the dual-system nature of VLA models.<n>VLA-Pruner achieves state-of-the-art performance across multiple VLA architectures and diverse robotic tasks.
  arXiv  Detail & Related papers  (2025-11-20T15:16:09Z)
• Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action models [8.452688845632995]
  We propose Oat-VLA, an Object-Agent-centric Tokenization for Vision-Language-Action (VLA) models.<n>We find that Oat-VLA can drastically reduce the number of visual tokens to just a few tokens without sacrificing performance.<n>We reveal that Oat-VLA converges at least twice as fast as OpenVLA on the LIBERO suite, as well as outperform OpenVLA in diverse real-world pick and place tasks.
  arXiv  Detail & Related papers  (2025-09-28T05:42:53Z)
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [95.89543460132413]
  Vision-language models (VLMs) have improved performance by increasing the number of visual tokens.<n>However, most real-world scenarios do not require such an extensive number of visual tokens.<n>We present a new paradigm for visual token compression, namely, VisionThink.
  arXiv  Detail & Related papers  (2025-07-17T17:59:55Z)
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models [89.44024245194315]
  We introduce a method that incorporates explicit visual chain-of-thought (CoT) reasoning into vision-language-action models (VLAs)<n>We introduce CoT-VLA, a state-of-the-art 7B VLA that can understand and generate visual and action tokens.<n>Our experimental results demonstrate that CoT-VLA achieves strong performance, outperforming the state-of-the-art VLA model by 17% in real-world manipulation tasks and 6% in simulation benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T22:23:04Z)
• Object-Centric Latent Action Learning [70.3173534658611]
  We propose a novel object-centric latent action learning framework that centers on objects rather than pixels.<n>We leverage self-supervised object-centric pretraining to disentangle action-related and distracting dynamics.<n>Our results show that object-centric pretraining mitigates the negative effects of distractors by 50%.
  arXiv  Detail & Related papers  (2025-02-13T11:27:05Z)
• COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training [49.2684130383925]
  We propose COSMOS: CrOSs-MOdality Self-distillation for vision-language pre-training.<n>It integrates a novel text-cropping strategy and cross-attention module into a self-supervised learning framework.<n>It consistently outperforms previous strong baselines on various zero-shot downstream tasks.
  arXiv  Detail & Related papers  (2024-12-02T18:56:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.