The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design

• URL: http://arxiv.org/abs/2605.01345v3
• Date: Sat, 09 May 2026 06:29:21 GMT
• Title: The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design
• Abstract summary: High-resolution visual reasoning is not only semantic reasoning but also task-relevant evidence acquisition under limited perceptual bandwidth.<n>We formalise this process as sequential Bayesian optimal experimental design (S-BOED)<n>Experiments on high-resolution benchmarks show consistent gains over direct and ReAct-style baselines.
• Score: 15.20939188156227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual perception in modern Vision-Language Models (VLMs) is constrained by a perceptual bandwidth bottleneck: a broad field of view preserves global context but sacrifices the fine-grained details required for complex reasoning. We argue that high-resolution visual reasoning is therefore not only semantic reasoning but also task-relevant evidence acquisition under limited perceptual bandwidth. Inspired by active vision and information foraging, we formalise this process as sequential Bayesian optimal experimental design (S-BOED), where an agent decides which visual evidence to acquire before answering. Since exact Bayesian inference is intractable in continuous gigapixel spaces, we derive a tractable coverage--resolution objective as a proxy for task-relevant information gain. We instantiate this framework with FOVEA, a training-free procedure that refines VLM crop proposals through evidence-oriented probing. Experiments on high-resolution benchmarks show consistent gains over direct and ReAct-style baselines, with particularly strong improvements in search-dominated remote-sensing settings.

Related papers

• ViQ: Text-Aligned Visual Quantized Representations at Any Resolution [91.46185855575789]
  We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations.<n>Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization.<n>Experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders.
  arXiv  Detail & Related papers  (2026-06-25T17:29:27Z)
• CAVE: A Structured Credit Assignment Approach for Fragmented Visual Evidence Reasoning [13.024838183478858]
  We propose Credit Assignment for Visual Evidence (CAVE) for interleaved visual reasoning.<n>CAVE evaluates the contribution of intermediate steps at the action level via three complementary reasoning process signals.<n>We construct TRACER-Bench, which covers four nonlocal and semantically confusable reasoning dimensions.
  arXiv  Detail & Related papers  (2026-05-13T16:50:24Z)
• Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs [66.15429821530503]
  Persistent Visual Memory is a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence.<n>Experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead.<n>In-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.
  arXiv  Detail & Related papers  (2026-05-01T17:54:37Z)
• VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models [74.05545957865228]
  Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images and video frames in vision-language models (VLMs)<n>We introduce, a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations.<n>Our approach employs continuous relaxation and straight-through estimators to enable gradient-based search, solved via the Augmented Lagrangian method.
  arXiv  Detail & Related papers  (2026-04-16T16:21:05Z)
• Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects [26.144665202231014]
  Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance.<n>We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding.
  arXiv  Detail & Related papers  (2026-04-07T07:44:11Z)
• Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models [65.4947731385794]
  We present a unified multi-agent visual reasoning framework that evolves from our foundational image-centric model, Insight-V.<n>We introduce two novel algorithms, ST-GRPO and J-GRPO, which enhance spatial-temporal reasoning and improve evaluative robustness.<n>Experiments on base models like LLaVA-NeXT and Qwen2.5-VL demonstrate significant performance gains across challenging image and video reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-18T15:28:07Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• Function-Space Empirical Bayes Regularisation with Large Vision-Language Model Priors [12.285161219785294]
  VLM-FS-EB is a novel function-space empirical Bayes regularisation framework.<n>We show that our method consistently improves predictive performance and yields more reliable uncertainty estimates.
  arXiv  Detail & Related papers  (2026-02-03T05:24:11Z)
• Towards a Golden Classifier-Free Guidance Path via Foresight Fixed Point Iterations [12.366757123129402]
  We propose a unified perspective that reframes conditional guidance as fixed point iterations.<n>We introduce Foresight Guidance (FSG), which prioritizes solving longer-interval subproblems in early diffusion stages.<n>Our work offers novel perspectives for conditional guidance and unlocks the potential of adaptive design.
  arXiv  Detail & Related papers  (2025-10-24T14:39:07Z)
• Infusing fine-grained visual knowledge to Vision-Language Models [5.487134463783365]
  Large-scale contrastive pre-training produces powerful Vision-and-Language Models (VLMs)<n>We propose a fine-tuning method explicitly designed to achieve optimal balance between fine-grained domain adaptation and retention of the pretrained VLM's broad multimodal knowledge.<n>Our approach consistently achieves strong results, notably retaining the visual-text alignment without utilizing any text data or the original text encoder during fine-tuning.
  arXiv  Detail & Related papers  (2025-08-16T19:12:09Z)
• Interpretable Reward Modeling with Active Concept Bottlenecks [54.00085739303773]
  We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning.<n>Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts.<n>We formalize an active learning strategy that dynamically acquires the most informative concept labels.
  arXiv  Detail & Related papers  (2025-07-07T06:26:04Z)
• Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models [44.27801276966812]
  This paper pioneers a systematic exploration of Zeroth-Order (ZO) optimization for vision-language continual learning (VLCL)<n>We first identify the incompatibility of naive full-ZO adoption in VLCL due to modality-specific instability.<n>We develop a layer-wise optimization paradigm that interleaves ZO and FO across network layers, capitalizing on the heterogeneous learning dynamics of shallow versus deep representations.
  arXiv  Detail & Related papers  (2025-06-14T08:59:19Z)
• Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning [70.57180215148125]
  Visual instruction tuning aims to enable large language models to comprehend the visual world.<n>Existing methods often grapple with the intractable trade-off between accuracy and efficiency.<n>We present LLaVA-Meteor, a novel approach that strategically compresses visual tokens without compromising core information.
  arXiv  Detail & Related papers  (2025-05-17T10:22:29Z)
• LSSInst: Improving Geometric Modeling in LSS-Based BEV Perception with Instance Representation [10.434754671492723]
  We propose LSSInst, a two-stage object detector incorporating BEV and instance representations in tandem. The proposed detector exploits fine-grained pixel-level features that can be flexibly integrated into existing LSS-based BEV networks. Our proposed framework is of excellent generalization ability and performance, which boosts the performances of modern LSS-based BEV perception methods without bells and whistles.
  arXiv  Detail & Related papers  (2024-11-09T13:03:54Z)
• Towards Robust and Accurate Visual Prompting [11.918195429308035]
  We study whether a visual prompt derived from a robust model can inherit the robustness while suffering from the generalization performance decline. We introduce a novel technique named Prompt Boundary Loose (PBL) to effectively mitigates the suboptimal results of visual prompt on standard accuracy. Our findings are universal and demonstrate the significant benefits of our proposed method.
  arXiv  Detail & Related papers  (2023-11-18T07:00:56Z)
• Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning [19.28860833813788]
  Existing models commonly train a visual encoder with weak cross-modal supervision signals. We propose a novel textbfVisually-textbfAsymmetric cotextbfNsistentextbfCy textbfLearning (textscVancl) approach to capture fine-grained visual and layout features.
  arXiv  Detail & Related papers  (2023-10-23T10:37:22Z)
• Planning with Diffusion for Flexible Behavior Synthesis [125.24438991142573]
  We consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories.
  arXiv  Detail & Related papers  (2022-05-20T07:02:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.