On Locality and Length Generalization in Visual Reasoning

• URL: http://arxiv.org/abs/2607.09061v1
• Date: Fri, 10 Jul 2026 03:08:31 GMT
• Title: On Locality and Length Generalization in Visual Reasoning
• Abstract summary: We study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image.<n>Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.
• Score: 9.795082945107277
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes human vision distinctly different from most popular computer vision models in use today, which input images globally and in a single shot. A natural question therefore is whether local, sequential vision models may provide any fundamental computational benefits in addition to being biologically more plausible than global models. In this work, we investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies of length generalization in language models, we study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity. We also show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks. Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.

Related papers

• Where To Look? : Causal Tracing of Vision Encoders in VLM [0.6980863185266349]
  Vision-language models can describe an image with remarkable accuracy, yet what visual information actually drives their answers?<n>We investigate this question through causal tracing, and we observe that highly causal vision tokens often lie outside the target region.<n>We further investigate: can these models preserve visual structure when appearance cues are removed? and find that visual cues are exploited to understand visual structures.
  arXiv  Detail & Related papers  (2026-08-11T10:17:34Z)
• Trimming the Long-Tail of Visual World Modeling Evaluation [67.17191772079316]
  We introduce Tailor-Bench, a benchmark that challenges world models to simulate irregular physical interactions.<n>Regular scenarios reflect common tool-task pairs, Unconventional scenarios replace conventional tools with attribute-compatible substitutes, and Impossible scenarios introduce attribute-violating tools.<n>Our experimental results reveal a clear long-tail gap in physical world modeling, indicating limited generalization beyond common interactions.
  arXiv  Detail & Related papers  (2026-06-23T07:43:41Z)
• Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models [60.543714835980325]
  Humans construct internal world models and reason by manipulating the concepts within these models.<n>Recent advances in AI approximate such human cognitive abilities, where world models are believed to be embedded within large language models.<n>This paper presents the first principled study of when and how visual generation benefits reasoning.
  arXiv  Detail & Related papers  (2026-01-27T17:40:07Z)
• TopoPerception: A Shortcut-Free Evaluation of Global Visual Perception in Large Vision-Language Models [11.7808701773328]
  Large Vision-Language Models (LVLMs) typically align visual features from an encoder with a pre-trained Large Language Model (LLM)<n>Here, we introduce TopoPerception, a benchmark that leverages topological properties to rigorously evaluate the global visual perception capabilities of LVLMs.<n>We evaluate state-of-the-art models on TopoPerception and find that even at the coarsest perceptual granularity, all models perform no better than random chance.
  arXiv  Detail & Related papers  (2025-11-14T19:45:56Z)
• Visual Bridge: Universal Visual Perception Representations Generating [27.034175361589572]
  We propose a universal visual perception framework based on flow matching that can generate diverse visual representations across multiple tasks.<n>Our approach formulates the process as a universal flow-matching problem from image patch tokens to task-specific representations.<n>Our model achieves competitive performance in both zero-shot and fine-tuned settings, outperforming prior generalist and several specialist models.
  arXiv  Detail & Related papers  (2025-11-11T06:25:30Z)
• Grounded Reinforcement Learning for Visual Reasoning [51.94871616778874]
  We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with reinforcement learning.<n>Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces.<n>Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.
  arXiv  Detail & Related papers  (2025-05-29T17:20:26Z)
• Chain-of-Sketch: Enabling Global Visual Reasoning [37.09634351533441]
  We introduce an expanded set of global visual datasets involving graphs, strings, mazes, and image grids.<n>We show that large vision models still struggle to learn these tasks efficiently.<n>We propose a method called chain-of-sketch (CoS) to mitigate this learning inefficiency.
  arXiv  Detail & Related papers  (2024-10-10T17:44:13Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• What does CLIP know about a red circle? Visual prompt engineering for VLMs [116.8806079598019]
  We explore the idea of visual prompt engineering for solving computer vision tasks beyond classification by editing in image space instead of text. We show the power of this simple approach by achieving state-of-the-art in zero-shot referring expressions comprehension and strong performance in keypoint localization tasks.
  arXiv  Detail & Related papers  (2023-04-13T17:58:08Z)
• Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models [57.08925810659545]
  We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models. Our empirical observations suggest that vision-and-language models are better at label prediction tasks. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
  arXiv  Detail & Related papers  (2022-12-01T05:00:18Z)
• Does language help generalization in vision models? [0.0]
  We show that a visual model trained on a very large supervised image dataset (ImageNet-21k) can be as efficient for generalization as its multimodal counterpart (CLIP) When compared to other standard visual or language models, the latent representations of BiT-M were found to be just as "linguistic" as those of CLIP.
  arXiv  Detail & Related papers  (2021-04-16T18:54:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.