Do multimodal models imagine electric sheep?

• URL: http://arxiv.org/abs/2605.09693v1
• Date: Sun, 10 May 2026 18:25:52 GMT
• Title: Do multimodal models imagine electric sheep?
• Abstract summary: We fine-tune a Qwen3.5 VLM to solve 12 diverse visual reasoning tasks.<n>We show that the model's activations after each action encode meaningful visual information about the intermediate state.<n>We find that integrating as few as sixteen visual tokens per step improves the average solve rate from 83% to 89%.
• Score: 99.83000217195644
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Yes. We find that large multimodal models develop mental imagery when solving spatial puzzles, and they do imagine sheep when solving sheep puzzles. We fine-tune a Qwen3.5 VLM to solve twelve diverse visual reasoning tasks -- including tangram, jigsaw, sokoban, 3D mental rotation, and rush hour -- that require understanding geometry, spatial relationships, and the consequences of actions. By supervising the model to predict the open-loop sequence of actions to solve a puzzle from an initial state, we show that the model's activations after each action encode meaningful visual information about the intermediate state. This finding suggests that an imperfect visual world model begins to form as a byproduct of learning to select correct actions, in the absence of any explicit visual supervision. Building on this observation, we propose two ways to sharpen and use the mental images formed by the model. We find that integrating as few as sixteen visual tokens per step into the chain of thought improves the average solve rate from 83% to 89%, with particularly strong gains on reasoning-heavy tasks such as jigsaw and 3D mental rotation.

Related papers

• Thinker: A vision-language foundation model for embodied intelligence [9.661713829767605]
  We propose Thinker, a large vision-language foundation model for embodied intelligence.<n>We construct a large-scale dataset tailored for robotic perception and reasoning.<n>We introduce a simple yet effective approach that substantially enhances the model's capacity for video comprehension.
  arXiv  Detail & Related papers  (2026-01-29T02:52:08Z)
• Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search [85.201906907271]
  Mini-o3 is a system that executes deep, multi-turn reasoning spanning tens of steps.<n>Our recipe for reproducing OpenAI o3-style behaviors comprises three key components.<n>Extensive experiments demonstrate that Mini-o3 produces rich reasoning patterns and deep thinking paths.
  arXiv  Detail & Related papers  (2025-09-09T17:54:21Z)
• Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning [109.84783476090028]
  We introduce $textbfZebra-CoT$, a diverse large-scale dataset with 182,384 samples.<n>We focus on four categories of tasks where sketching or visual reasoning is especially natural.<n>Fine-tuning Bagel-7B yields a model that generates high-quality interleaved visual reasoning chains.
  arXiv  Detail & Related papers  (2025-07-22T16:35:36Z)
• MiCo: Multi-image Contrast for Reinforcement Visual Reasoning [72.81576836419373]
  Chain-of-Thought (CoT) reasoning can be used to link visual cues across multiple images.<n>We adapt rule-based reinforcement learning for Vision-Language Models (VLMs)<n>Our method achieves significant improvements on multi-image reasoning benchmarks and shows strong performance on general vision tasks.
  arXiv  Detail & Related papers  (2025-06-27T17:59:27Z)
• Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations [61.235500325327585]
  Existing AI benchmarks primarily assess verbal reasoning, neglecting the complexities of non-verbal, multi-step visual simulation.<n>We introduce STARE, a benchmark designed to rigorously evaluate multimodal large language models on tasks better solved through visual simulation.<n>Our evaluations show that models excel at reasoning over simpler 2D transformations, but perform close to random chance on more complex tasks.
  arXiv  Detail & Related papers  (2025-06-05T05:09:46Z)
• PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns [69.17409440805498]
  We evaluate large multimodal models with abstract patterns based on fundamental concepts. We find that they are not able to generalize well to simple abstract patterns. Our systematic analysis finds that the main bottlenecks of GPT-4V are weaker visual perception and inductive reasoning abilities.
  arXiv  Detail & Related papers  (2024-03-20T05:37:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.