Advancing Creative Physical Intelligence in Large Multimodal Models

• URL: http://arxiv.org/abs/2605.26396v2
• Date: Fri, 29 May 2026 07:01:08 GMT
• Title: Advancing Creative Physical Intelligence in Large Multimodal Models
• Abstract summary: We introduce MM-CreativityBench, a benchmark for affordance-grounded creative tool use in visually rich, physically constrained environments.<n>Our experiments show that current LMMs often fall short, not due to lack of generative capability, but because they do not sustain grounded exploration.<n>Motivated by this failure mode, we propose affordance-grounded alignment, which casts creative tool use as a preference learning problem.
• Score: 62.56522271769017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large multimodal models (LMMs) have rapidly advanced in perception and reasoning; however, it remains unclear whether these capabilities generalize to discovering visually grounded solutions in open-ended environments, beyond pattern recognition. In such settings, intelligence requires more than answering well-posed questions: it involves identifying how elements in a scene can be repurposed in non-obvious yet physically feasible ways. This form of creative problem-solving is central to human intelligence, but remains largely untested in current benchmarks. To evaluate this ability, we introduce MM-CreativityBench, a benchmark for affordance-grounded creative tool use in visually rich, physically constrained environments. Each instance presents a scenario image with structured views of candidate entities and their parts, enabling fine-grained, interactive evaluation of how models iteratively inspect the scene, identify relevant affordances, and compose visually and physically grounded solutions. Our experiments show that current LMMs often fall short, not due to lack of generative capability, but because they do not sustain grounded exploration. Models often overlook relevant entities, under-examine critical parts, or hallucinate attributes not grounded in the image. Motivated by this failure mode, we propose affordance-grounded alignment, which casts creative tool use as a preference learning problem. Using Direct Preference Optimization, we encourage models to prefer attribute-affordance reasoning grounded in visual evidence over hallucinated alternatives. In addition, we incorporate supervision derived from an affordance knowledge base to guide broader entity exploration and multi-turn planning. Our results show consistent gains in selecting the correct entities and parts, while substantially reducing hallucination and grounding-related errors.

Related papers

• MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts [10.608824155356377]
  We present MissingBench-Verified, a benchmark designed to evaluate a specific and practically relevant scenario.<n>We observe consistent and significant failure rates that persist even when external tool evidence explicitly contradicts the model's visual perception.<n>We find that existing mitigation strategies, including tool-assisted verification, autonomous visual reasoning, longer reasoning durations, and fine-tuning on an easier dataset, provide negligible improvement.
  arXiv  Detail & Related papers  (2026-07-21T03:43:50Z)
• POINTS-Seeker: Towards Training a Multimodal Agentic Search Model from Scratch [84.73366911912512]
  We introduce Agentic Seeding, a dedicated phase designed to weave the precursors necessary for eliciting agentic behaviors.<n>We propose V-Fold, an adaptive history-aware compression scheme that preserves recent dialogue turns in high fidelity while folding historical context into the visual space via rendering.<n>We develop POINTS-Seeker-8B, a state-of-the-art multimodal agentic search model that consistently outperforms existing models across six diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-15T16:09:37Z)
• CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing [67.2958108309687]
  CreativityBench is a benchmark for evaluating affordance-based creativity in LLMs.<n>We generate 14K grounded tasks that require identifying non-obvious yet physically plausible solutions under constraints.<n> Evaluations across 10 state-of-the-art LLMs, including closed and open-source models, show that models can often select a plausible object, but fail to identify the correct parts.
  arXiv  Detail & Related papers  (2026-04-06T16:33:23Z)
• DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories [52.57197752244638]
  We introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task.<n>Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues.<n>We construct DISBench, a challenging benchmark built on interconnected visual data.
  arXiv  Detail & Related papers  (2026-02-11T12:51:10Z)
• MentisOculi: Revealing the Limits of Reasoning with Mental Imagery [63.285794947638614]
  We develop MentisOculi, a suite of multi-step reasoning problems amenable to visual solution.<n> evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance.<n>Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning.
  arXiv  Detail & Related papers  (2026-02-02T18:49:06Z)
• Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies [40.03295633717008]
  We introduce VIA-Bench, a benchmark designed to probe model performance on visual illusions and anomalies.<n>We construct over 1K high-quality question-answer pairs that require nuanced visual reasoning.<n>Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence.
  arXiv  Detail & Related papers  (2026-02-02T08:48:03Z)
• Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection [71.8243083897721]
  Vision-language models often hallucinate details, generating non-existent objects or inaccurate attributes that compromise output reliability.<n>We present a novel framework that leverages the model's self-consistency between long responses and short answers to generate preference pairs for training.
  arXiv  Detail & Related papers  (2025-09-27T10:37:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.