WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

• URL: http://arxiv.org/abs/2606.06538v1
• Date: Thu, 04 Jun 2026 01:11:21 GMT
• Title: WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
• Abstract summary: We build a taxonomy of thousands of visual concepts across multiple domains.<n>We manually design challenging questions that frontier MLLMs fail to answer.<n>WorldBench achieves higher visual diversity than any existing diverse benchmark.
• Score: 85.74548332724977
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. On quantitative and human evaluations, WorldBench achieves higher visual diversity than any existing diverse benchmark. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance-level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.

Related papers

• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues [54.31138496553705]
  DiCoBench is a comprehensive, multi-image high-resolution benchmark for fine-grained perception.<n>Our evaluation reveals a striking performance gap compared to human accuracy (98.3%)<n>We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.
  arXiv  Detail & Related papers  (2026-06-25T05:02:38Z)
• BrowseComp-$V^3$: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents [30.849897676091327]
  Multimodal large language models (MLLMs) are evolving into autonomous agents capable of performing multimodal web browsing and deep search in open-world environments.<n>We introduce BrowseComp-$V3$, a novel benchmark consisting of 300 carefully curated and challenging questions spanning diverse domains.<n>Our results highlight a fundamental gap between current model capabilities and robust multimodal deep search in real-world settings.
  arXiv  Detail & Related papers  (2026-02-13T12:25:13Z)
• VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priors [32.4515119002324]
  VisChainBench is a benchmark designed to rigorously evaluate Large Vision-Language Models (LVLMs)<n>It contains 1,457 tasks spanning over 20,000 images across three diverse domains (e.g., daily scenarios, engineering troubleshooting)<n>Uniquely, the benchmark is constructed using a multi-agent generation pipeline, ensuring high visual diversity and controlled language bias.
  arXiv  Detail & Related papers  (2025-12-07T09:48:10Z)
• MMTBENCH: A Unified Benchmark for Complex Multimodal Table Reasoning [40.95790862132066]
  MMTBENCH is a benchmark consisting of 500 real world multimodal tables drawn from diverse real world sources.<n>MMTBENCH questions cover four question types (Explicit, Implicit, Answer Mention, and Visual Based), five reasoning types (Mathematical, Extrema Identification, Fact Verification, Vision Based, and Others), and eight table types.
  arXiv  Detail & Related papers  (2025-05-27T21:09:11Z)
• Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models [70.41727912081463]
  Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images.<n>We propose a framework to equip MLLMs with robust multi-frame spatial understanding by integrating depth perception, visual correspondence, and dynamic perception.<n>Our model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable, generalizable multi-frame reasoning.
  arXiv  Detail & Related papers  (2025-05-22T17:59:39Z)
• EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents [63.43699771428243]
  EmbodiedBench is an extensive benchmark designed to evaluate vision-driven embodied agents.<n>We evaluated 24 leading proprietary and open-source MLLMs within EmbodiedBench.<n> MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only 28.9% on average.
  arXiv  Detail & Related papers  (2025-02-13T18:11:34Z)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs [61.143381152739046]
  We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach.<n>Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations.<n>We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes.
  arXiv  Detail & Related papers  (2024-06-24T17:59:42Z)
• Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning [15.296263261737026]
  We introduce a Multi-Image MIRB Benchmark to evaluate visual language models' ability to compare, analyze, and reason across multiple images. Our benchmark encompasses four categories: perception, visual world knowledge, reasoning, and multi-hop reasoning. We demonstrate that while open-source VLMs were shown to approach the GPT-4V in single-image tasks, a significant gap remains in multi-image reasoning tasks.
  arXiv  Detail & Related papers  (2024-06-18T16:02:18Z)
• MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks [56.60050181186531]
  We introduce MM-BigBench, which incorporates a diverse range of metrics to offer an extensive evaluation of the performance of various models and instructions. Our paper evaluates a total of 20 language models (14 MLLMs) on 14 multimodal datasets spanning 6 tasks, with 10 instructions for each task, and derives novel insights.
  arXiv  Detail & Related papers  (2023-10-13T11:57:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.