ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

• URL: http://arxiv.org/abs/2502.09696v2
• Date: Thu, 06 Mar 2025 17:45:33 GMT
• Title: ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
• Abstract summary: Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting images and, by some measures, have poorer spatial cognition than small children or animals.<n>We introduce ZeroBench, a lightweight visual reasoning benchmark that is entirely impossible for contemporary frontier LMMs.<n>We evaluate 20 LMMs on ZeroBench, all of which score 0.0%, and rigorously analyse the errors.
• Score: 79.90955995669158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting images and, by some measures, have poorer spatial cognition than small children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by an ongoing surge of model progress. To address this, there is a pressing need for difficult benchmarks that remain relevant for longer. We take this idea to its limit by introducing ZeroBench-a lightweight visual reasoning benchmark that is entirely impossible for contemporary frontier LMMs. Our benchmark consists of 100 manually curated questions and 334 less difficult subquestions. We evaluate 20 LMMs on ZeroBench, all of which score 0.0%, and rigorously analyse the errors. To encourage progress in visual understanding, we publicly release ZeroBench.

Related papers

• PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models [48.41009082629578]
  We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs)<n>By diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities.<n> Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge.
  arXiv  Detail & Related papers  (2026-07-27T18:04:54Z)
• OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents [34.58999637034811]
  Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning.<n>To address this limitation, OmniMapBench is introduced to foster visual-centric reasoning for map documents.<n>The benchmark comprises 2,096 manually annotated question-answer pairs across 1,603 map documents from nine categories.
  arXiv  Detail & Related papers  (2026-07-10T03:27:25Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models [5.6525926183880255]
  We introduce TurnBench, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task.<n>In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds.<n>TurnBench includes two modes: Classic, which tests standard reasoning, and Nightmare, which introduces increased complexity and requires robust inferential chains.
  arXiv  Detail & Related papers  (2025-06-02T05:47:50Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning [72.57452266982642]
  OCRBench v2 is a large-scale bilingual text-centric benchmark.<n>It covers 31 diverse scenarios, 10,000 human-verified question-answering pairs, and thorough evaluation metrics.<n>We find that most LMMs score below 50 (100 in total) and suffer from five-type limitations.
  arXiv  Detail & Related papers  (2024-12-31T07:32:35Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs) MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts. It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions? [86.94616033250068]
  R-Bench is a benchmark focused on the **Real-world Robustness of LMMs**. We show that while LMMs can correctly handle the original reference images, their performance is not stable when faced with distorted images. We hope that R-Bench will inspire improving the robustness of LMMs, **extending them from experimental simulations to the real-world application**.
  arXiv  Detail & Related papers  (2024-10-07T20:12:08Z)
• Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark [63.296342841358815]
  Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images.<n>The ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering.<n>We introduce MIRAGE, an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU.
  arXiv  Detail & Related papers  (2024-07-18T17:59:30Z)
• LiveBench: A Challenging, Contamination-Limited LLM Benchmark [93.57775429120488]
  We release LiveBench, the first benchmark that contains frequently-updated questions from recent information sources. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time.
  arXiv  Detail & Related papers  (2024-06-27T16:47:42Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Efficient Lifelong Model Evaluation in an Era of Rapid Progress [40.57576540258748]
  We introduce Sort & Search (S&S), which reuses previously evaluated models by leveraging dynamic programming algorithms to selectively rank and sub-select test samples. S&S achieves highly-efficient approximate accuracy measurement, reducing compute cost from 180 GPU days to 5 GPU hours on a single A100 GPU, with low approximation error and memory cost of 100MB. Our work highlights issues with current accuracy prediction metrics, suggesting a need to move towards sample-level evaluation metrics.
  arXiv  Detail & Related papers  (2024-02-29T18:58:26Z)
• Benchmarking Large Multimodal Models against Common Corruptions [45.26424202601339]
  This technical report aims to fill a deficiency in the assessment of large multimodal models (LMMs) We investigate the cross-modal interactions between text, image, and speech, encompassing four essential generation tasks. We create a benchmark, named MMCBench, that covers more than 100 popular LMMs.
  arXiv  Detail & Related papers  (2024-01-22T13:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.