PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2607.24957v1
• Date: Mon, 27 Jul 2026 18:04:54 GMT
• Title: PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
• Abstract summary: We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs)<n>By diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities.<n> Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge.
• Score: 48.41009082629578
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60\% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.

Related papers

• MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs [16.970480774847857]
  We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts.<n>We propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence.<n>It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises.
  arXiv  Detail & Related papers  (2026-08-29T14:19:10Z)
• KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation [9.739392718456843]
  Knowledge-related failures are often investigated separately, lacking a unified evaluation framework across different hallucination dimensions.<n>We propose KnowHal, a benchmark that explicitly incorporates knowledge hallucination into multimodal hallucination evaluation spanning four dimensions.<n>KnowHal constructs paired positive and negative questions over shared images and entities, enabling controlled comparisons among perceptual errors, knowledge-related errors, and false-premise acceptance.
  arXiv  Detail & Related papers  (2026-08-04T15:03:29Z)
• DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues [54.31138496553705]
  DiCoBench is a comprehensive, multi-image high-resolution benchmark for fine-grained perception.<n>Our evaluation reveals a striking performance gap compared to human accuracy (98.3%)<n>We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.
  arXiv  Detail & Related papers  (2026-06-25T05:02:38Z)
• LTD-Bench: Evaluating Large Language Models by Letting Them Draw [57.237152905238084]
  LTD-Bench is a breakthrough benchmark for large language models (LLMs)<n>It transforms LLM evaluation from abstract scores to directly observable visual outputs by requiring models to generate drawings through dot matrices or executable code.<n> LTD-Bench's visual outputs enable powerful diagnostic analysis, offering a potential approach to investigate model similarity.
  arXiv  Detail & Related papers  (2025-11-04T08:11:23Z)
• PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection [33.98107848014898]
  We introduce PRISM-Bench, a benchmark of puzzle-based visual challenges designed to evaluate multimodal large language models (MLLMs)<n>Unlike prior evaluations that measure only final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual puzzle and a step-by-step chain-of-thought containing exactly one error, models must identify the first incorrect step.<n>The puzzles in PRISM-Bench require multi-step symbolic, geometric, and analogical reasoning, resisting shortcuts based on superficial pattern matching.
  arXiv  Detail & Related papers  (2025-10-27T17:57:52Z)
• Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models [17.922450921582794]
  Occlusion perception is a critical foundation for human-level spatial understanding.<n>We introduce O-Bench, the first visual question answering (VQA) benchmark specifically designed for occlusion perception.
  arXiv  Detail & Related papers  (2025-08-06T03:39:21Z)
• SpatialViz-Bench: An MLLM Benchmark for Spatial Visualization [44.427830927596204]
  SpatialViz-Bench is a comprehensive benchmark for spatial visualization with 12 tasks across 4 sub-abilities, comprising 1,180 automatically generated problems.<n>Our evaluation of 33 state-of-the-art MLLMs reveals wide performance variations and uncovers counter-intuitive findings.
  arXiv  Detail & Related papers  (2025-07-10T10:27:20Z)
• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models [121.03333569013148]
  We introduce VisuLogic: a benchmark of 1,000 human-verified problems across six categories.<n>These types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives.<n>Most models score below 30% accuracy-only slightly above the 25% random baseline and far below the 51.4% achieved by humans.
  arXiv  Detail & Related papers  (2025-04-21T17:59:53Z)
• AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs [70.4578433679737]
  We introduce Audio-Visual Trustworthiness assessment Benchmark (AVTrustBench), comprising 600K samples spanning over 9 meticulously crafted tasks.<n>Using our benchmark we extensively evaluate 13 state-of-the-art AVLLMs.<n>The findings reveal that the majority of existing models fall significantly short of achieving human-like comprehension.
  arXiv  Detail & Related papers  (2025-01-03T23:03:24Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.