VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2511.11438v1
• Date: Fri, 14 Nov 2025 16:06:25 GMT
• Title: VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
• Abstract summary: We introduce VP-Bench, a benchmark for assessing MLLMs' capability in visual prompting and utilization.<n>VP-Bench employs a two-stage evaluation framework: Stage 1 examines models' ability to perceive VPs in natural scenes, using 30k attribute prompts spanning eight shapes and 355 combinations.<n> Stage 2 investigates the impact of VPs on downstream tasks, measuring their effectiveness in real-world problem-solving scenarios.
• Score: 62.486978131415924
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multimodal large language models (MLLMs) have enabled a wide range of advanced vision-language applications, including fine-grained object recognition and contextual understanding. When querying specific regions or objects in an image, human users naturally use "visual prompts" (VPs), such as bounding boxes, to provide reference. However, no existing benchmark systematically evaluates the ability of MLLMs to interpret such VPs. This gap leaves it unclear whether current MLLMs can effectively recognize VPs, an intuitive prompting method for humans, and use them to solve problems. To address this limitation, we introduce VP-Bench, a benchmark for assessing MLLMs' capability in VP perception and utilization. VP-Bench employs a two-stage evaluation framework: Stage 1 examines models' ability to perceive VPs in natural scenes, using 30k visualized prompts spanning eight shapes and 355 attribute combinations. Stage 2 investigates the impact of VPs on downstream tasks, measuring their effectiveness in real-world problem-solving scenarios. Using VP-Bench, we evaluate 28 MLLMs, including proprietary systems (e.g., GPT-4o) and open-source models (e.g., InternVL3 and Qwen2.5-VL), and provide a comprehensive analysis of factors that affect VP understanding, such as variations in VP attributes, question arrangement, and model scale. VP-Bench establishes a new reference framework for studying how MLLMs comprehend and resolve grounded referring questions.

Related papers

• EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models [39.77362541405281]
  EPIC-Bench is a grounding benchmark for large vision-language models (VLMs)<n>It systematically evaluates the visual perceptual capabilities of VLMs in real-world embodied environments.
  arXiv  Detail & Related papers  (2026-05-16T16:38:51Z)
• Sign Language Recognition in the Age of LLMs [0.2609784101826761]
  Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks.<n>We investigate the capability of modern VLMs to perform isolated sign language recognition in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-04-13T09:26:16Z)
• How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding [39.342366994703376]
  We introduce a probing framework to analyze how MLLMs process visual and textual inputs across layers.<n>We show that while the overall stage-wise structure remains stable across variations in visual tokenization, instruction tuning data, and pretraining corpus, the specific layer allocation to each stage shifts.
  arXiv  Detail & Related papers  (2025-08-27T21:22:01Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs [83.24033574914425]
  We present Prism, an innovative framework designed to disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: a perception stage that utilizes a VLM to extract and articulate visual information in textual form, and a reasoning stage that formulates responses based on the extracted visual information. Our analytical framework provides several valuable insights, underscoring Prism's potential as a cost-effective solution for vision-language tasks.
  arXiv  Detail & Related papers  (2024-06-20T17:54:03Z)
• AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models [34.843603169616486]
  We introduce AlignMMBench, a benchmark for evaluating the alignment capabilities of large Vision-Language Models (VLMs)<n>This benchmark is meticulously curated from real-world scenarios and internet sources, and includes both single-turn and multi-turn dialogue scenarios.<n>We also develop CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability.
  arXiv  Detail & Related papers  (2024-06-13T16:30:14Z)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models [73.40350756742231]
  Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored.
  arXiv  Detail & Related papers  (2024-02-12T18:21:14Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting [83.21164539349273]
  Pre-trained language models (PLMs) have played an increasing role in multimedia research. In this paper, we focus on exploring PLMs as a stand-alone model for vision-language reasoning tasks. We propose a novel transfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP)
  arXiv  Detail & Related papers  (2023-06-01T07:19:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.