UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios

• URL: http://arxiv.org/abs/2408.17267v2
• Date: Mon, 23 Dec 2024 07:25:51 GMT
• Title: UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios
• Authors: Baichuan Zhou, Haote Yang, Dairong Chen, Junyan Ye, Tianyi Bai, Jinhua Yu, Songyang Zhang, Dahua Lin, Conghui He, Weijia Li,
• Abstract summary: We present UrBench, a benchmark designed for evaluating LMMs in complex multi-view urban scenarios. UrBench contains 11.6K meticulously curated questions at both region-level and role-level. Our evaluations on 21 LMMs show that current LMMs struggle in the urban environments in several aspects.
• Score: 60.492736455572015
• License:
• Abstract: Recent evaluations of Large Multimodal Models (LMMs) have explored their capabilities in various domains, with only few benchmarks specifically focusing on urban environments. Moreover, existing urban benchmarks have been limited to evaluating LMMs with basic region-level urban tasks under singular views, leading to incomplete evaluations of LMMs' abilities in urban environments. To address these issues, we present UrBench, a comprehensive benchmark designed for evaluating LMMs in complex multi-view urban scenarios. UrBench contains 11.6K meticulously curated questions at both region-level and role-level that cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene Understanding, and Object Understanding, totaling 14 task types. In constructing UrBench, we utilize data from existing datasets and additionally collect data from 11 cities, creating new annotations using a cross-view detection-matching method. With these images and annotations, we then integrate LMM-based, rule-based, and human-based methods to construct large-scale high-quality questions. Our evaluations on 21 LMMs show that current LMMs struggle in the urban environments in several aspects. Even the best performing GPT-4o lags behind humans in most tasks, ranging from simple tasks such as counting to complex tasks such as orientation, localization and object attribute recognition, with an average performance gap of 17.4%. Our benchmark also reveals that LMMs exhibit inconsistent behaviors with different urban views, especially with respect to understanding cross-view relations. UrBench datasets and benchmark results will be publicly available at https://opendatalab.github.io/UrBench/.

Related papers

• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning [72.57452266982642]
  We introduce OCRBench v2, a large-scale bilingual text-centric benchmark for text recognition. We find that 20 out of 22 LMMs score below 50 (100 in total) and suffer from five-type limitations.
  arXiv  Detail & Related papers  (2024-12-31T07:32:35Z)
• Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models [10.828419851213528]
  We propose the Multi-Dimensional Insights benchmark, which includes over 500 images covering six common scenarios of human life. This design allows for a detailed assessment of LMMs' capabilities in meeting the preferences and needs of different age groups. Looking ahead, we anticipate that the MDI-Benchmark will open new pathways for aligning real-world personalization in LMMs.
  arXiv  Detail & Related papers  (2024-12-17T07:06:10Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• MMR: Evaluating Reading Ability of Large Multimodal Models [52.953316772123586]
  Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Current benchmarks fail to accurately reflect performance of different models. We propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding.
  arXiv  Detail & Related papers  (2024-08-26T19:26:50Z)
• CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks [10.22654338686634]
  Large language models (LLMs) with extensive general knowledge and powerful reasoning abilities have seen rapid development and widespread application. In this paper, we design CityBench, an interactive simulator based evaluation platform. We design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the CityBench.
  arXiv  Detail & Related papers  (2024-06-20T02:25:07Z)
• Exploring the Capabilities of Large Multimodal Models on Dense Text [58.82262549456294]
  We propose the DT-VQA dataset, with 170k question-answer pairs. In this paper, we conduct a comprehensive evaluation of GPT4V, Gemini, and various open-source LMMs. We find that even with automatically labeled training datasets, significant improvements in model performance can be achieved.
  arXiv  Detail & Related papers  (2024-05-09T07:47:25Z)
• Are We on the Right Way for Evaluating Large Vision-Language Models? [92.5761176224556]
  Large vision-language models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. We identify two primary issues: Visual content is unnecessary for many samples and intentional data leakage exists. We present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans.
  arXiv  Detail & Related papers  (2024-03-29T17:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.