MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding

• URL: http://arxiv.org/abs/2503.09348v1
• Date: Wed, 12 Mar 2025 12:49:31 GMT
• Title: MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding
• Authors: Zhoutong Ye, Mingze Sun, Huan-ang Gao, Chun Yu, Yuanchun Shi,
• Abstract summary: Large multimodal models (LMMs) have demonstrated significant potential as generalists in vision-language (VL) tasks.<n>There remains a significant gap between state-of-the-art LMMs and human performance.<n>We propose MOAT, a benchmark with complex real-world VL tasks that are challenging for LMMs.
• Score: 27.140576967695413
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large multimodal models (LMMs) have demonstrated significant potential as generalists in vision-language (VL) tasks. However, there remains a significant gap between state-of-the-art LMMs and human performance when it comes to complex tasks that require a combination of fundamental VL capabilities, as well as tasks involving the grounding of complex instructions. To thoroughly investigate the human-LMM gap and its underlying causes, we propose MOAT, a diverse benchmark with complex real-world VL tasks that are challenging for LMMs. Specifically, the tasks in MOAT require LMMs to engage in generalist problem solving by integrating fundamental VL capabilities such as reading text, counting, understanding spatial relations, grounding textual and visual instructions, etc. All these abilities fit into a taxonomy proposed by us that contains 10 fundamental VL capabilities, enabling MOAT to provide a fine-grained view of LMMs' strengths and weaknesses. Besides, MOAT is the first benchmark to explicitly evaluate LMMs' ability to ground complex text and visual instructions, which is essential to many real-world applications. We evaluate over 20 proprietary and open source LMMs, as well as humans, on MOAT, and found that humans achieved 82.7% accuracy while the best performing LMM (OpenAI o1) achieved only 38.8%. To guide future model development, we analyze common trends in our results and discuss the underlying causes of observed performance gaps between LMMs and humans, focusing on which VL capability forms the bottleneck in complex tasks, whether test time scaling improves performance on MOAT, and how tiling harms LMMs' capability to count. Code and data are available at https://cambrian-yzt.github.io/MOAT.

Related papers

• HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks [25.959032350818795]
  We present HumanEval-V, a benchmark of human-annotated coding tasks.<n>Each task features carefully crafted diagrams paired with function signatures and test cases.<n>We find that even top-performing models achieve modest success rates.
  arXiv  Detail & Related papers  (2024-10-16T09:04:57Z)
• LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models [55.903148392998965]
  We introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to detect synthetic data across multiple modalities. The benchmark includes coarse-grained judgment and multiple-choice questions, as well as fine-grained anomaly selection and explanation tasks. We evaluate 22 open-source LMMs and 6 closed-source models on LOKI, highlighting their potential as synthetic data detectors and also revealing some limitations in the development of LMM capabilities.
  arXiv  Detail & Related papers  (2024-10-13T05:26:36Z)
• F-LMM: Grounding Frozen Large Multimodal Models [53.8059045627934]
  We present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations. Using only a few trainable CNN layers, we can translate word-pixel attention weights to mask logits. Our F-LMM neither learns special segmentation tokens nor utilises high-quality grounded instruction-tuning data.
  arXiv  Detail & Related papers  (2024-06-09T15:14:26Z)
• Exploring the Capabilities of Large Multimodal Models on Dense Text [58.82262549456294]
  We propose the DT-VQA dataset, with 170k question-answer pairs. In this paper, we conduct a comprehensive evaluation of GPT4V, Gemini, and various open-source LMMs. We find that even with automatically labeled training datasets, significant improvements in model performance can be achieved.
  arXiv  Detail & Related papers  (2024-05-09T07:47:25Z)
• Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Models [84.78457918843165]
  Unsolvable Problem Detection (UPD) is a novel task to evaluate the robust understanding capability of Large Multimodal Models (LMMs) UPD assesses the LMM's ability to withhold answers when encountering unsolvable problems of multiple-choice question answering. Our experiments reveal that even most LMMs, which demonstrate adequate performance on existing benchmarks, struggle significantly with MM-UPD.
  arXiv  Detail & Related papers  (2024-03-29T17:59:53Z)
• Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models [87.47400128150032]
  We propose a novel LMM architecture named Lumen, a Large multimodal model with versatile vision-centric capability enhancement. Lumen first promotes fine-grained vision-language concept alignment. Then the task-specific decoding is carried out by flexibly routing the shared representation to lightweight task decoders.
  arXiv  Detail & Related papers  (2024-03-12T04:13:45Z)
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities [153.37868034779385]
  We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks.<n>Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes.
  arXiv  Detail & Related papers  (2023-08-04T17:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.