UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark

• URL: http://arxiv.org/abs/2512.17196v1
• Date: Fri, 19 Dec 2025 03:20:59 GMT
• Title: UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark
• Abstract summary: UmniBench is a benchmark tailored for unified multimodal models with omni-dimensional evaluation.<n>It covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs.<n>Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models.
• Score: 46.85189386884927
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. However, evaluations of unified multimodal models (UMMs) remain decoupled, assessing their understanding and generation abilities separately with corresponding datasets. To address this, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. First, UmniBench can assess the understanding, generation, and editing ability within a single evaluation process. Based on human-examined prompts and QA pairs, UmniBench leverages UMM itself to evaluate its generation and editing ability with its understanding ability. This simple but effective paradigm allows comprehensive evaluation of UMMs. Second, UmniBench covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs. Moreover, UmniBench can also decouple and separately evaluate understanding, generation, and editing abilities, providing a fine-grained assessment. Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models. We hope this benchmark provides a more comprehensive and objective view of unified models and logistical support for improving the performance of the community model.

Related papers

• A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems [19.780193072520575]
  Current evaluation protocols treat generative and discriminative capabilities as separate tasks.<n>We propose Self-Generative-Understanding (SGU), which probes the integrated capabilities of unified models through a semantic closed-loop challenge.<n>This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems.
  arXiv  Detail & Related papers  (2026-08-12T10:35:52Z)
• Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation [80.99003754614365]
  We introduce Unison, a benchmark to evaluate joint understanding and generation in unified multimodal models.<n>Unison comprises 2,169 high-quality unified task samples.<n>We also introduce Unison-Judge, an evaluation model well aligned with human judgments.
  arXiv  Detail & Related papers  (2026-06-25T12:58:54Z)
• TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training [25.562880121429966]
  We present TorchUMM, the first unified framework for comprehensive evaluation, analysis, and post-training of unified multimodal models.<n>Our benchmark encompasses multimodal understanding, generation, and editing, and integrates both established and novel datasets.
  arXiv  Detail & Related papers  (2026-04-12T19:19:04Z)
• Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark [69.8473923357969]
  Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration.<n>We present Uni-MMMU, a comprehensive benchmark that unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains.
  arXiv  Detail & Related papers  (2025-10-15T17:10:35Z)
• UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation [15.938314589086914]
  We introduce UniEval, the first evaluation framework designed for unified multimodal models without extra models, images, or annotations.<n>The framework contains a holistic benchmark, UniBench, along with the corresponding UniScore metric.<n>We also extensively evaluated SoTA unified and visual generation models, uncovering new insights into Univeral's unique values.
  arXiv  Detail & Related papers  (2025-05-15T16:34:50Z)
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities [153.37868034779385]
  We propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks.<n>Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes.
  arXiv  Detail & Related papers  (2023-08-04T17:59:47Z)
• MMBench: Is Your Multi-modal Model an All-around Player? [114.45702807380415]
  We propose MMBench, a benchmark for assessing the multi-modal capabilities of vision-language models. MMBench is meticulously curated with well-designed quality control schemes. MMBench incorporates multiple-choice questions in both English and Chinese versions.
  arXiv  Detail & Related papers  (2023-07-12T16:23:09Z)
• Towards a Unified Multi-Dimensional Evaluator for Text Generation [101.47008809623202]
  We propose a unified multi-dimensional evaluator UniEval for Natural Language Generation (NLG) We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics.
  arXiv  Detail & Related papers  (2022-10-13T17:17:03Z)
• From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models [55.137869702763375]
  This paper explores a novel PLM reuse paradigm, Knowledge Integration (KI) KI aims to merge the knowledge from different teacher-PLMs, each of which specializes in a different classification problem, into a versatile student model. We then design a Model Uncertainty--aware Knowledge Integration (MUKI) framework to recover the golden supervision for the student.
  arXiv  Detail & Related papers  (2022-10-11T07:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.