Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

• URL: http://arxiv.org/abs/2606.26984v1
• Date: Thu, 25 Jun 2026 12:58:54 GMT
• Title: Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
• Abstract summary: We introduce Unison, a benchmark to evaluate joint understanding and generation in unified multimodal models.<n>Unison comprises 2,169 high-quality unified task samples.<n>We also introduce Unison-Judge, an evaluation model well aligned with human judgments.
• Score: 80.99003754614365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.

Related papers

• A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems [19.780193072520575]
  Current evaluation protocols treat generative and discriminative capabilities as separate tasks.<n>We propose Self-Generative-Understanding (SGU), which probes the integrated capabilities of unified models through a semantic closed-loop challenge.<n>This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems.
  arXiv  Detail & Related papers  (2026-08-12T10:35:52Z)
• UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? [50.92401586025528]
  Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear.<n>We introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks.
  arXiv  Detail & Related papers  (2026-03-03T18:36:16Z)
• GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks? [29.804627410258732]
  We introduce a unified evaluation framework that leverages a pairwise comparison paradigm to ensure stable and human-aligned evaluation.<n>Our method boosts evaluation accuracy by over 20% and achieves a Spearman correlation of 0.86 with the authoritative LMArena leaderboard.
  arXiv  Detail & Related papers  (2026-02-05T18:52:48Z)
• Quantifying the Gap between Understanding and Generation within Unified Multimodal Models [66.07644743841007]
  GapEval is a benchmark designed to quantify the gap between understanding and generation capabilities.<n>Experiments reveal a persistent gap between the two directions across a wide range of UMMs.<n>Our findings indicate that knowledge within UMMs often remains disjoint.
  arXiv  Detail & Related papers  (2026-02-02T14:19:37Z)
• Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark [69.8473923357969]
  Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration.<n>We present Uni-MMMU, a comprehensive benchmark that unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains.
  arXiv  Detail & Related papers  (2025-10-15T17:10:35Z)
• RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark [71.3555284685426]
  We introduce RealUnify, a benchmark designed to evaluate bidirectional capability synergy.<n>RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks.<n>We find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient.
  arXiv  Detail & Related papers  (2025-09-29T15:07:28Z)
• Exploring the Trade-off between Plausibility, Change Intensity and Adversarial Power in Counterfactual Explanations using Multi-objective Optimization [73.89239820192894]
  We argue that automated counterfactual generation should regard several aspects of the produced adversarial instances. We present a novel framework for the generation of counterfactual examples.
  arXiv  Detail & Related papers  (2022-05-20T15:02:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.