A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems

• URL: http://arxiv.org/abs/2608.11907v2
• Date: Thu, 20 Aug 2026 16:45:02 GMT
• Title: A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems
• Abstract summary: Current evaluation protocols treat generative and discriminative capabilities as separate tasks.<n>We propose Self-Generative-Understanding (SGU), which probes the integrated capabilities of unified models through a semantic closed-loop challenge.<n>This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems.
• Score: 19.780193072520575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs). In this work, we propose Self-Generative-Understanding (SGU), a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. Without requiring new annotations, SGU leverages the dual understanding-and-generation abilities of UMMs by asking them to first perceive an image and produce a textual description, subsequently reconstruct a visual context based on that description, and finally perform reasoning over the self-generated output. This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems. Extensive experiments show that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone. Our work provides a complementary holistic evaluation framework and offers a foundation for benchmarking the development of next-generation unified multimodal models.

Related papers

• Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation [80.99003754614365]
  We introduce Unison, a benchmark to evaluate joint understanding and generation in unified multimodal models.<n>Unison comprises 2,169 high-quality unified task samples.<n>We also introduce Unison-Judge, an evaluation model well aligned with human judgments.
  arXiv  Detail & Related papers  (2026-06-25T12:58:54Z)
• Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models [12.660879089125684]
  Unified Multimodal Models (uMMs) aim to support both visual understanding and visual generation within a shared representation.<n>Existing evaluation protocols assess these two capabilities independently and do not examine whether they are semantically aligned.<n>We introduce XTC-Bench, a scene-graph-grounded evaluation framework that measures cross-task visual semantic consistency.
  arXiv  Detail & Related papers  (2026-04-27T23:57:29Z)
• Free Lunch for Unified Multimodal Models: Enhancing Generation via Reflective Rectification with Inherent Understanding [20.397510070808238]
  Unified Multimodal Models (UMMs) aim to integrate visual understanding and generation within a single structure.<n>UMMs exhibit a notable capability mismatch, where their understanding capability significantly outperforms their generation.<n>We propose UniRect-CoT, a training-free unified rectification chain-of-thought framework.
  arXiv  Detail & Related papers  (2026-04-15T06:41:56Z)
• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models [98.8608163448532]
  unified multimodal models (UMMs) have made remarkable progress in integrating visual understanding and generation.<n>We propose a token-level intrinsic text-image alignment reward mechanism, GvU, enabling the UMM to act simultaneously as teacher and student.<n>We show that our method substantially boosts UMMs' generation, which in turn strengthens their fine-grained visual understanding.
  arXiv  Detail & Related papers  (2026-03-06T08:56:14Z)
• UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? [50.92401586025528]
  Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear.<n>We introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks.
  arXiv  Detail & Related papers  (2026-03-03T18:36:16Z)
• Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities? [61.533560295383786]
  Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture.<n>We observe that U-MLLMs fail to maintain semantic equivalence when required to render the same results in the image modality.<n>We introduce VGUBench, a framework to decouple reasoning logic from generation fidelity.
  arXiv  Detail & Related papers  (2026-02-27T06:23:56Z)
• Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Models [23.128973540926552]
  Endogenous Reprompting transforms the model's understanding into an explicit generative reasoning step.<n>We show that SEER consistently outperforms state-of-the-art baselines in evaluation accuracy, reprompting efficiency, and generation quality.
  arXiv  Detail & Related papers  (2026-01-28T06:54:36Z)
• UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark [46.85189386884927]
  UmniBench is a benchmark tailored for unified multimodal models with omni-dimensional evaluation.<n>It covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs.<n>Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models.
  arXiv  Detail & Related papers  (2025-12-19T03:20:59Z)
• Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark [69.8473923357969]
  Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration.<n>We present Uni-MMMU, a comprehensive benchmark that unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains.
  arXiv  Detail & Related papers  (2025-10-15T17:10:35Z)
• RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark [71.3555284685426]
  We introduce RealUnify, a benchmark designed to evaluate bidirectional capability synergy.<n>RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks.<n>We find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient.
  arXiv  Detail & Related papers  (2025-09-29T15:07:28Z)
• SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards [55.99492656542475]
  We propose textbfSUDER (textbfSelf-improving textbfUnified LMMs with textbfDual stextbfElf-textbfRewards), a framework reinforcing the understanding and generation capabilities of LMMs.
  arXiv  Detail & Related papers  (2025-06-09T17:38:45Z)
• UniEval: Unified Holistic Evaluation for Unified Multimodal Understanding and Generation [15.938314589086914]
  We introduce UniEval, the first evaluation framework designed for unified multimodal models without extra models, images, or annotations.<n>The framework contains a holistic benchmark, UniBench, along with the corresponding UniScore metric.<n>We also extensively evaluated SoTA unified and visual generation models, uncovering new insights into Univeral's unique values.
  arXiv  Detail & Related papers  (2025-05-15T16:34:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.