Related papers: TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

URL: http://arxiv.org/abs/2604.10784v1
Date: Sun, 12 Apr 2026 19:19:04 GMT
Title: TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training
Abstract summary: We present TorchUMM, the first unified framework for comprehensive evaluation, analysis, and post-training of unified multimodal models.<n>Our benchmark encompasses multimodal understanding, generation, and editing, and integrates both established and novel datasets.
Score: 25.562880121429966
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Recent advances in unified multimodal models (UMMs) have led to a proliferation of architectures capable of understanding, generating, and editing across visual and textual modalities. However, developing a unified framework for UMMs remains challenging due to the diversity of model architectures and the heterogeneity of training paradigms and implementation details. In this paper, we present TorchUMM, the first unified codebase for comprehensive evaluation, analysis, and post-training across diverse UMM backbones, tasks, and datasets. TorchUMM supports a broad spectrum of models covering a wide range of scales and design paradigms. Our benchmark encompasses three core task dimensions: multimodal understanding, generation, and editing, and integrates both established and novel datasets to evaluate perception, reasoning, compositionality, and instruction-following abilities. By providing a unified interface and standardized evaluation protocols, TorchUMM enables fair and reproducible comparisons across heterogeneous models and fosters deeper insights into their strengths and limitations, facilitating the development of more capable unified multimodal systems. Code is available at: https://github.com/AIFrontierLab/TorchUMM.

Related papers

InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing [87.5130783098133]
InternVL-U is a lightweight 4B- parameter UMM that democratizes multimodal understanding and reasoning capabilities.<n>It integrates a state-of-the-art Multimodal Large Language Model (MLLM) with a specialized MMDiT-based visual generation head.<n>It consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks.
arXiv Detail & Related papers (2026-03-10T16:38:33Z)
UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark [72.37370242707432]
This paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset.<n>UniM contains 31K high-quality instances across 30 domains and 7 representative modalities.<n>We also introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness & Generation Quality, Response Structure Integrity, and Interleaved Coherence.
arXiv Detail & Related papers (2026-03-05T11:45:16Z)
Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation [53.18286807225952]
Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework.<n>We propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method.<n>Our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.
arXiv Detail & Related papers (2026-01-29T08:42:25Z)
Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark [69.8473923357969]
Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration.<n>We present Uni-MMMU, a comprehensive benchmark that unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains.
arXiv Detail & Related papers (2025-10-15T17:10:35Z)
VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models [82.05514464090172]
Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding.<n>However, their ability to generate code from multimodal inputs remains limited.<n>We introduce VisCodex, a unified framework that seamlessly merges vision and coding language models.
arXiv Detail & Related papers (2025-08-13T17:00:44Z)
A Survey of Generative Categories and Techniques in Multimodal Large Language Models [3.7507324448128876]
Multimodal Large Language Models (MLLMs) have rapidly evolved beyond text generation.<n>This survey categorises six primary generative modalities and examines how foundational techniques enable cross-modal capabilities.
arXiv Detail & Related papers (2025-05-29T12:29:39Z)
IWISDM: Assessing instruction following in multimodal models at scale [1.2320972303448239]
We introduce the instructed-Virtual VISual Decision Making (iWISDM) environment engineered to generate a limitless array of vision-language tasks. Using iWISDM, we compiled three distinct benchmarks of instruction following visual tasks across varying complexity levels. Our findings establish iWISDM as a robust benchmark for assessing the instructional adherence of both existing and emergent multimodal models.
arXiv Detail & Related papers (2024-06-20T14:09:54Z)
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
arXiv Detail & Related papers (2024-05-18T12:16:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.