OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation

• URL: http://arxiv.org/abs/2505.23661v3
• Date: Mon, 02 Jun 2025 13:04:26 GMT
• Title: OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation
• Abstract summary: We present OpenUni, a lightweight, open-source baseline for unifying multimodal understanding and generation.<n>Inspired by prevailing practices in unified model learning, we adopt an efficient training strategy that minimizes the training complexity and overhead.<n>We demonstrate that OpenUni can: 1) generate high-quality and instruction-aligned images, and 2) achieve exceptional performance on standard benchmarks such as GenEval, DPG- Bench, and WISE, with only 1.1B and 3.1B activated parameters.
• Score: 44.34131932968147
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this report, we present OpenUni, a simple, lightweight, and fully open-source baseline for unifying multimodal understanding and generation. Inspired by prevailing practices in unified model learning, we adopt an efficient training strategy that minimizes the training complexity and overhead by bridging the off-the-shelf multimodal large language models (LLMs) and diffusion models through a set of learnable queries and a light-weight transformer-based connector. With a minimalist choice of architecture, we demonstrate that OpenUni can: 1) generate high-quality and instruction-aligned images, and 2) achieve exceptional performance on standard benchmarks such as GenEval, DPG- Bench, and WISE, with only 1.1B and 3.1B activated parameters. To support open research and community advancement, we release all model weights, training code, and our curated training datasets (including 23M image-text pairs) at https://github.com/wusize/OpenUni.

Related papers

• Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget [110.65684175617822]
  We introduce Boogu-Image-0.1, an open-source unified multimodal understanding and generation model family.<n>It delivers competitive performance in high-quality text-to-image generation, fast inference, instruction-based editing, and bilingual (Chinese-English) text rendering.
  arXiv  Detail & Related papers  (2026-07-14T17:52:05Z)
• UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark [72.37370242707432]
  This paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset.<n>UniM contains 31K high-quality instances across 30 domains and 7 representative modalities.<n>We also introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness & Generation Quality, Response Structure Integrity, and Interleaved Coherence.
  arXiv  Detail & Related papers  (2026-03-05T11:45:16Z)
• DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing [67.77471070868852]
  DeepGen 1.0 is a lightweight 5B unified model for image generation and editing.<n>It is trained on only 50M samples, surpassing the 80B HunyuanImage by 28% on WISE and the 27B Qwen-Image-Edit by 37% on UniREditBench.<n>By open-sourcing our training code, weights, and datasets, we provide an efficient, high-performance alternative to democratize unified multimodal research.
  arXiv  Detail & Related papers  (2026-02-12T17:44:24Z)
• DuoGen: Towards General Purpose Interleaved Multimodal Generation [65.13479486098419]
  DuoGen is a general-purpose interleaved generation framework that addresses data curation, architecture design, and evaluation.<n>We build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites.<n>A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences.
  arXiv  Detail & Related papers  (2026-01-31T04:35:15Z)
• FedUMM: A General Framework for Federated Learning with Unified Multimodal Models [11.0627306160262]
  Unified multimodal models (UMMs) are emerging as strong foundation models that can do both generation and understanding tasks in a single architecture.<n>We present FedUMM, a general federated learning framework for UMMs under non-IID multimodal data with low communication cost.
  arXiv  Detail & Related papers  (2026-01-21T19:02:52Z)
• UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning [17.68867710994329]
  We introduce UniRL, a self-improving post-training approach for multimodal large language models.<n>Our approach enables the model to generate images from prompts and use them as training data in each iteration.<n>We evaluate UniRL on top of Show-o and Janus, achieving a GenEval score of 0.77 for Show-o and 0.65 for Janus.
  arXiv  Detail & Related papers  (2025-05-29T12:00:15Z)
• Emerging Properties in Unified Multimodal Pretraining [32.856334401494145]
  We introduce BAGEL, an open-source foundational model that supports multimodal understanding and generation.<n>BAGEL is a unified, decoder-only model pretrained on trillions of tokens curated from large-scale interleaved text, image, video, and web data.<n>It significantly outperforms open-source unified models in both multimodal generation and understanding across standard benchmarks.
  arXiv  Detail & Related papers  (2025-05-20T17:59:30Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• xGen-MM (BLIP-3): A Family of Open Large Multimodal Models [157.44696790158784]
  This report introduces xGen-MM, a framework for developing Large Multimodal Models (LMMs) The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks.
  arXiv  Detail & Related papers  (2024-08-16T17:57:01Z)
• SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning [5.770978500931441]
  Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs) have leverage Attention-based Transformer architectures.<n>We develop SWIFT, a customizable one-stop infrastructure for large models.<n>We show that notable improvements on the ToolBench leader-board can be achieved by training with customized dataset on SWIFT.
  arXiv  Detail & Related papers  (2024-08-10T11:00:13Z)
• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
  We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
  arXiv  Detail & Related papers  (2024-05-18T12:16:01Z)
• Arcee's MergeKit: A Toolkit for Merging Large Language Models [0.6374098147778188]
  MergeKit is a framework to efficiently merge models on any hardware.<n>To date, thousands of models have been merged by the open-source community.
  arXiv  Detail & Related papers  (2024-03-20T02:38:01Z)
• Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants [65.47222691674074]
  Muffin framework employs pre-trained vision-language models to act as providers of visual signals. UniMM-Chat dataset explores the complementarities of datasets to generate 1.1M high-quality and diverse multimodal instructions.
  arXiv  Detail & Related papers  (2023-10-01T12:35:18Z)
• Otter: A Multi-Modal Model with In-Context Instruction Tuning [30.804061018682244]
  We introduce instruction tuning into multi-modal models, motivated by the Flamingo model's upstream interleaved format pretraining dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning.
  arXiv  Detail & Related papers  (2023-05-05T17:59:46Z)
• i-Code: An Integrative and Composable Multimodal Learning Framework [99.56065789066027]
  i-Code is a self-supervised pretraining framework where users may flexibly combine the modalities of vision, speech, and language into unified and general-purpose vector representations. The entire system is pretrained end-to-end with new objectives including masked modality unit modeling and cross-modality contrastive learning. Experimental results demonstrate how i-Code can outperform state-of-the-art techniques on five video understanding tasks and the GLUE NLP benchmark, improving by as much as 11%.
  arXiv  Detail & Related papers  (2022-05-03T23:38:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.