Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models

• URL: http://arxiv.org/abs/2509.24365v1
• Date: Mon, 29 Sep 2025 07:05:10 GMT
• Title: Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
• Abstract summary: We propose Uni-X, a two-end-separated, middle-shared architecture for unified multimodal models.<n>Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion.<n>Our results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling.
• Score: 27.38501052629525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified Multimodal Models (UMMs) built on shared autoregressive (AR) transformers are attractive for their architectural simplicity. However, we identify a critical limitation: when trained on multimodal inputs, modality-shared transformers suffer from severe gradient conflicts between vision and text, particularly in shallow and deep layers. We trace this issue to the fundamentally different low-level statistical properties of images and text, while noting that conflicts diminish in middle layers where representations become more abstract and semantically aligned. To overcome this challenge, we propose Uni-X, a two-end-separated, middle-shared architecture. Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion. This X-shaped design not only eliminates gradient conflicts at both ends but also further alleviates residual conflicts in the shared layers. Extensive experiments validate the effectiveness of Uni-X. Under identical training conditions, Uni-X achieves superior training efficiency compared to strong baselines. When scaled to 3B parameters with larger training data, Uni-X matches or surpasses 7B AR-based UMMs, achieving a GenEval score of 82 for image generation alongside strong performance in text and vision understanding tasks. These results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling. Our code is available at https://github.com/CURRENTF/Uni-X

Related papers

• UniWeTok: An Unified Binary Tokenizer with Codebook Size $\mathit{2^{128}}$ for Unified Multimodal Large Language Model [50.68870074090426]
  We introduce UniWeTok, a unified discrete tokenizer for Unified Multimodal Large Language Models.<n>For training framework, we introduce Pre-Post Distillation and a Generative-Aware Prior to enhance the semantic extraction and generative prior of the discrete tokens.<n>We propose a three-stage training framework designed to enhance UniWeTok's adaptability cross various image resolutions and perception-sensitive scenarios.
  arXiv  Detail & Related papers  (2026-02-15T15:07:19Z)
• UniX: Unifying Autoregression and Diffusion for Chest X-Ray Understanding and Generation [98.93314262366681]
  We present UniX, a next-generation unified medical foundation model for chest X-ray understanding and generation.<n>UniX decouples the two tasks into an autoregressive branch for understanding and a diffusion branch for high-fidelity generation.<n>On two representative benchmarks, UniX achieves a 46.1% improvement in understanding performance and a 24.2% gain in generation quality.
  arXiv  Detail & Related papers  (2026-01-16T18:59:58Z)
• UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation [54.38636515750502]
  We propose UniLayDiff: a Unified Diffusion Transformer for content-aware layout generation tasks.<n>We employ Multi-Modal Diffusion Transformer framework to capture the complex interplay between the background image, layout elements, and diverse constraints.<n>Experiments demonstrate that UniLayDiff achieves state-of-the-art performance across from unconditional to various conditional generation tasks.
  arXiv  Detail & Related papers  (2025-12-09T18:38:44Z)
• Architecture Decoupling Is Not All You Need For Unified Multimodal Model [64.19284951218098]
  We propose Attention Interaction Alignment (AIA) loss, which explicitly learns Task-Specific multimodal interaction patterns during training.<n>AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
  arXiv  Detail & Related papers  (2025-11-27T17:55:25Z)
• UniGame: Turning a Unified Multimodal Model Into Its Own Adversary [21.728770994708402]
  Unified Multimodal Models (UMMs) have shown impressive performance in both understanding and generation with a single architecture.<n>We present UniGame, a self-adversarial post-training framework that directly targets the inconsistencies.
  arXiv  Detail & Related papers  (2025-11-24T18:50:01Z)
• MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces [23.447713697204225]
  MAGE is a novel framework that bridges the semantic spaces of vision and text through an innovative alignment mechanism.<n>We employ a training strategy that combines cross-entropy and mean squared error, significantly enhancing the alignment effect.<n>Our proposed multimodal large model architecture, MAGE, achieved significantly better performance compared to similar works across various evaluation benchmarks.
  arXiv  Detail & Related papers  (2025-07-29T12:17:46Z)
• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation [39.921363034430875]
  Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence.<n>We study the modality alignment behaviors of task-specific expert models for understanding and generation.<n>We introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference.
  arXiv  Detail & Related papers  (2025-06-20T17:52:31Z)
• Manifold-aware Representation Learning for Degradation-agnostic Image Restoration [135.90908995927194]
  Image Restoration (IR) aims to recover high quality images from degraded inputs affected by various corruptions such as noise, blur, haze, rain, and low light conditions.<n>We present MIRAGE, a unified framework for all in one IR that explicitly decomposes the input feature space into three semantically aligned parallel branches.<n>This modular decomposition significantly improves generalization and efficiency across diverse degradations.
  arXiv  Detail & Related papers  (2025-05-24T12:52:10Z)
• QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration [0.0]
  A server with a single NVIDIA A100 GPU (80GB) using Mixtral-8x7B models demonstrate an 85% average reduction in turnaround time compared to NVIDIA's multi-instance GPU (MIG)<n> experiments on Google's Switch Transformer Base-8 model with up to four variants demonstrate the scalability and resilience of our approach in maintaining output quality compared to other model merging baselines, highlighting its effectiveness.
  arXiv  Detail & Related papers  (2025-05-10T00:46:04Z)
• An Efficient and Mixed Heterogeneous Model for Image Restoration [71.85124734060665]
  Current mainstream approaches are based on three architectural paradigms: CNNs, Transformers, and Mambas.<n>We propose RestorMixer, an efficient and general-purpose IR model based on mixed-architecture fusion.
  arXiv  Detail & Related papers  (2025-04-15T08:19:12Z)
• ARNet: Self-Supervised FG-SBIR with Unified Sample Feature Alignment and Multi-Scale Token Recycling [11.129453244307369]
  FG-SBIR aims to minimize the distance between sketches and corresponding images in the embedding space.<n>We propose an effective approach to narrow the gap between the two domains.<n>It mainly facilitates unified mutual information sharing both intra- and inter-samples.
  arXiv  Detail & Related papers  (2024-06-17T13:49:12Z)
• UniCLIP: Unified Framework for Contrastive Language-Image Pre-training [62.97551575508387]
  We propose UniCLIP, a Unified framework for Contrastive Language-Image Pre-training. UniCLIP integrates the contrastive loss of both inter-domain pairs and intra-domain pairs into a single universal space. UniCLIP outperforms previous vision-language pre-training methods on various single- and multi-modality downstream tasks.
  arXiv  Detail & Related papers  (2022-09-27T14:36:16Z)
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training [71.37731379031487]
  We propose a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. Unlike OpenAI CLIP that adopts a simple contrastive learning method, we devise a more advanced algorithm by adapting the latest method MoCo into the cross-modal scenario. By building a large queue-based dictionary, our BriVL can incorporate more negative samples in limited GPU resources.
  arXiv  Detail & Related papers  (2021-03-11T09:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.