Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

• URL: http://arxiv.org/abs/2604.07753v1
• Date: Thu, 09 Apr 2026 03:19:26 GMT
• Title: Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
• Abstract summary: We present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture.<n> Modality-Aware Expert Disentanglement allows shared experts to absorb fine-grained visual semantics from generative tasks.<n>Experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy.
• Score: 45.254713953182716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this conflict through structural isolation, they fundamentally sever cross-modal synergy and suffer from capacity fragmentation. In this work, we present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture with zero-parameter overhead. We first identify that standard MoE tuning leads to routing collapse, where generative gradients dominate expert utilization. To address this, we introduce Modality-Aware Expert Disentanglement, which partitions experts into task-specific groups while utilizing shared experts as a multimodal semantic bridge. Crucially, this design allows shared experts to absorb fine-grained visual semantics from generative tasks to enrich textual representations. To optimize this, we propose a Progressive Training Strategy featuring differential learning rates and early-stage gradient shielding. This mechanism not only shields pre-trained knowledge from early volatility but eventually transforms generative signals into constructive feedback for understanding. Extensive experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy, boosting inherent understanding with remarkable gains on MMLU and OCRBench.

Related papers

• Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction [11.257989394325671]
  Generation-guided training framework reinterprets visual generation as auxiliary supervision for representation learning.<n>Gas improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension.
  arXiv  Detail & Related papers  (2026-08-12T16:03:31Z)
• Rosetta: Composable Native Multimodal Pretraining [45.254713953182716]
  Rosetta is a composable native multimodal pretraining framework for seamless and non-destructive modality expansion.<n>It preserves core foundational knowledge within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts.<n>It delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models.
  arXiv  Detail & Related papers  (2026-07-01T00:42:40Z)
• Semantic Generative Tuning for Unified Multimodal Models [62.18894352635965]
  Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture.<n> prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives.<n>This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs.
  arXiv  Detail & Related papers  (2026-05-18T17:46:46Z)
• Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities [52.189426539304065]
  This paper introduces a paradigm shift that redefines multimodal forensics from conventional "feature fusion" to "modality generalization"<n>We propose the first modality-agnostic forgery (MAF) detection framework. By explicitly decoupling modality-specific styles, MAF precisely extracts the essential, cross-modal latent forgery knowledge.
  arXiv  Detail & Related papers  (2026-04-09T03:35:21Z)
• Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Models [53.41258113970795]
  We investigate whether cross-functionality reciprocity itself constitutes a structural source of vulnerability in Unified Multimodal Models (UMMs)<n>We propose RICE: Reciprocal Interaction-based Cross-functionality Exploitation, a novel attack paradigm that explicitly exploits bidirectional interactions between understanding and generation.
  arXiv  Detail & Related papers  (2026-03-28T16:28:39Z)
• Understanding and Harnessing Sparsity in Unified Multimodal Models [32.09095929575726]
  Large multimodal models have achieved remarkable progress in both understanding and generation.<n>Recent efforts pursue unified multimodal models that integrate heterogeneous components to support both capabilities within a single framework.<n>Yet, a systematic understanding of how these inefficiencies manifest across different components remains limited.
  arXiv  Detail & Related papers  (2025-12-02T02:47:29Z)
• Architecture Decoupling Is Not All You Need For Unified Multimodal Model [64.19284951218098]
  We propose Attention Interaction Alignment (AIA) loss, which explicitly learns Task-Specific multimodal interaction patterns during training.<n>AIA not only refines cross-modal attention patterns, but also boosts both generation and understanding performance.
  arXiv  Detail & Related papers  (2025-11-27T17:55:25Z)
• HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation [72.69742127579508]
  Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models)<n>In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors.<n> Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge.
  arXiv  Detail & Related papers  (2025-11-25T17:23:38Z)
• Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learning [31.877595633244734]
  Learning from multiple data streams in real-world scenarios is fundamentally challenging due to intrinsic heterogeneity and unpredictable concept drifts.<n>Existing methods typically assume homogeneous streams and employ static architectures with indiscriminate knowledge fusion.<n>We propose CAMEL, a framework that assigns each stream an independent system with a dedicated feature extractor and task-specific head.<n>Furthermore, we propose an Autonomous Expert Tuner (AET) strategy, which dynamically manages expert lifecycles in response to drift.
  arXiv  Detail & Related papers  (2025-08-03T05:35:34Z)
• UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation [39.921363034430875]
  Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence.<n>We study the modality alignment behaviors of task-specific expert models for understanding and generation.<n>We introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference.
  arXiv  Detail & Related papers  (2025-06-20T17:52:31Z)
• More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing [5.846028298833611]
  Conditionally Overlapping Mixture of ExperTs (COMET) is a general deep learning method that inducing a modular, sparse architecture with an exponential number of overlapping experts.<n>We demonstrate the effectiveness of COMET on a range of tasks, including image classification, language modeling, and regression.
  arXiv  Detail & Related papers  (2024-10-10T14:58:18Z)
• Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment [103.05005690990271]
  Mixture of insighTful Experts (MoTE) is a novel framework that combines reasoning chains and expert mixtures to improve self-alignments.<n>MoTE significantly improves model safety, jailbreak resistance, and over-refusal capabilities, achieving performance comparable to OpenAI's state-of-the-art o1 model.
  arXiv  Detail & Related papers  (2024-05-01T15:06:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.