Rosetta: Composable Native Multimodal Pretraining

• URL: http://arxiv.org/abs/2607.00293v1
• Date: Wed, 01 Jul 2026 00:42:40 GMT
• Title: Rosetta: Composable Native Multimodal Pretraining
• Abstract summary: Rosetta is a composable native multimodal pretraining framework for seamless and non-destructive modality expansion.<n>It preserves core foundational knowledge within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts.<n>It delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models.
• Score: 45.254713953182716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. However, accommodating continuous generative objectives alongside discrete understanding tasks causes severe gradient conflicts. Existing architectures, including standard Mixture-of-Experts (MoE), are highly susceptible to representation overwriting. Even structurally partitioned paradigms like Mixture-of-Transformers (MoT) remain vulnerable to catastrophic forgetting, severely impeding multimodal scalability. In this work, we introduce Rosetta, a composable native multimodal pretraining framework designed for seamless and non-destructive modality expansion. Rosetta adopts a modular paradigm where core foundational knowledge is preserved within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts. To guarantee non-destructive composition, we propose Momentum-Anchored Orthogonal Projection (MAOP). MAOP leverages the optimizer's momentum state as an implicit semantic anchor, selectively neutralizing conflicting gradient components from new modalities while preserving synergistic updates. Extensive evaluations demonstrate that, while standard MoE and MoT architectures suffer catastrophic forgetting of previously acquired knowledge, Rosetta robustly preserves established language and visual understanding. Furthermore, it delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models. To facilitate further multimodal research, we release our code and checkpoints to the community. Project page at https://rosetta-lmm.github.io/.

Related papers

• CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning [16.033361968793162]
  Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs)<n>We propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts.<n>We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models.
  arXiv  Detail & Related papers  (2026-05-18T06:31:14Z)
• SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture [110.20462227888915]
  We introduce SenseNova-U1, a native unified multimodal paradigm built upon NEO-unify.<n>We show two native unified variants, SenseNova-U1-8B-MoT and SenseNova-U1-A3B-MoT, built on dense (8B) and mixture-of-experts (30B-A3B) understanding baselines.<n>We also show detailed model design, data preprocessing, pre-/post-training, and inference strategies to support community research.
  arXiv  Detail & Related papers  (2026-05-12T17:59:58Z)
• Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding [45.254713953182716]
  We present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture.<n> Modality-Aware Expert Disentanglement allows shared experts to absorb fine-grained visual semantics from generative tasks.<n>Experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy.
  arXiv  Detail & Related papers  (2026-04-09T03:19:26Z)
• LongCat-Next: Lexicalizing Modalities as Discrete Tokens [111.50794040093656]
  LongCat-Next is a native multimodal model that processes text, vision, and audio under a single autoregressive objective.<n>In particular, LongCat-Next addresses the long-standing performance ceiling of discrete vision modeling on understanding tasks.<n>As an attempt toward native multimodality, we open-source the LongCat-Next and its tokenizers, hoping to foster further research and development in the community.
  arXiv  Detail & Related papers  (2026-03-29T06:35:03Z)
• Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variants [85.33837131101342]
  We propose a strategic roadmap organized into four pillars: foundational infrastructure, algorithmic optimization, cognitive reasoning, and unified multimodal intelligence.<n>We argue that this transition is essential for developing next-generation AI capable of complex structural reasoning, dynamic self-correction, and seamless multimodal integration.
  arXiv  Detail & Related papers  (2026-01-20T14:58:23Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate [1.0152838128195467]
  The prevailing paradigm for scaling large language models (LLMs) involves monolithic, end-to-end training.<n>This paper explores an alternative, constructive scaling paradigm, enabled by the principle of emergent semantics in Transformers.<n>We operationalize this with a layer-wise constructive methodology that combines strict layer freezing in early stages with efficient, holistic fine-tuning of the entire model stack.
  arXiv  Detail & Related papers  (2025-07-08T20:01:15Z)
• Mixture of Experts Made Intrinsically Interpretable [34.36996159677674]
  We present textbfMoE-X, a Mixture-of-Experts (MoE) language model designed to be emphintrinsically interpretable.<n>Our approach is motivated by the observation that, in language models, wider networks with sparse activations are more likely to capture interpretable factors.<n>MoE-X achieves perplexity better than GPT-2, with interpretability surpassing even sparse autoencoder (SAE)-based approaches.
  arXiv  Detail & Related papers  (2025-03-05T17:40:54Z)
• AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Framework [21.10693332367192]
  We present AllSpark, a multimodal-temporal general artificial intelligence model.<n>Our model integrates ten different modalities into a unified framework.<n> Experiments indicate that the incorporation of language enables AllSpark to excel in few-shot classification tasks.
  arXiv  Detail & Related papers  (2023-12-31T17:21:02Z)
• Language Models are General-Purpose Interfaces [109.45478241369655]
  We propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language) We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders.
  arXiv  Detail & Related papers  (2022-06-13T17:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.