MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

• URL: http://arxiv.org/abs/2607.25948v1
• Date: Tue, 28 Jul 2026 16:34:23 GMT
• Title: MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
• Abstract summary: Any-to-any models predict any modality from any combination of others within a single network.<n>In this work, we investigate decoder-only any-to-any multimodal modeling.<n>Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications.
• Score: 40.11485744925299
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

Related papers

• Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion [66.78268790956493]
  We introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models.<n>Our method outperforms or performs on par with existing multimodal systems that process two or more modalities.
  arXiv  Detail & Related papers  (2026-03-06T18:59:57Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs [27.331524018411926]
  We compare encoder-only and decoder-only models on cross-modal adaptation for time-dependent simulation tasks.<n>We find that decoder-only models are far worse than encoder-only models, when existing approaches are applied unmodified.<n>We introduce two novel approaches, Parallel Flipping and Sequence Doubling, attempting to mimic bidirectionality in autoregressive models.
  arXiv  Detail & Related papers  (2025-10-06T18:46:50Z)
• Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces [10.85468238780625]
  We propose a novel framework for building multimodal diffusion models on arbitrary state spaces.<n>By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously.
  arXiv  Detail & Related papers  (2025-06-09T16:20:20Z)
• Multimodal ELBO with Diffusion Decoders [0.9208007322096533]
  We propose a new variant of the multimodal VAE ELBO that incorporates a better decoder using a diffusion generative model.<n>The diffusion decoder enables the model to learn complex modalities and generate high-quality outputs.<n>Our model provides state-of-the-art results compared to other multimodal VAEs in different datasets with higher coherence and superior quality in the generated modalities.
  arXiv  Detail & Related papers  (2024-08-29T20:12:01Z)
• Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine [10.774128925670183]
  Multimodal Lego (MM-Lego) is a general-purpose fusion framework to turn any set of encoders into a competitive multimodal model with no or minimal fine-tuning.<n>We show that MM-Lego can be used as a model merging method which achieves competitive performance with end-to-end fusion models without any fine-tuning.
  arXiv  Detail & Related papers  (2024-05-30T11:14:01Z)
• XTrack: Multimodal Training Boosts RGB-X Video Object Trackers [88.72203975896558]
  It is crucial to ensure that knowledge gained from multimodal sensing is effectively shared.<n>Similar samples across different modalities have more knowledge to share than otherwise.<n>We propose a method for RGB-X tracker during inference, with an average +3% precision improvement over the current SOTA.
  arXiv  Detail & Related papers  (2024-05-28T03:00:58Z)
• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing [3.3031006227198003]
  We present Modality-Agnostic Vision Transformer (MA-ViT), which aims to improve the performance of arbitrary modal attacks with the help of multi-modal data. Specifically, MA-ViT adopts the early fusion to aggregate all the available training modalities data and enables flexible testing of any given modal samples. Experiments demonstrate that the single model trained on MA-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin.
  arXiv  Detail & Related papers  (2023-04-15T13:03:44Z)
• Attention Bottlenecks for Multimodal Fusion [90.75885715478054]
  Machine perception models are typically modality-specific and optimised for unimodal benchmarks. We introduce a novel transformer based architecture that uses fusion' for modality fusion at multiple layers. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks.
  arXiv  Detail & Related papers  (2021-06-30T22:44:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.