Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion

• URL: http://arxiv.org/abs/2605.29591v1
• Date: Thu, 28 May 2026 08:33:43 GMT
• Title: Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
• Abstract summary: Mind- Omni is a versatile framework that unifies seven distinct encoding and decoding tasks.<n>At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens.<n>Our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity.
• Score: 20.46826223980239
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modeling the interplay between external stimuli and internal neural representations is a pivotal research area for Brain-Computer Interfaces (BCIs). A major limitation of prior work is the prevailing paradigm of specialized, single-task models, which curtails versatility and neglects inter-task synergies. To address this, we propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens. This enables direct, token-level interactions for mutual understanding and generation between any two or more modalities within a shared semantic space. To unlock advanced reasoning capabilities, we further curate a specialized Brain Question Answering (BQA) instruction-tuning dataset. Our model not only establishes a new state-of-the-art among multi-task unified frameworks but also provides strong evidence for multi-task synergy. By demonstrating performance competitive with, and at times superior to, larger specialized models, our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity. The code is publicly available at https://github.com/ReedOnePeck/Mind-Omni.

Related papers

• BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language [65.1211456110319]
  BrainJanus is the first unified brain model that integrates brain, vision, and language within a single framework.<n>Our framework exhibits zero-shot generalization preserves and interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm.
  arXiv  Detail & Related papers  (2026-06-29T14:02:15Z)
• Toward a Multi-View Brain Network Foundation Model: Cross-View Consistency Learning Across Arbitrary Atlases [62.33465338932216]
  MV-BrainFM is a multi-view brain network foundation model designed to learn generalizable and scalable representations from brain networks constructed with arbitrary atlases.<n>Experiments on more than 20K subjects from 17 fMRI datasets show that MV-BrainFM consistently outperforms 14 existing brain network foundation models.
  arXiv  Detail & Related papers  (2026-03-20T11:55:00Z)
• InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing [87.5130783098133]
  InternVL-U is a lightweight 4B- parameter UMM that democratizes multimodal understanding and reasoning capabilities.<n>It integrates a state-of-the-art Multimodal Large Language Model (MLLM) with a specialized MMDiT-based visual generation head.<n>It consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks.
  arXiv  Detail & Related papers  (2026-03-10T16:38:33Z)
• Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG [2.783700146328046]
  We propose Brain-OF, the first brain foundation model jointly pretrained on fMRI, MEG and EEG inputs within a unified framework.<n>Brain-OF is pretrained on a large-scale corpus comprising around 40 datasets and demonstrates superior performance across diverse downstream tasks.
  arXiv  Detail & Related papers  (2026-02-26T15:47:13Z)
• A Unified Geometric Space Bridging AI Models and the Human Brain [24.54324712609098]
  Modern artificial neural networks now rival humans in language, perception, and reasoning.<n>It is still largely unknown whether these artificial systems organize information as the brain does.<n>Here we introduce a groundbreaking concept of Brain-like Space: a unified geometric space in which every AI model can be precisely situated.
  arXiv  Detail & Related papers  (2025-10-28T12:09:23Z)
• To See a World in a Spark of Neuron: Disentangling Multi-task Interference for Training-free Model Merging [16.81093103067372]
  We present the first study that relies on neuronal mechanisms for model merging.<n>We introduce NeuroMerging, a novel merging framework developed to mitigate task interference within neuronal subspaces.<n>Our findings highlighted the importance of aligning neuronal mechanisms in model merging.
  arXiv  Detail & Related papers  (2025-03-07T11:00:24Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks [59.09343552273045]
  We propose a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. We demonstrate that joint learning of these diverse objectives is simple, effective, and maximizes the weight-sharing of the model across these tasks. Our model achieves the state of the art on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models.
  arXiv  Detail & Related papers  (2023-03-29T16:42:30Z)
• Multimodal foundation models are better simulators of the human brain [65.10501322822881]
  We present a newly-designed multimodal foundation model pre-trained on 15 million image-text pairs. We find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
  arXiv  Detail & Related papers  (2022-08-17T12:36:26Z)
• WenLan 2.0: Make AI Imagine via a Multimodal Foundation Model [74.4875156387271]
  We develop a novel foundation model pre-trained with huge multimodal (visual and textual) data. We show that state-of-the-art results can be obtained on a wide range of downstream tasks.
  arXiv  Detail & Related papers  (2021-10-27T12:25:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.