BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

• URL: http://arxiv.org/abs/2606.30319v1
• Date: Mon, 29 Jun 2026 14:02:15 GMT
• Title: BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language
• Abstract summary: BrainJanus is the first unified brain model that integrates brain, vision, and language within a single framework.<n>Our framework exhibits zero-shot generalization preserves and interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm.
• Score: 65.1211456110319
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at \href{https://github.com/HaitaoWuTJU/BrainJanus}{GitHub}.

Related papers

• Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion [20.46826223980239]
  Mind- Omni is a versatile framework that unifies seven distinct encoding and decoding tasks.<n>At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens.<n>Our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity.
  arXiv  Detail & Related papers  (2026-05-28T08:33:43Z)
• Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective [53.556348738917166]
  Recent advances in agentic AI have led to systems capable of autonomous task execution and language-based reasoning.<n>Human spatial intelligence, rooted in integrated multisensory perception, spatial memory, and cognitive maps, enables flexible, context-aware decision-making in unstructured environments.
  arXiv  Detail & Related papers  (2025-09-11T05:23:22Z)
• BrainFLORA: Uncovering Brain Concept Representation via Multimodal Neural Embeddings [19.761793010311614]
  We introduce BrainFLORA, a unified framework for integrating cross-modal neuroimaging data to construct a shared neural representation.<n>Our approach leverages multimodal large language models (MLLMs) augmented with modality-specific adapters and task decoders, achieving state-of-the-art performance in joint-subject visual retrieval.<n>BrainFLORA offers novel implications for cognitive neuroscience and brain-computer interfaces (BCIs)
  arXiv  Detail & Related papers  (2025-07-13T18:56:17Z)
• Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing [36.47424671683706]
  We propose a unified framework to align brain signals with a shared semantic space encompassing text, images, and audio.<n>A router module dynamically selects and fuses modality-specific brain features according to the characteristics of each stimulus.<n>Experiments on various fMRI datasets with textual, visual, and auditory stimuli demonstrate state-of-the-art performance, achieving an 8.48% improvement on the most commonly used benchmark.
  arXiv  Detail & Related papers  (2025-05-15T14:46:45Z)
• Neural Brain: A Neuroscience-inspired Framework for Embodied Agents [78.61382193420914]
  Current AI systems, such as large language models, remain disembodied, unable to physically engage with the world.<n>At the core of this challenge lies the concept of Neural Brain, a central intelligence system designed to drive embodied agents with human-like adaptability.<n>This paper introduces a unified framework for the Neural Brain of embodied agents, addressing two fundamental challenges.
  arXiv  Detail & Related papers  (2025-05-12T15:05:34Z)
• MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data [64.92867794764247]
  MindAligner is a framework for cross-subject brain decoding from limited fMRI data.<n>Brain Transfer Matrix (BTM) projects the brain signals of an arbitrary new subject to one of the known subjects.<n>Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli.
  arXiv  Detail & Related papers  (2025-02-07T16:01:59Z)
• MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model [45.18716166499859]
  Deciphering the human visual experience through brain activities captured by fMRI represents a compelling and cutting-edge challenge. We introduce MindSemantix, a novel multi-modal framework that enables LLMs to comprehend visually-evoked semantic content in brain activity. MindSemantix generates high-quality captions that are deeply rooted in the visual and semantic information derived from brain activity.
  arXiv  Detail & Related papers  (2024-05-29T06:55:03Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey) [9.14580723964253]
  Can artificial intelligence unlock the secrets of the human brain?<n>Is it possible to enhance AI by tapping into the power of brain recordings?<n>Our survey focuses on human brain recording studies and cutting-edge cognitive neuroscience datasets.
  arXiv  Detail & Related papers  (2023-07-17T06:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.