Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy
- URL: http://arxiv.org/abs/2401.00430v2
- Date: Wed, 3 Jan 2024 08:50:27 GMT
- Title: Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy
- Authors: Weijian Mai, Jian Zhang, Pengfei Fang, Zhijun Zhang
- Abstract summary: Key to multimodal synthesis technology is to establish the mapping relationship between different modalities.
Brian-conditional multimodal synthesis refers to decoding brain signals back to perceptual experience.
This survey comprehensively examines the emerging field of AIGC-based Brain-conditional Multimodal Synthesis, termed AIGC-Brain.
- Score: 18.130004804879896
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In the era of Artificial Intelligence Generated Content (AIGC), conditional
multimodal synthesis technologies (e.g., text-to-image, text-to-video,
text-to-audio, etc) are gradually reshaping the natural content in the real
world. The key to multimodal synthesis technology is to establish the mapping
relationship between different modalities. Brain signals, serving as potential
reflections of how the brain interprets external information, exhibit a
distinctive One-to-Many correspondence with various external modalities. This
correspondence makes brain signals emerge as a promising guiding condition for
multimodal content synthesis. Brian-conditional multimodal synthesis refers to
decoding brain signals back to perceptual experience, which is crucial for
developing practical brain-computer interface systems and unraveling complex
mechanisms underlying how the brain perceives and comprehends external stimuli.
This survey comprehensively examines the emerging field of AIGC-based
Brain-conditional Multimodal Synthesis, termed AIGC-Brain, to delineate the
current landscape and future directions. To begin, related brain neuroimaging
datasets, functional brain regions, and mainstream generative models are
introduced as the foundation of AIGC-Brain decoding and analysis. Next, we
provide a comprehensive taxonomy for AIGC-Brain decoding models and present
task-specific representative work and detailed implementation strategies to
facilitate comparison and in-depth analysis. Quality assessments are then
introduced for both qualitative and quantitative evaluation. Finally, this
survey explores insights gained, providing current challenges and outlining
prospects of AIGC-Brain. Being the inaugural survey in this domain, this paper
paves the way for the progress of AIGC-Brain research, offering a foundational
overview to guide future work.
Related papers
- Converging Paradigms: The Synergy of Symbolic and Connectionist AI in LLM-Empowered Autonomous Agents [54.247747237176625]
Article explores the convergence of connectionist and symbolic artificial intelligence (AI)
Traditionally, connectionist AI focuses on neural networks, while symbolic AI emphasizes symbolic representation and logic.
Recent advancements in large language models (LLMs) highlight the potential of connectionist architectures in handling human language as a form of symbols.
arXiv Detail & Related papers (2024-07-11T14:00:53Z) - Neuro-Vision to Language: Enhancing Visual Reconstruction and Language Interaction through Brain Recordings [8.63068449082585]
Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition.
Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D.
We have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development.
arXiv Detail & Related papers (2024-04-30T10:41:23Z) - MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
Brain decoding aims to reconstruct stimuli from acquired brain signals.
Currently, brain decoding is confined to a per-subject-per-model paradigm.
We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
arXiv Detail & Related papers (2024-04-11T15:46:42Z) - A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian
Learning and Free Energy Minimization [55.11642177631929]
Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images.
We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
arXiv Detail & Related papers (2023-10-14T23:28:48Z) - UniBrain: Universal Brain MRI Diagnosis with Hierarchical
Knowledge-enhanced Pre-training [66.16134293168535]
We propose a hierarchical knowledge-enhanced pre-training framework for the universal brain MRI diagnosis, termed as UniBrain.
Specifically, UniBrain leverages a large-scale dataset of 24,770 imaging-report pairs from routine diagnostics.
arXiv Detail & Related papers (2023-09-13T09:22:49Z) - Multimodal foundation models are better simulators of the human brain [65.10501322822881]
We present a newly-designed multimodal foundation model pre-trained on 15 million image-text pairs.
We find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
arXiv Detail & Related papers (2022-08-17T12:36:26Z) - Interpretable Graph Neural Networks for Connectome-Based Brain Disorder
Analysis [31.281194583900998]
We propose an interpretable framework to analyze disorder-specific Regions of Interest (ROIs) and prominent connections.
The proposed framework consists of two modules: a brain-network-oriented backbone model for disease prediction and a globally shared explanation generator.
arXiv Detail & Related papers (2022-06-30T08:02:05Z) - Cross-Modality Neuroimage Synthesis: A Survey [71.27193056354741]
Multi-modality imaging improves disease diagnosis and reveals distinct deviations in tissues with anatomical properties.
The existence of completely aligned and paired multi-modality neuroimaging data has proved its effectiveness in brain research.
An alternative solution is to explore unsupervised or weakly supervised learning methods to synthesize the absent neuroimaging data.
arXiv Detail & Related papers (2022-02-14T19:29:08Z) - Towards Understanding Human Functional Brain Development with
Explainable Artificial Intelligence: Challenges and Perspectives [6.106661781836959]
This paper aims to understand the extent to which current state-of-the-art AI techniques can inform functional brain development.
A review of which AI techniques are more likely to explain their learning based on the processes of brain development is also undertaken.
arXiv Detail & Related papers (2021-12-24T02:13:13Z) - The whole brain architecture approach: Accelerating the development of
artificial general intelligence by referring to the brain [1.637145148171519]
It is difficult for an individual to design a software program that corresponds to the entire brain.
The whole-brain architecture approach divides the brain-inspired AGI development process into the task of designing the brain reference architecture.
This study proposes the Structure-constrained Interface Decomposition (SCID) method, which is a hypothesis-building method for creating a hypothetical component diagram.
arXiv Detail & Related papers (2021-03-06T04:58:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.