Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy

• URL: http://arxiv.org/abs/2401.00430v2
• Date: Wed, 3 Jan 2024 08:50:27 GMT
• Title: Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy
• Authors: Weijian Mai, Jian Zhang, Pengfei Fang, Zhijun Zhang
• Abstract summary: Key to multimodal synthesis technology is to establish the mapping relationship between different modalities. Brian-conditional multimodal synthesis refers to decoding brain signals back to perceptual experience. This survey comprehensively examines the emerging field of AIGC-based Brain-conditional Multimodal Synthesis, termed AIGC-Brain.
• Score: 18.130004804879896
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the era of Artificial Intelligence Generated Content (AIGC), conditional multimodal synthesis technologies (e.g., text-to-image, text-to-video, text-to-audio, etc) are gradually reshaping the natural content in the real world. The key to multimodal synthesis technology is to establish the mapping relationship between different modalities. Brain signals, serving as potential reflections of how the brain interprets external information, exhibit a distinctive One-to-Many correspondence with various external modalities. This correspondence makes brain signals emerge as a promising guiding condition for multimodal content synthesis. Brian-conditional multimodal synthesis refers to decoding brain signals back to perceptual experience, which is crucial for developing practical brain-computer interface systems and unraveling complex mechanisms underlying how the brain perceives and comprehends external stimuli. This survey comprehensively examines the emerging field of AIGC-based Brain-conditional Multimodal Synthesis, termed AIGC-Brain, to delineate the current landscape and future directions. To begin, related brain neuroimaging datasets, functional brain regions, and mainstream generative models are introduced as the foundation of AIGC-Brain decoding and analysis. Next, we provide a comprehensive taxonomy for AIGC-Brain decoding models and present task-specific representative work and detailed implementation strategies to facilitate comparison and in-depth analysis. Quality assessments are then introduced for both qualitative and quantitative evaluation. Finally, this survey explores insights gained, providing current challenges and outlining prospects of AIGC-Brain. Being the inaugural survey in this domain, this paper paves the way for the progress of AIGC-Brain research, offering a foundational overview to guide future work.

Related papers

• Brain-like Functional Organization within Large Language Models [58.93629121400745]
  The human brain has long inspired the pursuit of artificial intelligence (AI) Recent neuroimaging studies provide compelling evidence of alignment between the computational representation of artificial neural networks (ANNs) and the neural responses of the human brain to stimuli. In this study, we bridge this gap by directly coupling sub-groups of artificial neurons with functional brain networks (FBNs) This framework links the AN sub-groups to FBNs, enabling the delineation of brain-like functional organization within large language models (LLMs)
  arXiv  Detail & Related papers  (2024-10-25T13:15:17Z)
• Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation [53.70131202548981]
  We present a two-step segmentation framework employing Knowledge-Guided Prompt Learning (KGPL) for brain MRI. Specifically, we first pre-train segmentation models on large-scale datasets with sub-optimal labels. The introduction of knowledge-wise prompts captures semantic relationships between anatomical variability and biological processes.
  arXiv  Detail & Related papers  (2024-07-31T04:32:43Z)
• Converging Paradigms: The Synergy of Symbolic and Connectionist AI in LLM-Empowered Autonomous Agents [55.63497537202751]
  Article explores the convergence of connectionist and symbolic artificial intelligence (AI) Traditionally, connectionist AI focuses on neural networks, while symbolic AI emphasizes symbolic representation and logic. Recent advancements in large language models (LLMs) highlight the potential of connectionist architectures in handling human language as a form of symbols.
  arXiv  Detail & Related papers  (2024-07-11T14:00:53Z)
• Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction [8.63068449082585]
  Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition. Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D. We have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development.
  arXiv  Detail & Related papers  (2024-04-30T10:41:23Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization [55.11642177631929]
  Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images. We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
  arXiv  Detail & Related papers  (2023-10-14T23:28:48Z)
• Multimodal foundation models are better simulators of the human brain [65.10501322822881]
  We present a newly-designed multimodal foundation model pre-trained on 15 million image-text pairs. We find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
  arXiv  Detail & Related papers  (2022-08-17T12:36:26Z)
• Interpretable Graph Neural Networks for Connectome-Based Brain Disorder Analysis [31.281194583900998]
  We propose an interpretable framework to analyze disorder-specific Regions of Interest (ROIs) and prominent connections. The proposed framework consists of two modules: a brain-network-oriented backbone model for disease prediction and a globally shared explanation generator.
  arXiv  Detail & Related papers  (2022-06-30T08:02:05Z)
• Towards Understanding Human Functional Brain Development with Explainable Artificial Intelligence: Challenges and Perspectives [6.106661781836959]
  This paper aims to understand the extent to which current state-of-the-art AI techniques can inform functional brain development. A review of which AI techniques are more likely to explain their learning based on the processes of brain development is also undertaken.
  arXiv  Detail & Related papers  (2021-12-24T02:13:13Z)
• The whole brain architecture approach: Accelerating the development of artificial general intelligence by referring to the brain [1.637145148171519]
  It is difficult for an individual to design a software program that corresponds to the entire brain. The whole-brain architecture approach divides the brain-inspired AGI development process into the task of designing the brain reference architecture. This study proposes the Structure-constrained Interface Decomposition (SCID) method, which is a hypothesis-building method for creating a hypothetical component diagram.
  arXiv  Detail & Related papers  (2021-03-06T04:58:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.