Cross-Subject Data Splitting for Brain-to-Text Decoding

• URL: http://arxiv.org/abs/2312.10987v3
• Date: Fri, 14 Jun 2024 07:30:33 GMT
• Title: Cross-Subject Data Splitting for Brain-to-Text Decoding
• Abstract summary: We propose a cross-subject data splitting criterion for brain-to-text decoding on various types of cognitive dataset (fMRI, EEG) We undertake a comprehensive analysis on existing cross-subject data splitting strategies and prove that all these methods suffer from data leakage. The proposed cross-subject splitting method successfully addresses the data leakage problem and we re-evaluate some SOTA brain-to-text decoding models as baselines for further research.
• Score: 36.30024741795527
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent major milestones have successfully decoded non-invasive brain signals (e.g. functional Magnetic Resonance Imaging (fMRI) and electroencephalogram (EEG)) into natural language. Despite the progress in model design, how to split the datasets for training, validating, and testing still remains a matter of debate. Most of the prior researches applied subject-specific data splitting, where the decoding model is trained and evaluated per subject. Such splitting method poses challenges to the utilization efficiency of dataset as well as the generalization of models. In this study, we propose a cross-subject data splitting criterion for brain-to-text decoding on various types of cognitive dataset (fMRI, EEG), aiming to maximize dataset utilization and improve model generalization. We undertake a comprehensive analysis on existing cross-subject data splitting strategies and prove that all these methods suffer from data leakage, namely the leakage of test data to training set, which significantly leads to overfitting and overestimation of decoding models. The proposed cross-subject splitting method successfully addresses the data leakage problem and we re-evaluate some SOTA brain-to-text decoding models as baselines for further research.

Related papers

• Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction [20.162170862752813]
  We introduce a framework that decomposes EEG-to-text decoding into semantic anchor extraction via contrastive learning and sentence reconstruction.<n>Brain-CLIPLM achieves 67.55% top-5 and 85.00% top-25 sentence retrieval accuracy, significantly outperforming direct decoding baseline.<n>Results suggest EEG-to-text decoding is better framed as recovering compressed semantic content rather than reconstructing full sentences.
  arXiv  Detail & Related papers  (2026-03-23T11:45:51Z)
• Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding [2.9949629644252376]
  Brain encoding and decoding aims to understand the relationship between external stimuli and brain activities.<n>We propose a lightweight alignment framework equipped with two statistical learning components.<n>We demonstrate competitive empirical performance on the large-scale fMRI-image reconstruction benchmark data.
  arXiv  Detail & Related papers  (2026-03-22T03:48:16Z)
• Decoding non-invasive brain activity with novel deep-learning approaches [0.10152838128195464]
  This thesis delves into the world of non-invasive electrophysiological brain signals like electroencephalography (EEG) and magnetoencephalography (MEG)<n>The research aims to investigate what happens in the brain when we perceive visual stimuli or engage in covert speech (inner speech) and enhance the decoding performance of such stimuli.
  arXiv  Detail & Related papers  (2025-10-13T20:50:20Z)
• Adapting HFMCA to Graph Data: Self-Supervised Learning for Generalizable fMRI Representations [57.054499278843856]
  Functional magnetic resonance imaging (fMRI) analysis faces significant challenges due to limited dataset sizes and domain variability between studies.<n>Traditional self-supervised learning methods inspired by computer vision often rely on positive and negative sample pairs.<n>We propose adapting a recently developed Hierarchical Functional Maximal Correlation Algorithm (HFMCA) to graph-structured fMRI data.
  arXiv  Detail & Related papers  (2025-10-05T12:35:01Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• The role of data partitioning on the performance of EEG-based deep learning models in supervised cross-subject analysis: a preliminary study [37.69303106863453]
  Deep learning is advancing the analysis of electroencephalography (EEG) data by effectively discovering highly nonlinear patterns.<n>No comprehensive guidelines for proper data partitioning and cross-validation exist in the domain.<n>This paper thoroughly investigates the role of data partitioning and cross-validation in evaluating EEG deep learning models.
  arXiv  Detail & Related papers  (2025-05-19T12:05:28Z)
• Precision Adaptive Imputation Network : An Unified Technique for Mixed Datasets [0.0]
  This study introduces the Precision Adaptive Imputation Network (PAIN), a novel algorithm designed to enhance data reconstruction. PAIN employs a tri-step process that integrates statistical methods, random forests, and autoencoders, ensuring balanced accuracy and efficiency in imputation. The findings highlight PAIN's superior ability to preserve data distributions and maintain analytical integrity, particularly in complex scenarios where missingness is not completely at random.
  arXiv  Detail & Related papers  (2025-01-18T06:22:27Z)
• A multimodal LLM for the non-invasive decoding of spoken text from brain recordings [0.4187344935012482]
  We propose and end-to-end multimodal LLM for decoding spoken text from fMRI signals. The proposed architecture is founded on (i) an encoder derived from a specific transformer incorporating an augmented embedding layer for the encoder and a better-adjusted attention mechanism than that present in the state of the art. A benchmark in performed on a corpus consisting of a set of interactions human-human and human-robot interactions where fMRI and conversational signals are recorded synchronously.
  arXiv  Detail & Related papers  (2024-09-29T14:03:39Z)
• Across-subject ensemble-learning alleviates the need for large samples for fMRI decoding [37.41192511246204]
  Within-subject decoding avoids between-subject correspondence problems but requires large sample sizes to make accurate predictions. Here, we investigate an ensemble approach to decoding that combines the classifiers trained on data from other subjects to decode cognitive states in a new subject. We find that it outperforms the conventional decoding approach by up to 20% in accuracy, especially for datasets with limited per-subject data.
  arXiv  Detail & Related papers  (2024-07-09T08:22:44Z)
• Few-shot learning for COVID-19 Chest X-Ray Classification with Imbalanced Data: An Inter vs. Intra Domain Study [49.5374512525016]
  Medical image datasets are essential for training models used in computer-aided diagnosis, treatment planning, and medical research. Some challenges are associated with these datasets, including variability in data distribution, data scarcity, and transfer learning issues when using models pre-trained from generic images. We propose a methodology based on Siamese neural networks in which a series of techniques are integrated to mitigate the effects of data scarcity and distribution imbalance.
  arXiv  Detail & Related papers  (2024-01-18T16:59:27Z)
• Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation [5.304395026626743]
  Hallucination of text ungrounded in the input is a well-known problem in neural data-to-text generation. We propose a new way to mitigate hallucinations by combining the probabilistic output of a generator language model with the output of a special "text critic" Our method does not need any changes to the underlying LM's architecture or training procedure.
  arXiv  Detail & Related papers  (2023-10-25T20:05:07Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Source-Free Collaborative Domain Adaptation via Multi-Perspective Feature Enrichment for Functional MRI Analysis [55.03872260158717]
  Resting-state MRI functional (rs-fMRI) is increasingly employed in multi-site research to aid neurological disorder analysis. Many methods have been proposed to reduce fMRI heterogeneity between source and target domains. But acquiring source data is challenging due to concerns and/or data storage burdens in multi-site studies. We design a source-free collaborative domain adaptation framework for fMRI analysis, where only a pretrained source model and unlabeled target data are accessible.
  arXiv  Detail & Related papers  (2023-08-24T01:30:18Z)
• UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language [23.623579364849526]
  We propose fMRI2text, the first openvocabulary task aiming to bridge fMRI time series and human language. We present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEGto-text decoding.
  arXiv  Detail & Related papers  (2023-07-06T05:26:49Z)
• Machine Learning Based Missing Values Imputation in Categorical Datasets [2.5611256859404983]
  This research looked into the use of machine learning algorithms to fill in the gaps in categorical datasets. The emphasis was on ensemble models constructed using the Error Correction Output Codes framework. Deep learning for missing data imputation has obstacles despite these encouraging results, including the requirement for large amounts of labeled data.
  arXiv  Detail & Related papers  (2023-06-10T03:29:48Z)
• Cross-Modal Causal Intervention for Medical Report Generation [107.76649943399168]
  Radiology Report Generation (RRG) is essential for computer-aided diagnosis and medication guidance.<n> generating accurate lesion descriptions remains challenging due to spurious correlations from visual-linguistic biases.<n>We propose a two-stage framework named CrossModal Causal Representation Learning (CMCRL)<n> Experiments on IU-Xray and MIMIC-CXR show that our CMCRL pipeline significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-16T07:23:55Z)
• FAST-AID Brain: Fast and Accurate Segmentation Tool using Artificial Intelligence Developed for Brain [0.8376091455761259]
  A novel deep learning method is proposed for fast and accurate segmentation of the human brain into 132 regions. The proposed model uses an efficient U-Net-like network and benefits from the intersection points of different views and hierarchical relations. The proposed method can be applied to brain MRI data including skull or any other artifacts without preprocessing the images or a drop in performance.
  arXiv  Detail & Related papers  (2022-08-30T16:06:07Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Fader Networks for domain adaptation on fMRI: ABIDE-II study [68.5481471934606]
  We use 3D convolutional autoencoders to build the domain irrelevant latent space image representation and demonstrate this method to outperform existing approaches on ABIDE data.
  arXiv  Detail & Related papers  (2020-10-14T16:50:50Z)
• Context-Aware Refinement Network Incorporating Structural Connectivity Prior for Brain Midline Delineation [50.868845400939314]
  We propose a context-aware refinement network (CAR-Net) to refine and integrate the feature pyramid representation generated by the UNet. For keeping the structural connectivity of the brain midline, we introduce a novel connectivity regular loss. The proposed method requires fewer parameters and outperforms three state-of-the-art methods in terms of four evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-10T14:01:20Z)
• Knowledge Distillation for Brain Tumor Segmentation [0.0]
  We study the relationship between the performance of the model and the amount of data employed during the training process. A single model trained with additional data achieves performance close to the ensemble of multiple models and outperforms individual methods.
  arXiv  Detail & Related papers  (2020-02-10T12:44:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.