Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decoding

• URL: http://arxiv.org/abs/2502.17465v1
• Date: Tue, 11 Feb 2025 14:43:14 GMT
• Title: Bridging Brain Signals and Language: A Deep Learning Approach to EEG-to-Text Decoding
• Authors: Mostafa El Gedawy, Omnia Nabil, Omar Mamdouh, Mahmoud Nady, Nour Alhuda Adel, Ahmed Fares,
• Abstract summary: We introduce a special framework which changes conventional closed-vocabulary EEG-to-text decoding approaches.<n>This research aims to create a connection between open-vocabulary Text generation systems and human brain signal interpretation.
• Score: 1.1655046053160683
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Brain activity translation into human language delivers the capability to revolutionize machine-human interaction while providing communication support to people with speech disability. Electronic decoding reaches a certain level of achievement yet current EEG-to-text decoding methods fail to reach open vocabularies and depth of meaning and individual brain-specific variables. We introduce a special framework which changes conventional closed-vocabulary EEG-to-text decoding approaches by integrating subject-specific learning models with natural language processing methods to resolve detection obstacles. This method applies a deep representation learning approach to extract important EEG features which allow training of neural networks to create elaborate sentences that extend beyond original data content. The ZuCo dataset analysis demonstrates that research findings achieve higher BLEU, ROUGE and BERTScore performance when compared to current methods. The research proves how this framework functions as an effective approach to generate meaningful and correct texts while understanding individual brain variations. The proposed research aims to create a connection between open-vocabulary Text generation systems and human brain signal interpretation for developing efficacious brain-to-text systems. The research produces interdisciplinary effects through innovative assistive technology development and personalized communication systems which extend possibilities for human-computer interaction in various settings.

Related papers

• A Survey on Bridging EEG Signals and Generative AI: From Image and Text to Beyond [4.720913027054481]
  Integration of Brain-Computer Interfaces (BCIs) and Generative Artificial Intelligence (GenAI) has opened new frontiers in brain signal decoding.<n>Recent advances in deep learning, including Generative Adversarial Networks (GANs) and Transformer-based Large Language Models (LLMs), have significantly improved EEG-based generation of images, text, and speech.
  arXiv  Detail & Related papers  (2025-02-17T17:16:41Z)
• On Creating A Brain-To-Text Decoder [6.084958172018792]
  This paper centers on the application of raw electroencephalogram (EEG) signals for decoding human brain activity. The investigation specifically scrutinizes the efficacy of brain-computer interfaces (BCI) in deciphering neural signals associated with speech production.
  arXiv  Detail & Related papers  (2025-01-10T20:04:54Z)
• DEEPER: Dense Electroencephalography Passage Retrieval [6.084958172018792]
  This paper introduces DEEPER Dense EEG Passage Retrieval, a novel framework that bypasses the need for explicit query translation.<n>Our approach employs dense retrieval architectures to create a unified semantic space where both EEG signals and text passages can be effectively compared.
  arXiv  Detail & Related papers  (2024-12-09T17:41:25Z)
• Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation [70.52558242336988]
  We focus on predicting engagement in dyadic interactions by scrutinizing verbal and non-verbal cues, aiming to detect signs of disinterest or confusion. In this work, we collect a dataset featuring 34 participants engaged in casual dyadic conversations, each providing self-reported engagement ratings at the end of each conversation. We introduce a novel fusion strategy using Large Language Models (LLMs) to integrate multiple behavior modalities into a multimodal transcript''
  arXiv  Detail & Related papers  (2024-09-13T18:28:12Z)
• MindSpeech: Continuous Imagined Speech Decoding using High-Density fNIRS and Prompt Tuning for Advanced Human-AI Interaction [0.0]
  This paper reports a novel method for human-AI interaction by developing a direct brain-AI interface. We discuss a novel AI model, called MindSpeech, which enables open-vocabulary, continuous decoding for imagined speech. We demonstrate significant improvements in key metrics, such as BLEU-1 and BERT P scores, for three out of four participants.
  arXiv  Detail & Related papers  (2024-07-25T16:39:21Z)
• Language Generation from Brain Recordings [68.97414452707103]
  We propose a generative language BCI that utilizes the capacity of a large language model and a semantic brain decoder. The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli. Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.
  arXiv  Detail & Related papers  (2023-11-16T13:37:21Z)
• Language-Oriented Communication with Semantic Coding and Knowledge Distillation for Text-to-Image Generation [53.97155730116369]
  We put forward a novel framework of language-oriented semantic communication (LSC) In LSC, machines communicate using human language messages that can be interpreted and manipulated via natural language processing (NLP) techniques for SC efficiency. We introduce three innovative algorithms: 1) semantic source coding (SSC), which compresses a text prompt into its key head words capturing the prompt's syntactic essence; 2) semantic channel coding ( SCC), that improves robustness against errors by substituting head words with their lenghthier synonyms; and 3) semantic knowledge distillation (SKD), that produces listener-customized prompts via in-context learning the listener's
  arXiv  Detail & Related papers  (2023-09-20T08:19:05Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification [78.120927891455]
  State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines.
  arXiv  Detail & Related papers  (2021-12-05T21:57:22Z)
• CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals [60.921888445317705]
  We propose a CogAlign approach to integrate cognitive language processing signals into natural language processing models. We show that CogAlign achieves significant improvements with multiple cognitive features over state-of-the-art models on public datasets.
  arXiv  Detail & Related papers  (2021-06-10T07:10:25Z)
• Decoding EEG Brain Activity for Multi-Modal Natural Language Processing [9.35961671939495]
  We present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. For a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines.
  arXiv  Detail & Related papers  (2021-02-17T09:44:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.