Dynamic Neural Communication: Convergence of Computer Vision and Brain-Computer Interface

• URL: http://arxiv.org/abs/2411.09211v1
• Date: Thu, 14 Nov 2024 06:15:05 GMT
• Title: Dynamic Neural Communication: Convergence of Computer Vision and Brain-Computer Interface
• Authors: Ji-Ha Park, Seo-Hyun Lee, Soowon Kim, Seong-Whan Lee,
• Abstract summary: We introduce a dynamic neural communication method that leverages computer vision and brain-computer interface technologies. Our approach captures the user's intentions from neural signals and decodes visemes in short time steps to produce dynamic visual outputs. Results demonstrate the potential to rapidly capture and reconstruct lip movements during natural speech attempts from human neural signals.
• Score: 25.555303640695577
• License:
• Abstract: Interpreting human neural signals to decode static speech intentions such as text or images and dynamic speech intentions such as audio or video is showing great potential as an innovative communication tool. Human communication accompanies various features, such as articulatory movements, facial expressions, and internal speech, all of which are reflected in neural signals. However, most studies only generate short or fragmented outputs, while providing informative communication by leveraging various features from neural signals remains challenging. In this study, we introduce a dynamic neural communication method that leverages current computer vision and brain-computer interface technologies. Our approach captures the user's intentions from neural signals and decodes visemes in short time steps to produce dynamic visual outputs. The results demonstrate the potential to rapidly capture and reconstruct lip movements during natural speech attempts from human neural signals, enabling dynamic neural communication through the convergence of computer vision and brain--computer interface.

Related papers

• Towards Unified Neural Decoding of Perceived, Spoken and Imagined Speech from EEG Signals [1.33134751838052]
  This research investigated the effectiveness of deep learning models for non-invasive neural signal decoding. It focused on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech.
  arXiv  Detail & Related papers  (2024-11-14T07:20:08Z)
• NeuroBind: Towards Unified Multimodal Representations for Neural Signals [20.02503060795981]
  We present NeuroBind, a representation that unifies multiple brain signal types, including EEG, fMRI, calcium imaging, and spiking data. This approach holds significant potential for advancing neuroscience research, improving AI systems, and developing neuroprosthetics and brain-computer interfaces.
  arXiv  Detail & Related papers  (2024-07-19T04:42:52Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Towards Decoding Brain Activity During Passive Listening of Speech [0.0]
  We attempt to decode heard speech from intracranial electroencephalographic (iEEG) data using deep learning methods. This approach diverges from the conventional focus on speech production and instead chooses to investigate neural representations of perceived speech. Despite the approach not having achieved a breakthrough yet, the research sheds light on the potential of decoding neural activity during speech perception.
  arXiv  Detail & Related papers  (2024-02-26T20:04:01Z)
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks [74.3099028063756]
  We develop a new method with neuronal operations based on lateral connections and Hebbian learning. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities. Our method consistently solves for spiking neural networks with nearly zero forgetting.
  arXiv  Detail & Related papers  (2024-02-19T09:29:37Z)
• Neural Speech Embeddings for Speech Synthesis Based on Deep Generative Networks [27.64740032872726]
  We introduce the current brain-to-speech technology with the possibility of speech synthesis from brain signals. Also, we perform comprehensive analysis on the neural features and neural speech embeddings underlying the neurophysiological activation while performing speech.
  arXiv  Detail & Related papers  (2023-12-10T08:12:08Z)
• BrainBERT: Self-supervised representation learning for intracranial recordings [18.52962864519609]
  We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, with higher accuracy and with much less data. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.
  arXiv  Detail & Related papers  (2023-02-28T07:40:37Z)
• Constraints on the design of neuromorphic circuits set by the properties of neural population codes [61.15277741147157]
  In the brain, information is encoded, transmitted and used to inform behaviour. Neuromorphic circuits need to encode information in a way compatible to that used by populations of neuron in the brain.
  arXiv  Detail & Related papers  (2022-12-08T15:16:04Z)
• Adapting Brain-Like Neural Networks for Modeling Cortical Visual Prostheses [68.96380145211093]
  Cortical prostheses are devices implanted in the visual cortex that attempt to restore lost vision by electrically stimulating neurons. Currently, the vision provided by these devices is limited, and accurately predicting the visual percepts resulting from stimulation is an open challenge. We propose to address this challenge by utilizing 'brain-like' convolutional neural networks (CNNs), which have emerged as promising models of the visual system.
  arXiv  Detail & Related papers  (2022-09-27T17:33:19Z)
• POPPINS : A Population-Based Digital Spiking Neuromorphic Processor with Integer Quadratic Integrate-and-Fire Neurons [50.591267188664666]
  We propose a population-based digital spiking neuromorphic processor in 180nm process technology with two hierarchy populations. The proposed approach enables the developments of biomimetic neuromorphic system and various low-power, and low-latency inference processing applications.
  arXiv  Detail & Related papers  (2022-01-19T09:26:34Z)
• Overcoming the Domain Gap in Neural Action Representations [60.47807856873544]
  3D pose data can now be reliably extracted from multi-view video sequences without manual intervention. We propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions.
  arXiv  Detail & Related papers  (2021-12-02T12:45:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.