Brain-Language-Action (BLA) Models: Language-Conditioned EEG for Robotics Control

• URL: http://arxiv.org/abs/2608.28967v1
• Date: Sat, 29 Aug 2026 00:37:52 GMT
• Title: Brain-Language-Action (BLA) Models: Language-Conditioned EEG for Robotics Control
• Abstract summary: We introduce Brain-Language-Action models, a framework in which language conditions the interpretation of neural representations for robotic action generation.<n>In a BLA, a small set of reliably distinguishable brain states can be dynamically associated with different actions through a language-defined control mapping.<n>We develop a proof-of-concept BLA for drone control using motor-imagery EEG from the BCI Competition IV 2a dataset.
• Score: 51.56484100374058
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Electroencephalography (EEG)-based robotic control is commonly formulated as a direct classification problem, in which electrical neural signals are mapped to a fixed set of discrete actions. However, the limited separability and high noise of EEG signals make it difficult to scale this approach to fine-grained robotic control spaces. We introduce Brain-Language-Action (BLA) models, a framework in which language conditions the interpretation of neural representations for robotic action generation. In a BLA, a small set of reliably distinguishable brain states can be dynamically associated with different actions through a language-defined control mapping, allowing a small number of neural classes to apply to a larger global action space. We develop a proof-of-concept BLA for drone control using motor-imagery EEG from the BCI Competition IV 2a dataset. The system is trained in two stages. First, we evaluate multiple candidate EEG encoder architectures using subject-specific four-class motor-imagery classification, converting 250Hz, 3.5-second, 22-channel EEG samples into five 128-dimensional brain-token embeddings. Second, these embeddings are projected into the embedding space of a pretrained large language model (LLM) and jointly fine-tuned with language instructions to autoregressively generate structured three-token drone actions. Across 840 possible language-defined mappings between four neural states and seven flight action combinations, the resulting BLA achieves 90% per-token accuracy during evaluation. These results provide an initial demonstration that language conditioning can expand the effective control range of EEG-based robotic interfaces without requiring a corresponding increase in the number of directly distinguishable neural states.

Related papers

• SENSE: Efficient EEG-to-Text via Privacy-Preserving Semantic Retrieval [6.012652594597545]
  Decoding brain activity into natural language is a major challenge in AI with important applications in assistive communication, neurotechnology, and human-computer interaction.<n>Most existing Brain-Computer Interface (BCI) approaches rely on memory-intensive fine-tuning of Large Language Models (LLMs) or encoder-decoder models on raw EEG signals.<n>We introduce SENSE, a lightweight and privacy-preserving framework that translates non-invasive electroencephalography (EEG) into text without LLM fine-tuning.
  arXiv  Detail & Related papers  (2026-03-17T19:59:41Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• A Brain Wave Encodes a Thousand Tokens: Modeling Inter-Cortical Neural Interactions for Effective EEG-based Emotion Recognition [0.41998444721319217]
  We propose RBTransformer, a Transformer-based neural network architecture that models inter-cortical neural dynamics of the brain in latent space.<n>We conducted extensive experiments, specifically under subject-dependent settings, on the SEED, DEAP, and DREAMER datasets.<n>The results demonstrate that the proposed RBTransformer outperforms all previous state-of-the-art methods across all three datasets.
  arXiv  Detail & Related papers  (2025-11-17T22:27:12Z)
• BrainLLM: Generative Language Decoding from Brain Recordings [77.66707255697706]
  We propose a generative language BCI that utilizes the capacity of a large language model and a semantic brain decoder.<n>The proposed model can generate coherent language sequences aligned with the semantic content of visual or auditory language stimuli.<n>Our findings demonstrate the potential and feasibility of employing BCIs in direct language generation.
  arXiv  Detail & Related papers  (2023-11-16T13:37:21Z)
• BELT:Bootstrapping Electroencephalography-to-Language Decoding and Zero-Shot Sentiment Classification by Natural Language Supervision [31.382825932199935]
  The proposed BELT method is a generic and efficient framework that bootstraps EEG representation learning. With a large LM's capacity for understanding semantic information and zero-shot generalization, BELT utilizes large LMs trained on Internet-scale datasets. We achieve state-of-the-art results on two featuring brain decoding tasks including the brain-to-language translation and zero-shot sentiment classification.
  arXiv  Detail & Related papers  (2023-09-21T13:24:01Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers [71.76680102779765]
  Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. We propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
  arXiv  Detail & Related papers  (2022-11-05T04:03:55Z)
• Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification [78.120927891455]
  State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines.
  arXiv  Detail & Related papers  (2021-12-05T21:57:22Z)
• End-to-end translation of human neural activity to speech with a dual-dual generative adversarial network [39.014888541156296]
  We propose an end-to-end model to translate human neural activity to speech directly. We create a new electroencephalogram (EEG) datasets for participants with good attention. The proposed method can translate word-length and sentence-length sequences of neural activity to speech.
  arXiv  Detail & Related papers  (2021-10-13T10:54:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.