Adapting Transformer Language Models for Predictive Typing in Brain-Computer Interfaces

• URL: http://arxiv.org/abs/2305.03819v1
• Date: Fri, 5 May 2023 19:47:41 GMT
• Title: Adapting Transformer Language Models for Predictive Typing in Brain-Computer Interfaces
• Authors: Shijia Liu, David A. Smith
• Abstract summary: This paper adapts several wordpiece-level Transformer LMs to make character predictions and evaluates them on typing tasks. GPT-2 fares best on clean text, but different LMs react differently to noisy histories.
• Score: 3.3961243538813837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Brain-computer interfaces (BCI) are an important mode of alternative and augmentative communication for many people. Unlike keyboards, many BCI systems do not display even the 26 letters of English at one time, let alone all the symbols in more complex systems. Using language models to make character-level predictions, therefore, can greatly speed up BCI typing (Ghosh and Kristensson, 2017). While most existing BCI systems employ character n-gram models or no LM at all, this paper adapts several wordpiece-level Transformer LMs to make character predictions and evaluates them on typing tasks. GPT-2 fares best on clean text, but different LMs react differently to noisy histories. We further analyze the effect of character positions in a word and context lengths.

Related papers

• American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM [0.0]
  American Sign Language is one of the sign languages used. Development of neural machine translation technology is moving towards sign language translation. Transformer became the state-of-the-art in natural language processing.
  arXiv  Detail & Related papers  (2024-09-17T04:00:33Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods mainly focus on aligning vision encoders with Multimodal Large Language Models (MLLMs) We introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Our method not only demonstrates state-of-the-art performance on multilingual MMBench and MMMB, but also excels across a broad range of multimodal tasks.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling [70.34758460372629]
  We introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. MYTE produces shorter encodings for all 99 analyzed languages. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
  arXiv  Detail & Related papers  (2024-03-15T21:21:11Z)
• Online Gesture Recognition using Transformer and Natural Language Processing [0.0]
  Transformer architecture is shown to provide a powerful machine framework for online gestures corresponding to glyph strokes of natural language sentences. Transformer architecture is shown to provide a powerful machine framework for online gestures corresponding to glyph strokes of natural language sentences.
  arXiv  Detail & Related papers  (2023-05-05T10:17:22Z)
• Alternative Input Signals Ease Transfer in Multilingual Machine Translation [21.088829932208945]
  We tackle inhibited transfer by augmenting the training data with alternative signals that unify different writing systems. We test these signals on Indic and Turkic languages, two language families where the writing systems differ but languages still share common features.
  arXiv  Detail & Related papers  (2021-10-15T01:56:46Z)
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization [50.16128796194463]
  We propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. We introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level.
  arXiv  Detail & Related papers  (2021-06-23T22:24:14Z)
• Revisiting Language Encoding in Learning Multilingual Representations [70.01772581545103]
  We propose a new approach called Cross-lingual Language Projection (XLP) to replace language embedding. XLP projects the word embeddings into language-specific semantic space, and then the projected embeddings will be fed into the Transformer model. Experiments show that XLP can freely and significantly boost the model performance on extensive multilingual benchmark datasets.
  arXiv  Detail & Related papers  (2021-02-16T18:47:10Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• Neural Machine Translation for Multilingual Grapheme-to-Phoneme Conversion [13.543705472805431]
  We present a single end-to-end trained neural G2P model that shares same encoder and decoder across multiple languages. We show 7.2% average improvement in phoneme error rate over low resource languages and no over high resource ones compared to monolingual baselines.
  arXiv  Detail & Related papers  (2020-06-25T06:16:29Z)
• Byte Pair Encoding is Suboptimal for Language Model Pretraining [49.30780227162387]
  We analyze differences between unigram LM tokenization and byte-pair encoding (BPE) We find that the unigram LM tokenization method matches or outperforms BPE across downstream tasks and two languages. We hope that developers of future pretrained LMs will consider adopting the unigram LM method over the more prevalent BPE.
  arXiv  Detail & Related papers  (2020-04-07T21:21:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.