Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models

• URL: http://arxiv.org/abs/2502.10378v1
• Date: Fri, 14 Feb 2025 18:57:04 GMT
• Title: Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models
• Authors: Jiexin Ding, Bowen Zhao, Yuntao Wang, Xinyun Liu, Rui Hao, Ishan Chatterjee, Yuanchun Shi,
• Abstract summary: This paper presents EyeLingo, a transformer-based machine learning method that predicts the probability of unknown words based on text content and eye gaze trajectory in real time with high accuracy. A 20-participant user study revealed that our method can achieve an accuracy of 97.6%, and an F1-score of 71.1%.
• Score: 24.607431783798425
• License:
• Abstract: English as a Second Language (ESL) learners often encounter unknown words that hinder their text comprehension. Automatically detecting these words as users read can enable computing systems to provide just-in-time definitions, synonyms, or contextual explanations, thereby helping users learn vocabulary in a natural and seamless manner. This paper presents EyeLingo, a transformer-based machine learning method that predicts the probability of unknown words based on text content and eye gaze trajectory in real time with high accuracy. A 20-participant user study revealed that our method can achieve an accuracy of 97.6%, and an F1-score of 71.1%. We implemented a real-time reading assistance prototype to show the effectiveness of EyeLingo. The user study shows improvement in willingness to use and usefulness compared to baseline methods.

Related papers

• Storyfier: Exploring Vocabulary Learning Support with Text Generation Models [52.58844741797822]
  We develop Storyfier to provide a coherent context for any target words of learners' interests. learners generally favor the generated stories for connecting target words and writing assistance for easing their learning workload. In read-cloze-write learning sessions, participants using Storyfier perform worse in recalling and using target words than learning with a baseline tool without our AI features.
  arXiv  Detail & Related papers  (2023-08-07T18:25:00Z)
• Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition [6.47452771256903]
  We take inspiration from how human babies acquire their first language, and developed a computational process for word acquisition through comparative learning. Motivated by cognitive findings, we generated a small dataset that enables the computation models to compare the similarities and differences of various attributes. We frame the acquisition of words as not only the information filtration process, but also as representation-symbol mapping.
  arXiv  Detail & Related papers  (2023-07-05T19:38:04Z)
• GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners [24.009130595261123]
  We propose GazeReader, an unknown word detection method only using a webcam. GazeReader tracks the learner's gaze and then applies a transformer-based machine learning model that encodes the text information to locate the unknown word.
  arXiv  Detail & Related papers  (2023-03-18T15:55:49Z)
• Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models [70.82705830137708]
  We introduce Data-driven Instruction Augmentation for Language-conditioned control (DIAL) We utilize semi-language labels leveraging the semantic understanding of CLIP to propagate knowledge onto large datasets of unlabelled demonstration data. DIAL enables imitation learning policies to acquire new capabilities and generalize to 60 novel instructions unseen in the original dataset.
  arXiv  Detail & Related papers  (2022-11-21T18:56:00Z)
• Unravelling Interlanguage Facts via Explainable Machine Learning [10.71581852108984]
  We focus on the internals of an NLI classifier trained by an emphexplainable machine learning algorithm. We use this perspective in order to tackle both NLI and a companion task, guessing whether a text has been written by a native or a non-native speaker. We investigate which kind of linguistic traits are most effective for solving our two tasks, namely, are most indicative of a speaker's L1.
  arXiv  Detail & Related papers  (2022-08-02T14:05:15Z)
• An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications [11.775048147405725]
  We develop models for estimating contextual informativeness, focusing on the instructional aspect of sentences. We show how our model identifies key contextual elements in a sentence that are likely to contribute most to a reader's understanding of the target word. We believe our results open new possibilities for applications that support language learning for both human and machine learners.
  arXiv  Detail & Related papers  (2022-04-21T05:17:49Z)
• Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer [61.34424171458634]
  We study whether integrating visual knowledge into a language model can fill the gap. Our experiments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings.
  arXiv  Detail & Related papers  (2022-03-14T22:02:40Z)
• Pedagogical Word Recommendation: A novel task and dataset on personalized vocabulary acquisition for L2 learners [4.507860128918788]
  We propose and release data for a novel task called Pedagogical Word Recommendation. The main goal of PWR is to predict whether a given learner knows a given word based on other words the learner has already seen. As a feature of this ITS, students can directly indicate words they do not know from the questions they solved to create wordbooks.
  arXiv  Detail & Related papers  (2021-12-27T17:52:48Z)
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer [76.3906723777229]
  We present VidLanKD, a video-language knowledge distillation method for improving language understanding. We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset. In our experiments, VidLanKD achieves consistent improvements over text-only language models and vokenization models.
  arXiv  Detail & Related papers  (2021-07-06T15:41:32Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• Word Sense Disambiguation for 158 Languages using Word Embeddings Only [80.79437083582643]
  Disambiguation of word senses in context is easy for humans, but a major challenge for automatic approaches. We present a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory. We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings.
  arXiv  Detail & Related papers  (2020-03-14T14:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.