SmartPhone: Exploring Keyword Mnemonic with Auto-generated Verbal and Visual Cues

• URL: http://arxiv.org/abs/2305.10436v1
• Date: Thu, 11 May 2023 20:58:10 GMT
• Title: SmartPhone: Exploring Keyword Mnemonic with Auto-generated Verbal and Visual Cues
• Authors: Jaewook Lee and Andrew Lan
• Abstract summary: We propose an end-to-end pipeline for auto-generating verbal and visual cues for keyword mnemonics. Our approach, an end-to-end pipeline for auto-generating verbal and visual cues, can automatically generate highly memorable cues.
• Score: 2.8047215329139976
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In second language vocabulary learning, existing works have primarily focused on either the learning interface or scheduling personalized retrieval practices to maximize memory retention. However, the learning content, i.e., the information presented on flashcards, has mostly remained constant. Keyword mnemonic is a notable learning strategy that relates new vocabulary to existing knowledge by building an acoustic and imagery link using a keyword that sounds alike. Beyond that, producing verbal and visual cues associated with the keyword to facilitate building these links requires a manual process and is not scalable. In this paper, we explore an opportunity to use large language models to automatically generate verbal and visual cues for keyword mnemonics. Our approach, an end-to-end pipeline for auto-generating verbal and visual cues, can automatically generate highly memorable cues. We investigate the effectiveness of our approach via a human participant experiment by comparing it with manually generated cues.

Related papers

• Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data [84.01401439030265]
  Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) We present a simple yet effective automatic process for creating speech-text pair data. Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data.
  arXiv  Detail & Related papers  (2024-09-30T07:01:21Z)
• Exploring Automated Keyword Mnemonics Generation with Large Language Models via Overgenerate-and-Rank [4.383205675898942]
  Keywords mnemonics are a technique for memorizing vocabulary through memorable associations with a target word via a verbal cue. We propose a novel overgenerate-and-rank method via prompting large language models to generate verbal cues. Results show that LLM-generated mnemonics are comparable to human-generated ones in terms of imageability, coherence, and perceived usefulness.
  arXiv  Detail & Related papers  (2024-09-21T00:00:18Z)
• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• Storyfier: Exploring Vocabulary Learning Support with Text Generation Models [52.58844741797822]
  We develop Storyfier to provide a coherent context for any target words of learners' interests. learners generally favor the generated stories for connecting target words and writing assistance for easing their learning workload. In read-cloze-write learning sessions, participants using Storyfier perform worse in recalling and using target words than learning with a baseline tool without our AI features.
  arXiv  Detail & Related papers  (2023-08-07T18:25:00Z)
• Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition [6.47452771256903]
  We take inspiration from how human babies acquire their first language, and developed a computational process for word acquisition through comparative learning. Motivated by cognitive findings, we generated a small dataset that enables the computation models to compare the similarities and differences of various attributes. We frame the acquisition of words as not only the information filtration process, but also as representation-symbol mapping.
  arXiv  Detail & Related papers  (2023-07-05T19:38:04Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)
• Short-Term Word-Learning in a Dynamically Changing Environment [63.025297637716534]
  We show how to supplement an end-to-end ASR system with a word/phrase memory and a mechanism to access this memory to recognize the words and phrases correctly. We demonstrate significant improvements in the detection rate of new words with only a minor increase in false alarms.
  arXiv  Detail & Related papers  (2022-03-29T10:05:39Z)
• Spell my name: keyword boosted speech recognition [25.931897154065663]
  uncommon words such as names and technical terminology are important to understanding conversations in context. We propose a simple but powerful ASR decoding method that can better recognise these uncommon keywords. The method boosts the probabilities of given keywords in a beam search based on acoustic model predictions. We demonstrate the effectiveness of our method on the LibriSpeeech test sets and also internal data of real-world conversations.
  arXiv  Detail & Related papers  (2021-10-06T14:16:57Z)
• Broccoli: Sprinkling Lightweight Vocabulary Learning into Everyday Information Diets [3.305377595864778]
  Broccoli is a new paradigm aimed at reducing the required effort by seamlessly embedding vocabulary learning into users' everyday information diets. We find that the efficacy of the lightweight Broccoli approach is competitive with traditional, memorization-based vocabulary learning.
  arXiv  Detail & Related papers  (2021-04-16T07:38:05Z)
• On Vocabulary Reliance in Scene Text Recognition [79.21737876442253]
  Methods perform well on images with words within vocabulary but generalize poorly to images with words outside vocabulary. We call this phenomenon "vocabulary reliance" We propose a simple yet effective mutual learning strategy to allow models of two families to learn collaboratively.
  arXiv  Detail & Related papers  (2020-05-08T11:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.