Evaluating LLM-Generated Lessons from the Language Learning Students' Perspective: A Short Case Study on Duolingo

• URL: http://arxiv.org/abs/2603.18873v2
• Date: Sun, 22 Mar 2026 14:33:22 GMT
• Title: Evaluating LLM-Generated Lessons from the Language Learning Students' Perspective: A Short Case Study on Duolingo
• Abstract summary: We surveyed five employees from a multinational company in the Philippines on their experiences with Duolingo.<n>Results show that respondents encountered general scenarios more frequently than work-related ones.<n>The latter helps bridge the gap toward professional fluency as it contains domain-specific vocabulary.
• Score: 0.07846698376004395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Popular language learning applications such as Duolingo use large language models (LLMs) to generate lessons for its users. Most lessons focus on general real-world scenarios such as greetings, ordering food, or asking directions, with limited support for profession-specific contexts. This gap can hinder learners from achieving professional-level fluency, which we define as the ability to communicate comfortably various work-related and domain-specific information in the target language. We surveyed five employees from a multinational company in the Philippines on their experiences with Duolingo. Results show that respondents encountered general scenarios more frequently than work-related ones, and that the former are relatable and effective in building foundational grammar, vocabulary, and cultural knowledge. The latter helps bridge the gap toward professional fluency as it contains domain-specific vocabulary. Each participant suggested lesson scenarios that diverge in contexts when analyzed in aggregate. With this understanding, we propose that language learning applications should generate lessons that adapt to an individual's needs through personalized, domain specific lesson scenarios while maintaining foundational support through general, relatable lesson scenarios.

Related papers

• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation [50.731838216268706]
  We study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus.<n>We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers.<n>We propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers.
  arXiv  Detail & Related papers  (2026-05-28T10:12:41Z)
• LangGPS: Language Separability Guided Data Pre-Selection for Joint Multilingual Instruction Tuning [49.22807995935406]
  Joint multilingual instruction tuning is a widely adopted approach to improve the multilingual instruction-following ability and downstream performance of large language models (LLMs)<n>Existing selection methods, often based on features like text quality, diversity, or task relevance, typically overlook the intrinsic linguistic structure of multilingual data.<n>We propose LangGPS, a lightweight two-stage pre-selection framework guided by language separability which quantifies how well samples in different languages can be distinguished in the model's representation space.
  arXiv  Detail & Related papers  (2025-11-13T12:02:32Z)
• Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set [28.592959007943538]
  This work investigates whether large language models (LLMs) capture discourse knowledge that generalizes across languages and frameworks.<n>Using multilingual discourse relation classification as a testbed, we examine a comprehensive set of 23 LLMs of varying sizes and multilingual capabilities.<n>Our results show that LLMs, especially those with multilingual training corpora, can generalize discourse information across languages and frameworks.
  arXiv  Detail & Related papers  (2025-03-13T16:20:25Z)
• How Do Multilingual Language Models Remember Facts? [50.13632788453612]
  We show that previously identified recall mechanisms in English largely apply to multilingual contexts.<n>We localize the role of language during recall, finding that subject enrichment is language-independent.<n>In decoder-only LLMs, FVs compose these two pieces of information in two separate stages.
  arXiv  Detail & Related papers  (2024-10-18T11:39:34Z)
• A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers [51.8203871494146]
  The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing.<n>Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient.<n>This survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.
  arXiv  Detail & Related papers  (2024-05-17T17:47:39Z)
• Spoken Language Intelligence of Large Language Models for Language Learning [3.1964044595140217]
  We focus on evaluating the efficacy of large language models (LLMs) in the realm of education.<n>We introduce a new multiple-choice question dataset to evaluate the effectiveness of LLMs in the aforementioned scenarios.<n>We also investigate the influence of various prompting techniques such as zero- and few-shot method.<n>We find that models of different sizes have good understanding of concepts in phonetics, phonology, and second language acquisition, but show limitations in reasoning for real-world problems.
  arXiv  Detail & Related papers  (2023-08-28T12:47:41Z)
• Cross-lingual Lifelong Learning [53.06904052325966]
  We present a principled Cross-lingual Continual Learning (CCL) evaluation paradigm. We provide insights into what makes multilingual sequential learning particularly challenging. The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata.
  arXiv  Detail & Related papers  (2022-05-23T09:25:43Z)
• Phylogeny-Inspired Adaptation of Multilingual Models to New Languages [43.62238334380897]
  We show how we can use language phylogenetic information to improve cross-lingual transfer leveraging closely related languages. We perform adapter-based training on languages from diverse language families (Germanic, Uralic, Tupian, Uto-Aztecan) and evaluate on both syntactic and semantic tasks.
  arXiv  Detail & Related papers  (2022-05-19T15:49:19Z)
• Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification [73.5497360800395]
  We develop an end-to-end system that supports multiple languages. We exploit knowledge from a pre-trained multi-lingual natural language processing model.
  arXiv  Detail & Related papers  (2021-09-28T04:43:11Z)
• Zero-Shot Cross-Lingual Transfer with Meta Learning [45.29398184889296]
  We consider the setting of training models on multiple languages at the same time, when little or no data is available for languages other than English. We show that this challenging setup can be approached using meta-learning. We experiment using standard supervised, zero-shot cross-lingual, as well as few-shot cross-lingual settings for different natural language understanding tasks.
  arXiv  Detail & Related papers  (2020-03-05T16:07:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.