When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk

• URL: http://arxiv.org/abs/2608.23780v2
• Date: Wed, 26 Aug 2026 18:36:32 GMT
• Title: When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk
• Abstract summary: We argue that measures of student talk are inadequate for teaching and learning, particularly for marginalized youth.<n>We employ multiple ethnographically-oriented methods to re-contextualize student conversations.
• Score: 4.210844064797028
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.

Related papers

• PeerMathDial: A Middle School Dialogue Dataset for Student Collaborative Math Problem Solving [10.652842345233559]
  Collaborative Problem Solving (CPS) is a core skill in education, where the process of peer interaction is highly important.<n>PeerMathDial is the first dataset of peer CPS dialogues collected from authentic middle school math classrooms.<n>It contains 55 dialogues from 27 students, totaling 6,406 turns.<n>Using the dataset and the dialogue act taxonomy, we demonstrate the practical applications of PeerMathDial across three use cases.
  arXiv  Detail & Related papers  (2026-06-19T15:55:18Z)
• Lived Experience in Dialogue: Co-designing Personalization in Large Language Models to Support Youth Mental Well-being [7.568079306155379]
  This study elicits community perspectives on how LLMs can facilitate more meaningful personalization to support youth mental well-being.<n>We identify three themes: person-centered contextualization responsive to momentary needs, explicit boundaries around scope and offline referral, and dialogic scaffolding for reflection and autonomy.<n>Our findings demonstrate how lived experience can be operationalized to inform design features in LLMs, which can enhance the alignment of LLM-based interventions with the realities of youth and their communities.
  arXiv  Detail & Related papers  (2025-11-07T23:53:36Z)
• LLMs syntactically adapt their language use to their conversational partner [58.92470092706263]
  It has been frequently observed that human speakers align their language use with each other during conversations.<n>We construct a corpus of conversations between large language models (LLMs) and find that two LLM agents end up making more similar syntactic choices as conversations go on.
  arXiv  Detail & Related papers  (2025-03-10T15:37:07Z)
• The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters [67.61587661660852]
  Theory-of-Mind (ToM) allows humans to understand and interpret the mental states of others.<n>In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM.<n>We introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels.
  arXiv  Detail & Related papers  (2025-01-03T09:04:45Z)
• Chain-of-MetaWriting: Linguistic and Textual Analysis of How Small Language Models Write Young Students Texts [0.8192907805418583]
  This paper introduces a fine-grained linguistic and textual analysis of multilingual Small Language Models' (SLMs) writing.<n>We mainly focused on short story and essay writing tasks in French for schoolchildren and undergraduate students respectively.
  arXiv  Detail & Related papers  (2024-12-19T15:58:53Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education.<n>Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs.<n>We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• Can LLMs Understand the Implication of Emphasized Sentences in Dialogue? [64.72966061510375]
  Emphasis is a crucial component in human communication, which indicates the speaker's intention and implication beyond pure text in dialogue. This paper introduces Emphasized-Talk, a benchmark with emphasis-annotated dialogue samples capturing the implications of emphasis. We evaluate various Large Language Models (LLMs), both open-source and commercial, to measure their performance in understanding emphasis.
  arXiv  Detail & Related papers  (2024-06-16T20:41:44Z)
• Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics [0.0]
  The advent of Large Language Models (LLMs) started a serious discussion among educators on how they would affect curricula, assessments, and students' competencies. This report presents an assignment within a course on professional competencies, including some related to ethics, that computing master's students need in their careers.
  arXiv  Detail & Related papers  (2024-04-09T09:03:47Z)
• Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models [51.75805497456226]
  This work focuses on the factual consistency issue with the help of the dialogue summarization task. Our evaluation shows that, on average, 26.8% of the summaries generated by LLMs contain factual inconsistency. To stimulate and enhance the dialogue comprehension ability of LLMs, we propose a fine-tuning paradigm with auto-constructed multi-task data.
  arXiv  Detail & Related papers  (2023-11-13T09:32:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.