Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

• URL: http://arxiv.org/abs/2106.03873v1
• Date: Mon, 7 Jun 2021 18:00:06 GMT
• Title: Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions
• Authors: Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, Tatsunori Hashimoto
• Abstract summary: In education, teachers' uptake of student contributions has been linked to higher student achievement. We propose a framework for measuring uptake, by releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts. We find that although repetition captures a significant part of uptake, pJSD outperforms repetition-based baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation.
• Score: 19.80258498803113
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In education, teachers' uptake of student contributions has been linked to higher student achievement. Yet measuring and improving teachers' uptake at scale is challenging, as existing methods require expensive annotation by experts. We propose a framework for computationally measuring uptake, by (1) releasing a dataset of student-teacher exchanges extracted from US math classroom transcripts annotated for uptake by experts; (2) formalizing uptake as pointwise Jensen-Shannon Divergence (pJSD), estimated via next utterance classification; (3) conducting a linguistically-motivated comparison of different unsupervised measures and (4) correlating these measures with educational outcomes. We find that although repetition captures a significant part of uptake, pJSD outperforms repetition-based baselines, as it is capable of identifying a wider range of uptake phenomena like question answering and reformulation. We apply our uptake measure to three different educational datasets with outcome indicators. Unlike baseline measures, pJSD correlates significantly with instruction quality in all three, providing evidence for its generalizability and for its potential to serve as an automated professional development tool for teachers.

Related papers

• Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference [82.28542500317445]
  We present an end-to-end learning framework, termed Sequential Posterior Inference (SPI), capable of selecting knowledge and generating dialogues. Unlike other methods, SPI does not require the inference network or assume a simple geometry of the posterior distribution.
  arXiv  Detail & Related papers  (2023-06-01T21:23:13Z)
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems [74.73881579517055]
  We propose a framework to generate such dialogues by pairing human teachers with a Large Language Model prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues.
  arXiv  Detail & Related papers  (2023-05-23T21:44:56Z)
• Dual Semantic Knowledge Composed Multimodal Dialog Systems [114.52730430047589]
  We propose a novel multimodal task-oriented dialog system named MDS-S2. It acquires the context related attribute and relation knowledge from the knowledge base. We also devise a set of latent query variables to distill the semantic information from the composed response representation.
  arXiv  Detail & Related papers  (2023-05-17T06:33:26Z)
• Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue [92.01165203498299]
  Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. This paper argues that imitation learning (IL) and related low-level metrics are actually misleading and do not align with the goals of embodied dialogue research.
  arXiv  Detail & Related papers  (2022-10-10T05:51:40Z)
• Computationally Identifying Funneling and Focusing Questions in Classroom Discourse [24.279653100481863]
  We propose the task of computationally detecting funneling and focusing questions in classroom discourse. We release an annotated dataset of 2,348 teacher utterances labeled for funneling and focusing questions, or neither. Our best model, a supervised RoBERTa model fine-tuned on our dataset, has a strong linear correlation of.76 with human expert labels and with positive educational outcomes.
  arXiv  Detail & Related papers  (2022-07-08T01:28:29Z)
• The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues [5.424153769988429]
  This paper reports on a first attempt at an AI teacher test. We built a solution around the insight that you can run conversational agents in parallel to human teachers in real-world dialogues. Our method builds on the reliability of comparative judgments in education and uses a probabilistic model and Bayesian sampling to infer estimates of pedagogical ability.
  arXiv  Detail & Related papers  (2022-05-16T09:36:30Z)
• Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering [29.545937716796082]
  Spoken question answering (SQA) requires fine-grained understanding of both spoken documents and questions. We propose novel training schemes for spoken question answering with a self-supervised training stage and a contrastive representation learning stage. Our model achieves state-of-the-art results on three SQA benchmarks.
  arXiv  Detail & Related papers  (2021-09-08T01:13:14Z)
• Mixtures of Deep Neural Experts for Automated Speech Scoring [11.860560781894458]
  The paper copes with the task of automatic assessment of second language proficiency from the language learners' spoken responses to test prompts. The approach relies on two separate modules: (1) an automatic speech recognition system that yields text transcripts of the spoken interactions involved, and (2) a multiple classifier system based on deep learners that ranks the transcripts into proficiency classes.
  arXiv  Detail & Related papers  (2021-06-23T15:44:50Z)
• Structural Pre-training for Dialogue Comprehension [51.215629336320305]
  We present SPIDER, Structural Pre-traIned DialoguE Reader, to capture dialogue exclusive features. To simulate the dialogue-like features, we propose two training objectives in addition to the original LM objectives. Experimental results on widely used dialogue benchmarks verify the effectiveness of the newly introduced self-supervised tasks.
  arXiv  Detail & Related papers  (2021-05-23T15:16:54Z)
• Probing Task-Oriented Dialogue Representation from Language Models [106.02947285212132]
  This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way.
  arXiv  Detail & Related papers  (2020-10-26T21:34:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.