Multimodal Classification of Teaching Activities from University Lecture Recordings

• URL: http://arxiv.org/abs/2312.17262v1
• Date: Sun, 24 Dec 2023 08:33:30 GMT
• Title: Multimodal Classification of Teaching Activities from University Lecture Recordings
• Authors: Oscar Sapena and Eva Onaindia
• Abstract summary: We present a multimodal classification algorithm that identifies the type of activity that is being carried out at any time of the lesson. Some academic activities are more easily identifiable with the audio signal while resorting to the text transcription is needed to identify others.
• Score: 0.9790236766474201
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The way of understanding online higher education has greatly changed due to the worldwide pandemic situation. Teaching is undertaken remotely, and the faculty incorporate lecture audio recordings as part of the teaching material. This new online teaching-learning setting has largely impacted university classes. While online teaching technology that enriches virtual classrooms has been abundant over the past two years, the same has not occurred in supporting students during online learning. {To overcome this limitation, our aim is to work toward enabling students to easily access the piece of the lesson recording in which the teacher explains a theoretical concept, solves an exercise, or comments on organizational issues of the course. To that end, we present a multimodal classification algorithm that identifies the type of activity that is being carried out at any time of the lesson by using a transformer-based language model that exploits features from the audio file and from the automated lecture transcription. The experimental results will show that some academic activities are more easily identifiable with the audio signal while resorting to the text transcription is needed to identify others. All in all, our contribution aims to recognize the academic activities of a teacher during a lesson.

Related papers

• Awaking the Slides: A Tuning-free and Knowledge-regulated AI Tutoring System via Language Model Coordination [52.20542825755132]
  We develop Slide2Lecture, a tuning-free and knowledge-regulated intelligent tutoring system. It can effectively convert an input lecture slide into a structured teaching agenda consisting of a set of heterogeneous teaching actions. For teachers and developers, Slide2Lecture enables customization to cater to personalized demands.
  arXiv  Detail & Related papers  (2024-09-11T16:03:09Z)
• Intelligent Interface: Enhancing Lecture Engagement with Didactic Activity Summaries [0.054204929130712134]
  The prototype utilizes machine learning-based techniques to recognise selected didactic and behavioural teachers' features within lecture video recordings. The system offers flexibility for (future) integration of new/additional machine-learning models and software modules for image and video analysis.
  arXiv  Detail & Related papers  (2024-06-20T12:45:23Z)
• Audio Representation Learning by Distilling Video as Privileged Information [25.71206255965502]
  We propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. We adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI) We show considerable improvements over sole audio-based recognition as well as prior works that use LUPI.
  arXiv  Detail & Related papers  (2023-02-06T15:09:34Z)
• Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides [57.86931911522967]
  We test the capabilities of machine learning models in multimodal understanding of educational content. Our dataset contains aligned slides and spoken language, for 180+ hours of video and 9000+ slides, with 10 lecturers from various subjects. We introduce PolyViLT, a multimodal transformer trained with a multi-instance learning loss that is more effective than current approaches.
  arXiv  Detail & Related papers  (2022-08-17T05:30:18Z)
• The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves [8.090330715662962]
  This paper describes the TalkMoves dataset, composed of 567 human-annotated K-12 mathematics lesson transcripts. The dataset can be used by educators, policymakers, and researchers to understand the nature of teacher and student discourse in K-12 math classrooms.
  arXiv  Detail & Related papers  (2022-04-06T18:12:30Z)
• Real-time Attention Span Tracking in Online Education [0.0]
  This paper intends to provide a mechanism that uses the camera feed and microphone input to monitor the real-time attention level of students during online classes. We propose a system that uses five distinct non-verbal features to calculate the attention score of the student during computer based tasks and generate real-time feedback for both students and the organization.
  arXiv  Detail & Related papers  (2021-11-29T17:05:59Z)
• Iterative Teacher-Aware Learning [136.05341445369265]
  In human pedagogy, teachers and students can interact adaptively to maximize communication efficiency. We propose a gradient optimization based teacher-aware learner who can incorporate teacher's cooperative intention into the likelihood function.
  arXiv  Detail & Related papers  (2021-10-01T00:27:47Z)
• Seminar Learning for Click-Level Weakly Supervised Semantic Segmentation [149.9226057885554]
  We propose seminar learning, a new learning paradigm for semantic segmentation with click-level supervision. The rationale of seminar learning is to leverage the knowledge from different networks to compensate for insufficient information provided in click-level annotations. Experimental results demonstrate the effectiveness of seminar learning, which achieves the new state-of-the-art performance of 72.51%.
  arXiv  Detail & Related papers  (2021-08-30T17:27:43Z)
• Interactive Knowledge Distillation [79.12866404907506]
  We propose an InterActive Knowledge Distillation scheme to leverage the interactive teaching strategy for efficient knowledge distillation. In the distillation process, the interaction between teacher and student networks is implemented by a swapping-in operation. Experiments with typical settings of teacher-student networks demonstrate that the student networks trained by our IAKD achieve better performance than those trained by conventional knowledge distillation methods.
  arXiv  Detail & Related papers  (2020-07-03T03:22:04Z)
• Neural Multi-Task Learning for Teacher Question Detection in Online Classrooms [50.19997675066203]
  We build an end-to-end neural framework that automatically detects questions from teachers' audio recordings. By incorporating multi-task learning techniques, we are able to strengthen the understanding of semantic relations among different types of questions.
  arXiv  Detail & Related papers  (2020-05-16T02:17:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.