TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

• URL: http://arxiv.org/abs/2605.30673v1
• Date: Fri, 29 May 2026 00:06:54 GMT
• Title: TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation
• Abstract summary: We present textitTeachObs, a human-validated benchmark for multimodal teaching observation in classroom videos.<n>textitTeachObs includes 30 public lesson videos from eight countries divided into 5,158 fixed 15-second scenes.
• Score: 0.12079404108544865
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Classroom videos contain observable teaching practices, but their pedagogical and visual signals are rarely organized in forms suitable for model evaluation. We present \textit{TeachObs}, a human-validated benchmark for multimodal teaching observation in classroom videos. \textit{TeachObs} includes 30 public lesson videos from eight countries divided into 5,158 fixed 15-second scenes. Seven researchers annotated each scene with 39 binary observation codes, covering 20 visual codes, such as gesture, board work, pointing, and visual materials, and 19 nonvisual codes, such as instruction, monitoring, questioning, feedback, and reflection. Gold segment labels are constructed using reliability- and prevalence-aware rules based on Krippendorff's alpha. In addition to segment-level labels, three expert raters produced lesson-level ratings and qualitative evaluations of instructional design, instructional delivery, learner response, learning materials, and lesson closure across the 30 lessons, with rater coverage detailed in the body. Using these two human reference layers, we evaluate five vision-capable frontier LLMs across three tracks - text-only segment coding, text + frame segment coding, and lesson-level coverage scored under an LLM-as-judge protocol - and find that no single model consistently outperforms others across all three tracks, that adding a mid-frame inflates both true and false attributions per scene, and that model evaluations over-rate procedurally clear lessons relative to expert raters. \textit{TeachObs} therefore supports both fine-grained annotation benchmarking and whole-lesson evaluation, showing where AI systems can assist classroom video analysis and where expert judgment remains necessary across varied subjects, classroom formats, and annotation difficulty levels.

Related papers

• Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios [61.926801508808616]
  We introduce Video-IFBench, a benchmark for evaluating instruction following in video understanding.<n>We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions.<n>We build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples.
  arXiv  Detail & Related papers  (2026-08-26T08:35:42Z)
• EduIllustrate: Towards Scalable Automated Generation Of Multimodal Educational Content [19.131221541276332]
  We present EduIllustrate, a benchmark for evaluating large language models.<n>The benchmark comprises 230 problems spanning five subjects and three grade levels.<n>We show that Gemini 3.0 Pro Preview leads at 87.8%, while Kimi-K2.5 achieves the best cost-efficiency.
  arXiv  Detail & Related papers  (2026-04-06T08:58:31Z)
• TeachPro: Multi-Label Qualitative Teaching Evaluation via Cross-View Graph Synergy and Semantic Anchored Evidence Encoding [12.647867305450289]
  We propose TeachPro, a multi-label learning framework that systematically assesses five key teaching dimensions.<n>We first propose a Dimension-Anchored Evidence, which integrates three core components.<n>We then propose a Cross-View Graph Synergy Network to represent student comments.
  arXiv  Detail & Related papers  (2026-01-14T07:27:57Z)
• Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data [8.014320244550243]
  This work explores AI-driven analysis of classroom recordings, focusing on multimodal instructional activity and discourse recognition.<n>Using a densely annotated dataset of 164 hours of video and 68 lesson transcripts, we design parallel, modality-specific pipelines.<n>Fine-tuned models consistently outperform prompting-based approaches, achieving macro-F1 scores of 0.577 for video and 0.460 for transcripts.
  arXiv  Detail & Related papers  (2025-11-26T11:57:22Z)
• Code2Video: A Code-centric Paradigm for Educational Video Generation [60.03043132859077]
  We propose Code2Video, a code-centric agent framework for generating educational videos via Python code.<n>The framework comprises three collaborative agents: (i) Planner, which structures lecture content into temporally coherent flows; (ii) Coder, which converts structured instructions into executable Python codes while incorporating scope-guided auto-fix to enhance efficiency; and (iii) Critic, which leverages vision-language models (VLM) with visual anchor prompts to refine spatial layout and ensure clarity.<n>Our results demonstrate the potential of Code2Video as a scalable, interpretable, and controllable approach, achieving 40% improvement over direct code
  arXiv  Detail & Related papers  (2025-10-01T17:56:48Z)
• VideoScore2: Think before You Score in Generative Video Evaluation [69.43069741467603]
  VideoScore2 is a multi-dimensional, interpretable, and human-aligned framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency.<n>Our model is trained on a large-scale dataset VideoFeedback2 containing 27,168 human-annotated videos.
  arXiv  Detail & Related papers  (2025-09-26T18:09:03Z)
• HumanVideo-MME: Benchmarking MLLMs for Human-Centric Video Understanding [120.84817886550765]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks involving both images and videos.<n>Existing human-centric benchmarks predominantly emphasize video generation quality and action recognition, while overlooking essential perceptual and cognitive abilities required in human-centered scenarios.<n>We propose a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric video understanding.
  arXiv  Detail & Related papers  (2025-07-07T11:52:24Z)
• Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach [7.273857543125784]
  Our study proposes a novel text-centered multimodal fusion architecture to assess the quality of three discourse components grounded in the Global Teaching InSights (GTI) observation protocol.<n>We employ attention mechanisms to capture inter- and intra-modal interactions from transcript, audio, and video streams.<n>Our results highlight the dominant role of text modality in approaching this task.
  arXiv  Detail & Related papers  (2025-05-12T09:24:21Z)
• Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT [7.273857543125784]
  Our work explores a multimodal approach to automatically estimating encouragement and warmth in classrooms. We employed facial and speech emotion recognition with sentiment analysis to extract interpretable features from video, audio, and transcript data. We demonstrated our approach on the GTI dataset, comprising 367 16-minute video segments from 92 authentic lesson recordings.
  arXiv  Detail & Related papers  (2024-04-01T16:58:09Z)
• Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation [55.429541407920304]
  Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature. Recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes. We introduce a multi-label meta-learning framework to deal with the biased predicate distribution.
  arXiv  Detail & Related papers  (2023-06-16T18:14:23Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)
• Multiview Pseudo-Labeling for Semi-supervised Learning from Video [102.36355560553402]
  We present a novel framework that uses complementary views in the form of appearance and motion information for semi-supervised learning in video. Our method capitalizes on multiple views, but it nonetheless trains a model that is shared across appearance and motion input. On multiple video recognition datasets, our method substantially outperforms its supervised counterpart, and compares favorably to previous work on standard benchmarks in self-supervised video representation learning.
  arXiv  Detail & Related papers  (2021-04-01T17:59:48Z)
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision [54.73758942064708]
  We teach machines to understand visuals and natural language by learning the mapping between sentences and noisy video snippets without explicit annotations. For training and evaluation, we contribute a new dataset ApartmenTour' that contains a large number of online videos and subtitles.
  arXiv  Detail & Related papers  (2020-11-19T03:43:56Z)
• Watch, read and lookup: learning to spot signs from multiple supervisors [99.50956498009094]
  Given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign language video. We train a model using multiple types of available supervision by: (1) watching existing sparsely labelled footage; (2) reading associated subtitles which provide additional weak-supervision; and (3) looking up words in visual sign language dictionaries. These three tasks are integrated into a unified learning framework using the principles of Noise Contrastive Estimation and Multiple Instance Learning.
  arXiv  Detail & Related papers  (2020-10-08T14:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.