Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization

• URL: http://arxiv.org/abs/2606.21861v1
• Date: Sat, 20 Jun 2026 03:53:40 GMT
• Title: Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization
• Abstract summary: We present a systematic benchmark that evaluates five widely-used Vision-Language Models (VLMs)<n>Our experiments reveal three primary failure modes of zero-shot VLMs for engagement recognition.
• Score: 0.44787896002954924
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated classroom engagement recognition holds substantial promise for scalable learning analytics, yet the suitability of modern Vision-Language Models (VLMs) for this task under zero-shot conditions remains largely unexplored. We present a systematic benchmark that evaluates five widely-used VLMs: CLIP, BLIP-VQA, GPT-4o, LLaVA-1.5-7B, and Qwen2.5VL-7B-Instruct across two complementary educational datasets: DAiSEE, an individual-student video dataset (300 sampled test clips), and the Student Classroom Behaviour dataset (SCB, 1,168 scene-level images). Each model is probed with three prompt variants spanning minimal, rubric-anchored, and chain-of-thought designs. Our experiments reveal three primary failure modes of zero-shot VLMs for engagement recognition: (1) near-random performance on individual students, with Cohen's kappa never exceeding 0.10 on DAiSEE; (2) severe class collapse, where models assign 85-100% of predictions to a single engagement level regardless of visual content; and (3) extreme prompt sensitivity, with accuracy swings of up to 32 percentage points on identical images depending solely on prompt phrasing. Remarkably, scene-level classification on SCB is substantially more tractable: CLIP and GPT-4o achieve kappa approximately 0.60 when prompted with behaviorally-grounded rubrics. We also document a practical barrier for deployment: GPT-4o's safety filters reject 98% of chain-of-thought requests involving individual student faces. Our findings provide a calibrated baseline and surface critical design considerations for the use of VLMs in educational observation systems.

Related papers

• Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education [0.3686530147760243]
  This study evaluates vision-language models for handwritten outcome-based assessment.<n>We compare 20 configurations spanning Qwen2.5-VL, InternVL3, Pixtral, a Donut baseline, and a cascade ensemble.
  arXiv  Detail & Related papers  (2026-09-13T04:55:55Z)
• The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence [1.7844382164707184]
  We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI)<n>We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T08:01:57Z)
• AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals [65.71225905458182]
  Self-distillation enables language models to learn on-policy from their own trajectories by using the same model as both student and teacher.<n>This setup provides dense token-level feedback without relying on a separate external model.<n>We introduce AVSD, a novel method of self-distillation with multiple privileged-information views.
  arXiv  Detail & Related papers  (2026-05-20T03:06:36Z)
• TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models [76.92890872338235]
  Large-scale pre-trained Vision-Language models (VLMs) exhibit strong zero-shot generalization, yet are highly vulnerable to imperceptible adversarial perturbations.<n>To enhance robustness without requiring downstream task-specific retraining, we propose TAME, a novel test-time defense.
  arXiv  Detail & Related papers  (2026-05-17T18:07:08Z)
• Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions [11.509375725817195]
  We propose a two-stage knowledge distillation framework that mines high-value samples from existing data.<n>With augmented training on only 10.30% of filtered samples, we achieve MAP@3 of 0.9585 (+17.8%) on the MAP-Charting dataset.<n>Using only a 4B parameter model, we attain 84.38% accuracy on cross-topic tests of middle school algebra misconception benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T12:17:38Z)
• LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models [9.095828883716907]
  Ghost-100 is a benchmark of 800 synthetically generated images spanning eight categories across three task families.<n>Every image is paired with five prompts drawn from a structured 5-Level Prompt Intensity Framework.<n>We adopt a dual-track evaluation protocol: a rule-based H-Rate measuring the proportion of responses in which a model crosses from grounded refusal into unsupported positive commitment.
  arXiv  Detail & Related papers  (2026-04-20T20:21:27Z)
• CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks [53.88194225946438]
  Chain-of-Thought for Detection (CoT4Det) is a simple but efficient strategy that reformulates perception tasks into three interpretable steps.<n>We show that CoT4Det significantly improves perception performance without compromising general vision language capabilities.
  arXiv  Detail & Related papers  (2025-12-07T05:26:30Z)
• Vision Large Language Models Are Good Noise Handlers in Engagement Analysis [54.397912827957164]
  We propose a framework leveraging Vision Large Language Models (VLMs) to refine annotations and guide the training process.<n>Our framework uses a questionnaire to extract behavioral cues and split data into high- and low-reliability subsets.<n>We demonstrate that classical computer vision models trained on refined high-reliability subsets and enhanced with our curriculum strategy show improvements.
  arXiv  Detail & Related papers  (2025-11-18T18:50:26Z)
• Bias-Aware Machine Unlearning: Towards Fairer Vision Models via Controllable Forgetting [3.1959623025848405]
  Deep neural networks often rely on spurious correlations in training data, leading to biased or unfair predictions in safety-critical domains such as medicine and autonomous driving.<n>Recent advances in machine unlearning provide a promising alternative for post-hoc model correction.
  arXiv  Detail & Related papers  (2025-09-09T07:25:51Z)
• DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection [72.25697820290502]
  This work introduces a straightforward and efficient strategy to identify potential novel classes through zero-shot classification. We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training. Empirical evaluations on three datasets, including LVIS, V3Det, and COCO, demonstrate significant improvements over the baseline performance.
  arXiv  Detail & Related papers  (2023-10-02T17:52:24Z)
• Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback [9.51494089949975]
  Large Language Models (LLMs) can be used to estimate Instructional Support'' domain scores of the CLassroom Assessment Scoring System (CLASS) We design a machine learning architecture that uses either zero-shot prompting of MetaPearson's Llama2, and/or a classic Bag of Words (BoW) model, to classify individual utterances of teachers' speech. These utterance-level judgments are aggregated over a 15-min observation session to estimate a global CLASS score.
  arXiv  Detail & Related papers  (2023-10-02T12:11:17Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• Single-Class Target-Specific Attack against Interpretable Deep Learning Systems [14.453881413188455]
  Single-class target-specific Adversa attack called SingleADV. We present a novel Single-class target-specific Adversa attack called SingleADV.
  arXiv  Detail & Related papers  (2023-07-12T23:07:06Z)
• CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition [52.66360172784038]
  We propose a clustering-based model, which considers all training samples at once, instead of optimizing for each instance individually. We call the proposed method CLASTER and observe that it consistently improves over the state-of-the-art in all standard datasets.
  arXiv  Detail & Related papers  (2021-01-18T12:46:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.