Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors

• URL: http://arxiv.org/abs/2604.03631v1
• Date: Sat, 04 Apr 2026 08:01:02 GMT
• Title: Single-agent vs. Multi-agents for Automated Video Analysis of On-Screen Collaborative Learning Behaviors
• Abstract summary: On-screen learning behavior provides valuable insights into how students seek, use, and create information during learning.<n>Recent development of Vision Language Models (VLMs) offers new opportunities to automate the labor-intensive manual coding.<n>This study demonstrates the effectiveness of VLM-based Multi-agent System for video analysis and contributes a scalable framework for multimodal data analytics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-screen learning behavior provides valuable insights into how students seek, use, and create information during learning. Analyzing on-screen behavioral engagement is essential for capturing students' cognitive and collaborative processes. The recent development of Vision Language Models (VLMs) offers new opportunities to automate the labor-intensive manual coding often required for multimodal video data analysis. In this study, we compared the performance of both leading closed-source VLMs (Claude-3.7-Sonnet, GPT-4.1) and open-source VLM (Qwen2.5-VL-72B) in single- and multi-agent settings for automated coding of screen recordings in collaborative learning contexts based on the ICAP framework. In particular, we proposed and compared two multi-agent frameworks: 1) a three-agent workflow multi-agent system (MAS) that segments screen videos by scene and detects on-screen behaviors using cursor-informed VLM prompting with evidence-based verification; 2) an autonomous-decision MAS inspired by ReAct that iteratively interleaves reasoning, tool-like operations (segmentation/ classification/ validation), and observation-driven self-correction to produce interpretable on-screen behavior labels. Experimental results demonstrated that the two proposed MAS frameworks achieved viable performance, outperforming the single VLMs in scene and action detection tasks. It is worth noting that the workflow-based agent achieved best on scene detection, and the autonomous-decision MAS achieved best on action detection. This study demonstrates the effectiveness of VLM-based Multi-agent System for video analysis and contributes a scalable framework for multimodal data analytics.

Related papers

• GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models [17.834853391238035]
  Teach VLM is a core model designed to translate mobile screen trajectories into step-wise operational knowledge.<n>Building upon Teach VLM, we propose the Teach-and-Repeat paradigm, where the generated operational knowledge serves as an interpretable procedural reference.
  arXiv  Detail & Related papers  (2026-06-11T02:24:39Z)
• InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning [61.87294123934038]
  We present InternVideo3, a framework enhancing multimodal understanding in video tasks.<n>MCR treats understanding as a closed-loop process over a shared, evolving context.<n>We show InternVideo3 achieves strong performance on benchmarks like Video-MME, MLVU, and Ego.
  arXiv  Detail & Related papers  (2026-06-10T15:17:08Z)
• MVR: Multi-view Video Reward Shaping for Reinforcement Learning [17.20077949643041]
  Multi-View Video Reward Shaping (MVR) is a framework that models the relevance of states regarding the target task using videos captured from multiple viewpoints.<n>MVR learns a state relevance function that mitigates the bias towards specific static poses inherent in image-based methods.<n>We introduce a state-dependent reward shaping formulation that integrates task-specific rewards and VLM-based guidance.
  arXiv  Detail & Related papers  (2026-03-02T10:24:04Z)
• Training Multi-Image Vision Agents via End2End Reinforcement Learning [51.81337984526068]
  We propose IMAgent, an open-source vision agent trained via end-to-end reinforcement learning.<n>By leveraging a multi-agent system, we generate challenging and visually-rich multi-image QA pairs.<n>We develop two specialized tools for visual reflection and confirmation, allowing the model to proactively reallocate its attention to image content.
  arXiv  Detail & Related papers  (2025-12-05T10:02:38Z)
• Learning from Online Videos at Inference Time for Computer-Use Agents [41.90425060535666]
  We study how to enable computer-use agents to learn from online videos at inference time effectively.<n>We propose a framework that retrieves and filters tutorial videos, converts them into structured demonstration trajectories, and dynamically selects trajectories as in-context guidance.<n>Our framework consistently outperforms strong base agents and variants that use only textual tutorials or transcripts.
  arXiv  Detail & Related papers  (2025-11-06T07:29:02Z)
• Dual Learning with Dynamic Knowledge Distillation and Soft Alignment for Partially Relevant Video Retrieval [53.54695034420311]
  In practice, videos are typically untrimmed in long durations with much more complicated background content.<n>We propose a novel framework that distills generalization knowledge from a powerful large-scale vision-language pre-trained model.<n>Experiment results demonstrate that our proposed model achieves state-of-the-art performance on TVR, ActivityNet, and Charades-STA datasets.
  arXiv  Detail & Related papers  (2025-10-14T08:38:20Z)
• VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents [105.43882565434444]
  We propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms.<n>First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types.<n>Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs.
  arXiv  Detail & Related papers  (2025-07-07T00:51:57Z)
• CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation [31.167405688707575]
  We propose a framework for instance-level visual analysis on video frames. It can simultaneously conduct object detection, instance segmentation, and multi-object tracking. We evaluate the proposed method extensively on KITTI MOTS and MOTS Challenge datasets.
  arXiv  Detail & Related papers  (2023-11-02T04:32:24Z)
• Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting [83.21164539349273]
  Pre-trained language models (PLMs) have played an increasing role in multimedia research. In this paper, we focus on exploring PLMs as a stand-alone model for vision-language reasoning tasks. We propose a novel transfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP)
  arXiv  Detail & Related papers  (2023-06-01T07:19:28Z)
• MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection [76.80153360498797]
  We develop a multiple instance self-training framework (MIST) to efficiently refine task-specific discriminative representations. MIST is composed of 1) a multiple instance pseudo label generator, which adapts a sparse continuous sampling strategy to produce more reliable clip-level pseudo labels, and 2) a self-guided attention boosted feature encoder. Our method performs comparably to or even better than existing supervised and weakly supervised methods, specifically obtaining a frame-level AUC 94.83% on ShanghaiTech.
  arXiv  Detail & Related papers  (2021-04-04T15:47:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.