Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models

• URL: http://arxiv.org/abs/2604.04482v1
• Date: Mon, 06 Apr 2026 07:12:46 GMT
• Title: Scalable and Explainable Learner-Video Interaction Prediction using Multimodal Large Language Models
• Abstract summary: Video controls in educational videos provide implicit signals of cognitive processing and instructional design quality.<n>We propose a scalable, interpretable pipeline for predicting population-level watching, pausing, skipping, and rewinding behavior.
• Score: 7.74287993376099
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learners' use of video controls in educational videos provides implicit signals of cognitive processing and instructional design quality, yet the lack of scalable and explainable predictive models limits instructors' ability to anticipate such behavior before deployment. We propose a scalable, interpretable pipeline for predicting population-level watching, pausing, skipping, and rewinding behavior as proxies for cognitive load from video content alone. Our approach leverages multimodal large language models (MLLMs) to compute embeddings of short video segments and trains a neural classifier to identify temporally fine-grained interaction peaks. Drawing from multimedia learning theory on instructional design for optimal cognitive load, we code features of the video segments using GPT-5 and employ them as a basis for interpreting model predictions via concept activation vectors. We evaluate our pipeline on 77 million video control events from 66 online courses. Our findings demonstrate that classifiers based on MLLM embeddings reliably predict interaction peaks, generalize to unseen academic fields, and encode interpretable, theory-relevant instructional concepts. Overall, our results show the feasibility of cost-efficient, interpretable pre-screening of educational video design and open new opportunities to empirically examine multimedia learning theory at scale.

Related papers

• Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning [15.521458272682308]
  Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments.<n>We ask whether models can learn to think visually during training while reasoning directly at inference.<n>We introduce Internalized Visual Thinking (IVT), a framework that jointly optimize textual prediction and next-embedding prediction over unlabeled videos.
  arXiv  Detail & Related papers  (2026-08-16T17:35:38Z)
• DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning [28.673173377342593]
  General robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions.<n>DeVA is a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically guidance.<n> Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.
  arXiv  Detail & Related papers  (2026-07-27T08:37:35Z)
• LLEXICORP: End-user Explainability of Convolutional Neural Networks [4.417922173735815]
  Concept relevance propagation (CRP) methods can backtrack predictions to these channels and find images that most activate these channels.<n>CRP are largely manual: experts must inspect activation images to name the discovered concepts and must synthesize verbose explanations from relevance maps.<n>Our approach automatically assigns descriptive names to concept prototypes and generates natural-language explanations.<n>Our findings suggest that integrating concept-based attribution methods with large language models can significantly lower the barrier to interpreting deep neural networks.
  arXiv  Detail & Related papers  (2025-11-04T16:44:45Z)
• Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models [78.32948112203228]
  Video understanding represents the most challenging frontier in computer vision.<n>Recent emergence of Video-Large Multitemporal Models has demonstrated remarkable capabilities in video understanding tasks.<n>Survey aims to provide researchers and practitioners with a unified framework for advancing Video-LMM capabilities.
  arXiv  Detail & Related papers  (2025-10-06T17:10:44Z)
• Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark [27.487587901232057]
  We evaluate over 90 open-source and proprietary models, ranging from 0.5B to 40B parameters.<n>Our results highlight the limitations of current models in addressing the cognitive challenges presented by these lectures.
  arXiv  Detail & Related papers  (2025-04-20T17:58:46Z)
• Predicting Long-horizon Futures by Conditioning on Geometry and Time [49.86180975196375]
  We explore the task of generating future sensor observations conditioned on the past. We leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-04-17T16:56:31Z)
• LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos [15.127197238628396]
  LifelongMemory is a new framework for accessing long-form egocentric videographic memory through natural language question answering and retrieval. Our approach achieves state-of-the-art performance on the benchmark for question answering and is highly competitive on the natural language query (NLQ) challenge of Ego4D.
  arXiv  Detail & Related papers  (2023-12-07T19:19:25Z)
• Graph Neural Prompting with Large Language Models [32.97391910476073]
  Graph Neural Prompting (GNP) is a novel plug-and-play method to assist pre-trained language models in learning beneficial knowledge from knowledge graphs. Extensive experiments on multiple datasets demonstrate the superiority of GNP on both commonsense and biomedical reasoning tasks.
  arXiv  Detail & Related papers  (2023-09-27T06:33:29Z)
• Evaluating and Explaining Large Language Models for Code Using Syntactic Structures [74.93762031957883]
  This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes. We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
  arXiv  Detail & Related papers  (2023-08-07T18:50:57Z)
• Reinforcement Learning with Action-Free Pre-Training from Videos [95.25074614579646]
  We introduce a framework that learns representations useful for understanding the dynamics via generative pre-training on videos. Our framework significantly improves both final performances and sample-efficiency of vision-based reinforcement learning.
  arXiv  Detail & Related papers  (2022-03-25T19:44:09Z)
• Modular Action Concept Grounding in Semantic Video Prediction [28.917125574895422]
  We introduce the task of semantic action-conditional video prediction, which uses semantic action labels to describe interactions. Inspired by the idea of Mixture of Experts, we embody each abstract label by a structured combination of various visual concept learners. Our method is evaluated on two newly designed synthetic datasets and one real-world dataset.
  arXiv  Detail & Related papers  (2020-11-23T04:12:22Z)
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language [148.0843278195794]
  We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
  arXiv  Detail & Related papers  (2020-11-18T20:21:19Z)
• Video Understanding as Machine Translation [53.59298393079866]
  We tackle a wide variety of downstream video understanding tasks by means of a single unified framework. We report performance gains over the state-of-the-art on several downstream tasks including video classification (EPIC-Kitchens), question answering (TVQA), captioning (TVC, YouCook2, and MSR-VTT)
  arXiv  Detail & Related papers  (2020-06-12T14:07:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.