LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learning

• URL: http://arxiv.org/abs/2505.02078v1
• Date: Sun, 04 May 2025 12:06:47 GMT
• Title: LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learning
• Authors: Joy Lim Jia Yin, Daniel Zhang-Li, Jifan Yu, Haoxuan Li, Shangqing Tu, Yuanchun Wang, Zhiyuan Liu, Huiqin Liu, Lei Hou, Juanzi Li, Bin Xu,
• Abstract summary: We introduce LecEval, an automated metric grounded in Mayer's Cognitive Theory of Multimedia Learning.<n>LecEval assesses effectiveness using four rubrics: Content Relevance (CR), Expressive Clarity (EC), Logical Structure (LS) and Audience Engagement (AE)<n>We curate a large-scale dataset of over 2,000 slides from more than 50 online course videos, annotated with fine-grained human ratings.
• Score: 58.98865450345401
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating the quality of slide-based multimedia instruction is challenging. Existing methods like manual assessment, reference-based metrics, and large language model evaluators face limitations in scalability, context capture, or bias. In this paper, we introduce LecEval, an automated metric grounded in Mayer's Cognitive Theory of Multimedia Learning, to evaluate multimodal knowledge acquisition in slide-based learning. LecEval assesses effectiveness using four rubrics: Content Relevance (CR), Expressive Clarity (EC), Logical Structure (LS), and Audience Engagement (AE). We curate a large-scale dataset of over 2,000 slides from more than 50 online course videos, annotated with fine-grained human ratings across these rubrics. A model trained on this dataset demonstrates superior accuracy and adaptability compared to existing metrics, bridging the gap between automated and human assessments. We release our dataset and toolkits at https://github.com/JoylimJY/LecEval.

Related papers

• Teach2Eval: An Indirect Evaluation Method for LLM by Judging How It Teaches [46.0474342507327]
  We introduce Teach2Eval, an indirect evaluation framework inspired by the Feynman Technique.<n>Our method evaluates a model's multiple abilities to teach weaker student models to perform tasks effectively.
  arXiv  Detail & Related papers  (2025-05-18T06:51:10Z)
• SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning [78.44705665291741]
  We present a comprehensive evaluation of modern video self-supervised models.<n>We focus on generalization across four key downstream factors: domain shift, sample efficiency, action granularity, and task diversity.<n>Our analysis shows that, despite architectural advances, transformer-based models remain sensitive to downstream conditions.
  arXiv  Detail & Related papers  (2025-04-08T06:00:28Z)
• If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs [55.8331366739144]
  We introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in large language models (LLMs)<n>Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches.
  arXiv  Detail & Related papers  (2025-03-30T16:50:57Z)
• LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms [0.0]
  LLM-SEM (Language Model-Based Student Engagement Metric) is a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement.<n>We generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes.<n>Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels.
  arXiv  Detail & Related papers  (2024-12-18T12:01:53Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation [2.0411082897313984]
  This study introduces a novel methodology that integrates human annotators and Large Language Models. The proposed framework integrates human annotation with the output of LLMs, depending on the model uncertainty levels. The empirical results show a substantial decrease in the costs associated with data annotation while either maintaining or improving model accuracy.
  arXiv  Detail & Related papers  (2024-06-17T21:45:48Z)
• Polos: Multimodal Metric Learning from Human Feedback for Image Captioning [1.3654846342364308]
  Polos is a supervised automatic evaluation metric for image captioning models. We constructed the Polaris dataset, which comprises 131K human judgments from 550 evaluators. Our approach achieved state-of-the-art performance on Composite, Flickr8K-Expert, Flickr8K-CF, PASCAL-50S, FOIL, and the Polaris dataset.
  arXiv  Detail & Related papers  (2024-02-28T06:24:39Z)
• Self-Supervised Multimodal Learning: A Survey [23.526389924804207]
  Multimodal learning aims to understand and analyze information from multiple modalities. The heavy dependence on data paired with expensive human annotations impedes scaling up models. Given the availability of large-scale unannotated data in the wild, self-supervised learning has become an attractive strategy to alleviate the annotation bottleneck.
  arXiv  Detail & Related papers  (2023-03-31T16:11:56Z)
• Revisiting Classifier: Transferring Vision-Language Models for Video Recognition [102.93524173258487]
  Transferring knowledge from task-agnostic pre-trained deep models for downstream tasks is an important topic in computer vision research. In this study, we focus on transferring knowledge for video classification tasks. We utilize the well-pretrained language model to generate good semantic target for efficient transferring learning.
  arXiv  Detail & Related papers  (2022-07-04T10:00:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.