Post-Training VLMs for Video Mistake Detection

• URL: http://arxiv.org/abs/2608.28406v1
• Date: Fri, 28 Aug 2026 14:56:58 GMT
• Title: Post-Training VLMs for Video Mistake Detection
• Abstract summary: We argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details.<n>We introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark.<n>Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video.
• Score: 30.215996979058236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human mistakes are inevitable when following instructions, yet they can lead to severe consequences. As such, there has been an increased interest in developing methods for detecting mistakes in videos, with current methods mostly focusing on closed-set protocols. While successful in controlled settings, the closed-set assumption limits their wider applicability, as any changes to the task require collecting new data and re-training models. Instead, we argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details. To reflect this, we introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark. MD-VQA tests whether methods can discern if a step was executed correctly with respect to its description, for both seen and unseen actions. To address this important challenge, we propose the first video-language-model post-training technique for mistake detection. Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video. Extensive evaluations demonstrate that this approach outperforms zero-shot, supervised fine-tuning, and post-training baselines. Notably, our method generalizes especially well to unseen procedures, for instance, with an improvement of up to 11.6% over the best-performing baseline on EP-VQA, paving the way toward general mistake detection. We release our code and benchmark at https://github.com/FedeSpu/mstk.

Related papers

• The Unreasonable Effectiveness of VLMs for Zero-shot Procedural Mistake Detection [27.97762014835384]
  Procedural mistake detection is important for quality control and user assistance across many disciplines.<n>Recent work in this field has achieved significant gains by using the reasoning capabilities of Video-Language Models (VLMs) as components within multi-stage pipelines.<n>We propose a unified Zero-shot Procedural Mistake detection framework that jointly solves procedural mistake detection and temporal action segmentation with a single pre-trained VLM.
  arXiv  Detail & Related papers  (2026-06-19T16:31:44Z)
• MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos [53.5354731425141]
  We introduce the task of early mistake detection in video.<n>The goal is to determine whether a keystep in a procedural activity is performed correctly while observing as little of the streaming video as possible.<n>We propose a method comprising a mistake detector and a reinforcement learning policy.
  arXiv  Detail & Related papers  (2026-03-15T07:14:49Z)
• Project Aletheia: Verifier-Guided Distillation of Backtracking for Small Language Models [0.0]
  Small Language Models (SLMs, under 10B parameters) are attractive for private, on-device deployment.<n>We introduce Verifier-Guided Distillation, a training protocol that transfers the process of error repair.<n>We show that latent verification behavior can emerge in small models, enabling them to occasionally stop, detect contradictions, and revise earlier assumptions.
  arXiv  Detail & Related papers  (2026-01-14T14:39:08Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge [5.257305312436567]
  We introduce an online mistake detection framework that handles both procedural and execution errors.<n>Upon detecting an error, we use a large language model (LLM) to generate explanatory feedback.<n>Experiments on the HoloAssist benchmark confirm the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-06-06T15:39:09Z)
• Rethinking Metrics and Benchmarks of Video Anomaly Detection [58.37571339811799]
  Video Anomaly Detection (VAD) aims to detect anomalies that deviate from expectation.<n>Existing VAD metrics are influenced by single annotation bias.<n>Existing benchmarks lack the capability to evaluate scene overfitting of fully/weakly-supervised algorithms.
  arXiv  Detail & Related papers  (2025-05-25T08:09:42Z)
• PREGO: online mistake detection in PRocedural EGOcentric videos [49.72812518471056]
  We propose PREGO, the first online one-class classification model for mistake detection in egocentric videos. PREGO is based on an online action recognition component to model the current action, and a symbolic reasoning module to predict the next actions. We evaluate PREGO on two procedural egocentric video datasets, Assembly101 and Epic-tent, which we adapt for online benchmarking of procedural mistake detection.
  arXiv  Detail & Related papers  (2024-04-02T13:27:28Z)
• Frustratingly Simple Few-Shot Object Detection [98.42824677627581]
  We find that fine-tuning only the last layer of existing detectors on rare classes is crucial to the few-shot object detection task. Such a simple approach outperforms the meta-learning methods by roughly 220 points on current benchmarks.
  arXiv  Detail & Related papers  (2020-03-16T00:29:14Z)
• Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detection [114.9714355807607]
  We show that applying self-trained deep ordinal regression to video anomaly detection overcomes two key limitations of existing methods. We devise an end-to-end trainable video anomaly detection approach that enables joint representation learning and anomaly scoring without manually labeled normal/abnormal data.
  arXiv  Detail & Related papers  (2020-03-15T08:44:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.