Training-free Online Video Step Grounding

• URL: http://arxiv.org/abs/2510.16989v1
• Date: Sun, 19 Oct 2025 20:11:52 GMT
• Title: Training-free Online Video Step Grounding
• Authors: Luca Zanella, Massimiliano Mancini, Yiming Wang, Alessio Tonioni, Elisa Ricci,
• Abstract summary: Video Step Grounding (VSG) aims to detect which steps are performed in a video.<n>We exploit the zero-shot capabilities of recent Large Multimodal Models (LMMs) to perform VSG online and without training.<n>We show that this online strategy without task-specific tuning outperforms offline and training-based models.
• Score: 42.545599316279954
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Given a task and a set of steps composing it, Video Step Grounding (VSG) aims to detect which steps are performed in a video. Standard approaches for this task require a labeled training set (e.g., with step-level annotations or narrations), which may be costly to collect. Moreover, they process the full video offline, limiting their applications for scenarios requiring online decisions. Thus, in this work, we explore how to perform VSG online and without training. We achieve this by exploiting the zero-shot capabilities of recent Large Multimodal Models (LMMs). In particular, we use LMMs to predict the step associated with a restricted set of frames, without access to the whole video. We show that this online strategy without task-specific tuning outperforms offline and training-based models. Motivated by this finding, we develop Bayesian Grounding with Large Multimodal Models (BaGLM), further injecting knowledge of past frames into the LMM-based predictions. BaGLM exploits Bayesian filtering principles, modeling step transitions via (i) a dependency matrix extracted through large language models and (ii) an estimation of step progress. Experiments on three datasets show superior performance of BaGLM over state-of-the-art training-based offline methods.

Related papers

• Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models [78.40747899831793]
  Video understanding represents the most challenging frontier in computer vision.<n>Recent emergence of Video-Large Multitemporal Models has demonstrated remarkable capabilities in video understanding tasks.<n>Survey aims to provide researchers and practitioners with a unified framework for advancing Video-LMM capabilities.
  arXiv  Detail & Related papers  (2025-10-06T17:10:44Z)
• Reinforced Language Models for Sequential Decision Making [6.971286730860635]
  Large Language Models (LLMs) show potential as sequential decision-making agents.<n>Existing post-training methods are designed for single-turn interactions and cannot handle credit assignment in multi-step agentic tasks.<n>This work demonstrates that targeted post-training is a practical and efficient alternative to relying on model scale for creating sequential decision-making agents.
  arXiv  Detail & Related papers  (2025-08-14T17:05:44Z)
• ProgRM: Build Better GUI Agents with Progress Rewards [18.654776061354895]
  We propose Progress Reward Model (ProgRM) to provide dense informative intermediate rewards by predicting a task completion progress for each step in online training.<n>ProgRM is evaluated with extensive experiments and analyses.
  arXiv  Detail & Related papers  (2025-05-23T17:23:11Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability.<n>To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT.<n>This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems [59.40480894948944]
  Large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. We prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning.
  arXiv  Detail & Related papers  (2024-05-30T09:42:54Z)
• Harnessing Large Language Models for Training-free Video Anomaly Detection [34.76811491190446]
  Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Training-based methods are prone to be domain-specific, thus being costly for practical deployment. We propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm.
  arXiv  Detail & Related papers  (2024-04-01T09:34:55Z)
• Learning and Verification of Task Structure in Instructional Videos [85.511888642497]
  We introduce a new pre-trained video model, VideoTaskformer, focused on representing the semantics and structure of instructional videos. Compared to prior work which learns step representations locally, our approach involves learning them globally. We introduce two new benchmarks for detecting mistakes in instructional videos, to verify if there is an anomalous step and if steps are executed in the right order.
  arXiv  Detail & Related papers  (2023-03-23T17:59:54Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.