DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation

• URL: http://arxiv.org/abs/2604.26565v1
• Date: Wed, 29 Apr 2026 11:51:35 GMT
• Title: DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation
• Abstract summary: We introduce an automated, training-free pipeline to extract high-quality procedural annotations from in-the-wild instructional videos.<n>Our approach segments videos into coherent shots, filters poorly aligned content, and generates temporally grounded procedural steps.<n>This pipeline yields DenseStep2M, a large-scale dataset comprising approximately 100K videos and 2M detailed instructional steps.
• Score: 48.62186630490615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-term video understanding requires interpreting complex temporal events and reasoning over procedural activities. While instructional video corpora, like HowTo100M, offer rich resources for model training, they present significant challenges, including noisy ASR transcripts and inconsistent temporal alignments between narration and visual content. In this work, we introduce an automated, training-free pipeline to extract high-quality procedural annotations from in-the-wild instructional videos. Our approach segments videos into coherent shots, filters poorly aligned content, and leverages state-of-the-art multimodal and large language models (Qwen2.5-VL and DeepSeek-R1) to generate structured, temporally grounded procedural steps. This pipeline yields DenseStep2M, a large-scale dataset comprising approximately 100K videos and 2M detailed instructional steps, designed to support comprehensive long-form video understanding. To rigorously evaluate our pipeline, we curate DenseCaption100, a benchmark of high-quality, human-written captions. Evaluations demonstrate strong alignment between our auto-generated steps and human annotations. Furthermore, we validate the utility of DenseStep2M across three core downstream tasks: dense video captioning, procedural step grounding, and cross-modal retrieval. Models fine-tuned on DenseStep2M achieve substantial gains in captioning quality and temporal localization, while exhibiting robust zero-shot generalization across egocentric, exocentric, and mixed-perspective domains. These results underscore the effectiveness of DenseStep2M in facilitating advanced multimodal alignment and long-term activity reasoning. Our dataset is available at https://huggingface.co/datasets/mingjige/DenseStep2M.

Related papers

• ARC-Chapter: Structuring Hour-Long Videos into Navigable Chapters and Hierarchical Summaries [77.41072125938636]
  ARC-Chapter is the first large-scale video chaptering model trained on over million-level long video chapters.<n>It unifies ASR transcripts, scene texts, visual captions into multi-level annotations, from short title to long summaries.<n>It establishes a new state-of-the-art by a significant margin, outperforming the previous best by 14.0% in F1 score and 11.3% in SODA score.
  arXiv  Detail & Related papers  (2025-11-18T10:53:14Z)
• Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment [53.12952107996463]
  This work proposes a novel training framework for learning to localize temporal boundaries of procedure steps in training videos. Motivated by the strong capabilities of Large Language Models (LLMs) in procedure understanding and text summarization, we first apply an LLM to filter out task-irrelevant information and summarize task-related procedure steps from narrations. To further generate reliable pseudo-matching between the LLM-steps and the video for training, we propose the Multi-Pathway Text-Video Alignment (MPTVA) strategy.
  arXiv  Detail & Related papers  (2024-09-22T18:40:55Z)
• Multi-Sentence Grounding for Long-term Instructional Video [63.27905419718045]
  We aim to establish an automatic, scalable pipeline for denoising a large-scale instructional dataset. We construct a high-quality video-text dataset with multiple descriptive steps supervision, named HowToStep.
  arXiv  Detail & Related papers  (2023-12-21T17:28:09Z)
• Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains [4.9347081318119015]
  We introduce Spacewalk-18, a benchmark containing two tasks: (1) step recognition and (2) video question answering.<n>In tandem, the two tasks quantify a model's ability to: (1) generalize to novel domains; (2) utilize long temporal context and multimodal (e.g. visual and speech) information.<n>We discover a promising adaptation via summarization technique that leads to significant performance improvement without model fine-tuning.
  arXiv  Detail & Related papers  (2023-11-30T18:19:23Z)
• Learning to Ground Instructional Articles in Videos through Narrations [50.3463147014498]
  We present an approach for localizing steps of procedural activities in narrated how-to videos. We source the step descriptions from a language knowledge base (wikiHow) containing instructional articles. Our model learns to temporally ground the steps of procedural articles in how-to videos by matching three modalities.
  arXiv  Detail & Related papers  (2023-06-06T15:45:53Z)
• Temporal Alignment Networks for Long-term Video [103.69904379356413]
  We propose a temporal alignment network that ingests long term video sequences, and associated text sentences. We train such networks from large-scale datasets, such as HowTo100M, where the associated text sentences have significant noise. Our proposed model, trained on HowTo100M, outperforms strong baselines (CLIP, MIL-NCE) on this alignment dataset.
  arXiv  Detail & Related papers  (2022-04-06T17:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.