HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding

• URL: http://arxiv.org/abs/2605.31227v1
• Date: Fri, 29 May 2026 12:30:53 GMT
• Title: HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding
• Abstract summary: Procedural activities naturally decompose in a hierarchy of steps and sub-steps.<n>Traditional approaches for step grounding require extensive annotations and scale poorly.<n>We argue that such hierarchical structure can emerge naturally from uncurated videos of human activities.
• Score: 9.489870795072266
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Procedural activities follow well-defined structures: whether we consider a cooking recipe or a mechanic repairing a car, these activities naturally decompose in a hierarchy of steps and sub-steps. Traditional approaches for step grounding require extensive annotations and scale poorly. Instead, we argue that such hierarchical structure can emerge naturally from uncurated videos of human activities through recurring patterns of co-occurring actions and activities. Our approach builds on HiERO, a weakly-supervised representation learning approach that maps close in the feature space actions that are functionally related to each other, leveraging only fine-grained action-level narrations. In this feature space, procedure steps can be detected by a simple clustering, with no additional task-specific fine-tuning. For the Ego4D Step Grounding challenge, we augment this approach by ensuring fine and coarse level agreement in step assignments, enforcing strict temporal monotonicity of the grounded steps and post-processing the detected steps to reduce the impact of noisy predictions. We call this approach HiERO-StepG and it achieves 56.27 % on the R@1 (IoU = 0.3) metric on the global leaderboard at submission time, ranking second while being completely zero-shot and not requiring procedure-specific annotations. Project page: https://github.com/andreazenotto/HiERO-StepG.

Related papers

• SkiP: When to Skip and When to Refine for Efficient Robot Manipulation [50.03201984643502]
  textbfSkip Policy (SkiP) dynamically leaps over skip segments and intensively refines actions in key segments.<n>emphMotion Spectrum Keying (MSK) is a fast, task-agnostic procedure that detects local motion complexity from action signals.<n>Experiments show SkiP reduces executed steps by $15$--$40%$ while matching or improving success rates across various policy backbones.
  arXiv  Detail & Related papers  (2026-05-15T02:16:34Z)
• Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding [12.083424392460406]
  We introduce'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors procedures to what a model can actually see.<n>We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states.<n>Our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction.
  arXiv  Detail & Related papers  (2025-11-25T08:46:11Z)
• BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulation [48.08416841005715]
  We introduce a novel keypose-conditioned consistency policy tailored for bimanual manipulation. It is a hierarchical imitation learning framework that consists of a high-level keypose predictor and a low-level trajectory generator. Simulated and real-world experimental results demonstrate that the proposed approach surpasses baseline methods in terms of success rate and operational efficiency.
  arXiv  Detail & Related papers  (2024-06-14T14:49:12Z)
• Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learning [85.84504287685884]
  Skip-Plan is a condensed action space learning method for procedure planning in instructional videos. By skipping uncertain nodes and edges in action chains, we transfer long and complex sequence functions into short but reliable ones. Our model explores all sorts of reliable sub-relations within an action sequence in the condensed action space.
  arXiv  Detail & Related papers  (2023-10-01T08:02:33Z)
• Learning to Ground Instructional Articles in Videos through Narrations [50.3463147014498]
  We present an approach for localizing steps of procedural activities in narrated how-to videos. We source the step descriptions from a language knowledge base (wikiHow) containing instructional articles. Our model learns to temporally ground the steps of procedural articles in how-to videos by matching three modalities.
  arXiv  Detail & Related papers  (2023-06-06T15:45:53Z)
• Generating Intermediate Steps for NLI with Next-Step Supervision [15.425765421938447]
  We train a sequence-to-sequence model to generate only the next step given an NLI premise and hypothesis pair. We then enhance it with external knowledge and symbolic search to generate intermediate steps with only next-step supervision. We show the correctness of such generated steps through automated and human verification.
  arXiv  Detail & Related papers  (2022-08-31T05:25:33Z)
• Learning To Recognize Procedural Activities with Distant Supervision [96.58436002052466]
  We consider the problem of classifying fine-grained, multi-step activities from long videos spanning up to several minutes. Our method uses a language model to match noisy, automatically-transcribed speech from the video to step descriptions in the knowledge base.
  arXiv  Detail & Related papers  (2022-01-26T15:06:28Z)
• SVIP: Sequence VerIfication for Procedures in Videos [68.07865790764237]
  We propose a novel sequence verification task that aims to distinguish positive video pairs performing the same action sequence from negative ones with step-level transformations. Such a challenging task resides in an open-set setting without prior action detection or segmentation. We collect a scripted video dataset enumerating all kinds of step-level transformations in chemical experiments.
  arXiv  Detail & Related papers  (2021-12-13T07:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.