Leveraging Procedural Knowledge and Task Hierarchies for Efficient Instructional Video Pre-training

• URL: http://arxiv.org/abs/2502.17352v1
• Date: Mon, 24 Feb 2025 17:29:10 GMT
• Title: Leveraging Procedural Knowledge and Task Hierarchies for Efficient Instructional Video Pre-training
• Authors: Karan Samel, Nitish Sontakke, Irfan Essa,
• Abstract summary: An instructional video model should be capable of inferring both the tasks and the steps that occur in an input video.<n>We explicitly mine task hierarchies and the procedural steps associated with instructional videos.<n>We test this pre-trained model on task recognition, step recognition, and step prediction tasks.
• Score: 16.84305914451672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Instructional videos provide a convenient modality to learn new tasks (ex. cooking a recipe, or assembling furniture). A viewer will want to find a corresponding video that reflects both the overall task they are interested in as well as contains the relevant steps they need to carry out the task. To perform this, an instructional video model should be capable of inferring both the tasks and the steps that occur in an input video. Doing this efficiently and in a generalizable fashion is key when compute or relevant video topics used to train this model are limited. To address these requirements we explicitly mine task hierarchies and the procedural steps associated with instructional videos. We use this prior knowledge to pre-train our model, $\texttt{Pivot}$, for step and task prediction. During pre-training, we also provide video augmentation and early stopping strategies to optimally identify which model to use for downstream tasks. We test this pre-trained model on task recognition, step recognition, and step prediction tasks on two downstream datasets. When pre-training data and compute are limited, we outperform previous baselines along these tasks. Therefore, leveraging prior task and step structures enables efficient training of $\texttt{Pivot}$ for instructional video recommendation.

Related papers

• Transfer Learning for Structured Pruning under Limited Task Data [15.946734013984184]
  We propose a framework which combines structured pruning with transfer learning to reduce the need for task-specific data. We demonstrate that our framework results in pruned models with improved generalization over strong baselines.
  arXiv  Detail & Related papers  (2023-11-10T20:23:35Z)
• Dense Video Object Captioning from Disjoint Supervision [77.47084982558101]
  We propose a new task and model for dense video object captioning. This task unifies spatial and temporal localization in video. We show how our model improves upon a number of strong baselines for this new task.
  arXiv  Detail & Related papers  (2023-06-20T17:57:23Z)
• StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos [47.03252542488226]
  We introduce StepFormer, a self-supervised model that discovers and localizes instruction steps in a video. We train our system on a large dataset of instructional videos, using their automatically-generated subtitles as the only source of supervision. Our model outperforms all previous unsupervised and weakly-supervised approaches on step detection and localization.
  arXiv  Detail & Related papers  (2023-04-26T03:37:28Z)
• Procedure-Aware Pretraining for Instructional Video Understanding [58.214549181779006]
  Key challenge in procedure understanding is to be able to extract from unlabeled videos the procedural knowledge. Our main insight is that instructional videos depict sequences of steps that repeat between instances of the same or different tasks. This graph can then be used to generate pseudo labels to train a video representation that encodes the procedural knowledge in a more accessible form.
  arXiv  Detail & Related papers  (2023-03-31T17:41:31Z)
• Learning and Verification of Task Structure in Instructional Videos [85.511888642497]
  We introduce a new pre-trained video model, VideoTaskformer, focused on representing the semantics and structure of instructional videos. Compared to prior work which learns step representations locally, our approach involves learning them globally. We introduce two new benchmarks for detecting mistakes in instructional videos, to verify if there is an anomalous step and if steps are executed in the right order.
  arXiv  Detail & Related papers  (2023-03-23T17:59:54Z)
• PIVOT: Prompting for Video Continual Learning [50.80141083993668]
  We introduce PIVOT, a novel method that leverages extensive knowledge in pre-trained models from the image domain. Our experiments show that PIVOT improves state-of-the-art methods by a significant 27% on the 20-task ActivityNet setup.
  arXiv  Detail & Related papers  (2022-12-09T13:22:27Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation [57.68890534164427]
  In this work, we ask if we may leverage semi-supervised learning in unlabeled video sequences and extra images to improve the performance on urban scene segmentation. We simply predict pseudo-labels for the unlabeled data and train subsequent models with both human-annotated and pseudo-labeled data. Our Naive-Student model, trained with such simple yet effective iterative semi-supervised learning, attains state-of-the-art results at all three Cityscapes benchmarks.
  arXiv  Detail & Related papers  (2020-05-20T18:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.