ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions

• URL: http://arxiv.org/abs/2412.01987v1
• Date: Mon, 02 Dec 2024 21:40:17 GMT
• Title: ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions
• Authors: Tomáš Souček, Prajwal Gatti, Michael Wray, Ivan Laptev, Dima Damen, Josef Sivic,
• Abstract summary: The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image. Part of the challenge stems from the lack of large-scale training data for this problem. We introduce an automatic approach for collecting large step-by-step visual instruction training data from instructional videos. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image.
• Score: 57.304601070962086
• License:
• Abstract: The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image that provides the scene context and the sequence of textual instructions. This is a challenging problem as it requires generating multi-step image sequences to achieve a complex goal while being grounded in a specific environment. Part of the challenge stems from the lack of large-scale training data for this problem. The contribution of this work is thus three-fold. First, we introduce an automatic approach for collecting large step-by-step visual instruction training data from instructional videos. We apply this approach to one million videos and create a large-scale, high-quality dataset of 0.6M sequences of image-text pairs. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image. Third, we evaluate the generated image sequences across three dimensions of accuracy (step, scene, and task) and show our model achieves state-of-the-art results on all of them. Our code, dataset, and trained models are publicly available.

Related papers

• T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation [56.054622766743414]
  We propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation. It enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models.
  arXiv  Detail & Related papers  (2024-12-18T04:01:32Z)
• VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model [34.35449902855767]
  Two fundamental questions are what data we use for training and how to ensure multi-view consistency. We propose a dense consistent multi-view generation model that is fine-tuned from off-the-shelf video generative models. Our approach can generate 24 dense views and converges much faster in training than state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-18T17:48:15Z)
• Non-Sequential Graph Script Induction via Multimedia Grounding [129.83134296316493]
  We train a script knowledge model capable of both generating explicit graph scripts for learnt tasks and predicting future steps given a partial step sequence. Human evaluation shows our model outperforming the WikiHow linear baseline by 48.76% absolute gains in capturing sequential and non-sequential step relationships.
  arXiv  Detail & Related papers  (2023-05-27T18:13:17Z)
• End-to-end Dense Video Captioning as Sequence Generation [83.90502354328679]
  We show how to model the two subtasks of dense video captioning jointly as one sequence generation task. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks integrated into large-scale pre-trained models.
  arXiv  Detail & Related papers  (2022-04-18T01:30:54Z)
• Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval [80.7397409377659]
  We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. We show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks.
  arXiv  Detail & Related papers  (2021-04-01T17:48:27Z)
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation [57.68890534164427]
  In this work, we ask if we may leverage semi-supervised learning in unlabeled video sequences and extra images to improve the performance on urban scene segmentation. We simply predict pseudo-labels for the unlabeled data and train subsequent models with both human-annotated and pseudo-labeled data. Our Naive-Student model, trained with such simple yet effective iterative semi-supervised learning, attains state-of-the-art results at all three Cityscapes benchmarks.
  arXiv  Detail & Related papers  (2020-05-20T18:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.