IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos

• URL: http://arxiv.org/abs/2411.11409v1
• Date: Mon, 18 Nov 2024 09:30:05 GMT
• Title: IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos
• Authors: Yunong Liu, Cristobal Eyzaguirre, Manling Li, Shubh Khanna, Juan Carlos Niebles, Vineeth Ravi, Saumitra Mishra, Weiyu Liu, Jiajun Wu,
• Abstract summary: We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense-temporal alignments between these data modalities. We present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals.
• Score: 34.67148665646724
• License:
• Abstract: Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existing datasets have not yet tackled the 4D grounding of assembly instructions in videos, essential for a holistic understanding of assembly in 3D space over time. We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities. To demonstrate the utility of IKEA Video Manuals, we present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals. For each application, we provide evaluation metrics and baseline methods. Through experiments on our annotated data, we highlight many challenges in grounding assembly instructions in videos to improve shape assembly, including handling occlusions, varying viewpoints, and extended assembly sequences.

Related papers

• Multi-3D-Models Registration-Based Augmented Reality (AR) Instructions for Assembly [7.716174636585781]
  BRICKxAR (M3D) visualizes rendered 3D assembly parts at the assembly location of the physical assembly model. BRICKxAR (M3D) utilizes deep learning-trained 3D model-based registration.
  arXiv  Detail & Related papers  (2023-11-27T21:53:17Z)
• HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding [5.233797258148846]
  HA-ViD is the first human assembly video dataset that features representative industrial assembly scenarios. We provide 3222 multi-view, multi-modality videos (each video contains one assembly task), 1.5M frames, 96K temporal labels and 2M spatial labels. We benchmark four foundational video understanding tasks: action recognition, action segmentation, object detection and multi-object tracking.
  arXiv  Detail & Related papers  (2023-07-09T08:44:46Z)
• Aligning Step-by-Step Instructional Diagrams to Video Demonstrations [51.67930509196712]
  We consider a novel setting where alignment is between (i) instruction steps that are depicted as assembly diagrams and (ii) video segments from in-the-wild videos. We introduce a novel supervised contrastive learning method that learns to align videos with the subtle details in the assembly diagrams. Experiments on IAW for Ikea assembly in the wild demonstrate superior performances of our approach against alternatives.
  arXiv  Detail & Related papers  (2023-03-24T04:45:45Z)
• IKEA-Manual: Seeing Shape Assembly Step by Step [26.79113677450921]
  We present IKEA-Manual, a dataset consisting of 102 IKEA objects paired with assembly manuals. We provide fine-grained annotations on the IKEA objects and assembly manuals, including assembly parts, assembly plans, manual segmentation, and 2D-3D correspondence between 3D parts and visual manuals.
  arXiv  Detail & Related papers  (2023-02-03T17:32:22Z)
• Class-agnostic Reconstruction of Dynamic Objects from Videos [127.41336060616214]
  We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. We develop two novel modules. First, we introduce a canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop a 4D transformation module which captures object dynamics to support temporal propagation and aggregation.
  arXiv  Detail & Related papers  (2021-12-03T18:57:47Z)
• Reconstructing and grounding narrated instructional videos in 3D [99.22297066405741]
  We aim to reconstruct such objects and to localize associated narrations in 3D. We propose an approach for correspondence estimation combining learnt local features and dense flow. We demonstrate the effectiveness of our approach for the domain of car maintenance.
  arXiv  Detail & Related papers  (2021-09-09T16:49:10Z)
• SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data [124.2624568006391]
  We present SAIL-VOS 3D: a synthetic video dataset with frame-by-frame mesh annotations. We also develop first baselines for reconstruction of 3D meshes from video data via temporal models.
  arXiv  Detail & Related papers  (2021-05-18T15:42:37Z)
• The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose [108.21037046507483]
  IKEA ASM is a three million frame, multi-view, furniture assembly video dataset that includes depth, atomic actions, object segmentation, and human pose. We benchmark prominent methods for video action recognition, object segmentation and human pose estimation tasks on this challenging dataset. The dataset enables the development of holistic methods, which integrate multi-modal and multi-view data to better perform on these tasks.
  arXiv  Detail & Related papers  (2020-07-01T11:34:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.