BrickPal: Augmented Reality-based Assembly Instructions for Brick Models

• URL: http://arxiv.org/abs/2307.03162v1
• Date: Thu, 6 Jul 2023 17:42:56 GMT
• Title: BrickPal: Augmented Reality-based Assembly Instructions for Brick Models
• Authors: Yao Shi, Xiaofeng Zhang, Ran zhang, Zhou Yang, Xiao Tang, Hongni Ye, Yi Wu
• Abstract summary: BrickPal is an augmented reality-based system, which visualizes assembly instructions in an augmented reality head-mounted display. It utilizes Natural Language Processing (NLP) techniques to generate plausible assembly sequences, and provide real-time guidance in the AR headset. Our user study demonstrates BrickPal's effectiveness at assisting users in brick assembly compared to traditional assembly methods.
• Score: 20.585631669565295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The assembly instruction is a mandatory component of Lego-like brick sets.The conventional production of assembly instructions requires a considerable amount of manual fine-tuning, which is intractable for casual users and customized brick sets.Moreover, the traditional paper-based instructions lack expressiveness and interactivity.To tackle the two problems above, we present BrickPal, an augmented reality-based system, which visualizes assembly instructions in an augmented reality head-mounted display. It utilizes Natural Language Processing (NLP) techniques to generate plausible assembly sequences, and provide real-time guidance in the AR headset.Our user study demonstrates BrickPal's effectiveness at assisting users in brick assembly compared to traditional assembly methods. Additionally, the NLP algorithm-generated assembly sequences achieve the same usability with manually adapted sequences.

Related papers

• Manual-PA: Learning 3D Part Assembly from Instruction Diagrams [54.555154845137906]
  We present Manual-PA, a transformer-based instruction Manual-guided 3D Part Assembly framework. Our results show that using the diagrams and the order of the parts lead to significant improvements in assembly performance against the state of the art.
  arXiv  Detail & Related papers  (2024-11-27T03:10:29Z)
• Learning to Build by Building Your Own Instructions [56.734927320020496]
  We develop a new technique for the recently proposed Break-and-Make problem in LTRON. An agent must learn to build a previously unseen LEGO assembly using a single interactive session. We train these models using online imitation learning which allows the model to learn from its own mistakes.
  arXiv  Detail & Related papers  (2024-10-01T22:39:58Z)
• Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking [6.919208054874144]
  Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications. This paper studies assembly sequence planning (ASP) for physical assembly. We employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object.
  arXiv  Detail & Related papers  (2024-08-19T17:16:35Z)
• TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
  We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning.
  arXiv  Detail & Related papers  (2024-07-22T14:05:27Z)
• Controllable Navigation Instruction Generation with Chain of Thought Prompting [74.34604350917273]
  We propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation. C-Instructor renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects.
  arXiv  Detail & Related papers  (2024-07-10T07:37:20Z)
• Multi-3D-Models Registration-Based Augmented Reality (AR) Instructions for Assembly [7.716174636585781]
  BRICKxAR (M3D) visualizes rendered 3D assembly parts at the assembly location of the physical assembly model. BRICKxAR (M3D) utilizes deep learning-trained 3D model-based registration.
  arXiv  Detail & Related papers  (2023-11-27T21:53:17Z)
• Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction [63.672314717599285]
  We tackle the problem of sequential brick assembly with LEGO bricks to create 3D structures. In particular, the number of assemblable structures increases exponentially as the number of bricks used increases. We propose a new method to predict the scores of the next brick position by employing a U-shaped sparse 3D convolutional neural network.
  arXiv  Detail & Related papers  (2022-10-03T15:35:08Z)
• Break and Make: Interactive Structural Understanding Using LEGO Bricks [61.01136603613139]
  We build a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models. We take a first step towards solving this problem using sequence-to-sequence models.
  arXiv  Detail & Related papers  (2022-07-27T18:33:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.