BrickPal: Augmented Reality-based Assembly Instructions for Brick Models
- URL: http://arxiv.org/abs/2307.03162v1
- Date: Thu, 6 Jul 2023 17:42:56 GMT
- Title: BrickPal: Augmented Reality-based Assembly Instructions for Brick Models
- Authors: Yao Shi, Xiaofeng Zhang, Ran zhang, Zhou Yang, Xiao Tang, Hongni Ye,
Yi Wu
- Abstract summary: BrickPal is an augmented reality-based system, which visualizes assembly instructions in an augmented reality head-mounted display.
It utilizes Natural Language Processing (NLP) techniques to generate plausible assembly sequences, and provide real-time guidance in the AR headset.
Our user study demonstrates BrickPal's effectiveness at assisting users in brick assembly compared to traditional assembly methods.
- Score: 20.585631669565295
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The assembly instruction is a mandatory component of Lego-like brick sets.The
conventional production of assembly instructions requires a considerable amount
of manual fine-tuning, which is intractable for casual users and customized
brick sets.Moreover, the traditional paper-based instructions lack
expressiveness and interactivity.To tackle the two problems above, we present
BrickPal, an augmented reality-based system, which visualizes assembly
instructions in an augmented reality head-mounted display. It utilizes Natural
Language Processing (NLP) techniques to generate plausible assembly sequences,
and provide real-time guidance in the AR headset.Our user study demonstrates
BrickPal's effectiveness at assisting users in brick assembly compared to
traditional assembly methods. Additionally, the NLP algorithm-generated
assembly sequences achieve the same usability with manually adapted sequences.
Related papers
- Learning to Build by Building Your Own Instructions [56.734927320020496]
We develop a new technique for the recently proposed Break-and-Make problem in LTRON.
An agent must learn to build a previously unseen LEGO assembly using a single interactive session.
We train these models using online imitation learning which allows the model to learn from its own mistakes.
arXiv Detail & Related papers (2024-10-01T22:39:58Z) - Physics-Aware Combinatorial Assembly Planning using Deep Reinforcement Learning [6.919208054874144]
Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications.
This paper studies sequence planning for physical assembly using Lego.
In particular, we aim to ensure the planned assembly sequence is physically executable.
arXiv Detail & Related papers (2024-08-19T17:16:35Z) - TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images.
A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice.
We mitigate this problem by leveraging synthetic-to-real transfer learning.
arXiv Detail & Related papers (2024-07-22T14:05:27Z) - Controllable Navigation Instruction Generation with Chain of Thought Prompting [74.34604350917273]
We propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation.
C-Instructor renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects.
arXiv Detail & Related papers (2024-07-10T07:37:20Z) - EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models [37.80143756214926]
EasyInstruct is an easy-to-use instruction processing framework for Large Language Models (LLMs)
EasyInstruct modularizes instruction generation, selection, and prompting, while also considering their combination and interaction.
arXiv Detail & Related papers (2024-02-05T14:33:56Z) - Multi-3D-Models Registration-Based Augmented Reality (AR) Instructions
for Assembly [7.716174636585781]
BRICKxAR (M3D) visualizes rendered 3D assembly parts at the assembly location of the physical assembly model.
BRICKxAR (M3D) utilizes deep learning-trained 3D model-based registration.
arXiv Detail & Related papers (2023-11-27T21:53:17Z) - From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning [63.63840740526497]
We investigate how instruction tuning adjusts pre-trained models with a focus on intrinsic changes.
The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models.
Our findings reveal three significant impacts of instruction tuning.
arXiv Detail & Related papers (2023-09-30T21:16:05Z) - Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction [63.672314717599285]
We tackle the problem of sequential brick assembly with LEGO bricks to create 3D structures.
In particular, the number of assemblable structures increases exponentially as the number of bricks used increases.
We propose a new method to predict the scores of the next brick position by employing a U-shaped sparse 3D convolutional neural network.
arXiv Detail & Related papers (2022-10-03T15:35:08Z) - Break and Make: Interactive Structural Understanding Using LEGO Bricks [61.01136603613139]
We build a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models.
We take a first step towards solving this problem using sequence-to-sequence models.
arXiv Detail & Related papers (2022-07-27T18:33:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.