BrickPal: Augmented Reality-based Assembly Instructions for Brick Models
- URL: http://arxiv.org/abs/2307.03162v1
- Date: Thu, 6 Jul 2023 17:42:56 GMT
- Title: BrickPal: Augmented Reality-based Assembly Instructions for Brick Models
- Authors: Yao Shi, Xiaofeng Zhang, Ran zhang, Zhou Yang, Xiao Tang, Hongni Ye,
Yi Wu
- Abstract summary: BrickPal is an augmented reality-based system, which visualizes assembly instructions in an augmented reality head-mounted display.
It utilizes Natural Language Processing (NLP) techniques to generate plausible assembly sequences, and provide real-time guidance in the AR headset.
Our user study demonstrates BrickPal's effectiveness at assisting users in brick assembly compared to traditional assembly methods.
- Score: 20.585631669565295
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The assembly instruction is a mandatory component of Lego-like brick sets.The
conventional production of assembly instructions requires a considerable amount
of manual fine-tuning, which is intractable for casual users and customized
brick sets.Moreover, the traditional paper-based instructions lack
expressiveness and interactivity.To tackle the two problems above, we present
BrickPal, an augmented reality-based system, which visualizes assembly
instructions in an augmented reality head-mounted display. It utilizes Natural
Language Processing (NLP) techniques to generate plausible assembly sequences,
and provide real-time guidance in the AR headset.Our user study demonstrates
BrickPal's effectiveness at assisting users in brick assembly compared to
traditional assembly methods. Additionally, the NLP algorithm-generated
assembly sequences achieve the same usability with manually adapted sequences.
Related papers
- TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images.
A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice.
We mitigate this problem by leveraging synthetic-to-real transfer learning.
arXiv Detail & Related papers (2024-07-22T14:05:27Z) - Controllable Navigation Instruction Generation with Chain of Thought Prompting [74.34604350917273]
We propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation.
C-Instructor renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects.
arXiv Detail & Related papers (2024-07-10T07:37:20Z) - EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models [37.80143756214926]
EasyInstruct is an easy-to-use instruction processing framework for Large Language Models (LLMs)
EasyInstruct modularizes instruction generation, selection, and prompting, while also considering their combination and interaction.
arXiv Detail & Related papers (2024-02-05T14:33:56Z) - Multi-3D-Models Registration-Based Augmented Reality (AR) Instructions
for Assembly [7.716174636585781]
BRICKxAR (M3D) visualizes rendered 3D assembly parts at the assembly location of the physical assembly model.
BRICKxAR (M3D) utilizes deep learning-trained 3D model-based registration.
arXiv Detail & Related papers (2023-11-27T21:53:17Z) - From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning [63.63840740526497]
We investigate how instruction tuning adjusts pre-trained models with a focus on intrinsic changes.
The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models.
Our findings reveal three significant impacts of instruction tuning.
arXiv Detail & Related papers (2023-09-30T21:16:05Z) - Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural
Language Instructions [53.21504989297547]
We propose a new method that combines a language model and reinforcement learning for the task of building objects in a Minecraft-like environment.
Our method first generates a set of consistently achievable sub-goals from the instructions and then completes associated sub-tasks with a pre-trained RL policy.
arXiv Detail & Related papers (2022-11-01T18:30:42Z) - Sequential Brick Assembly with Efficient Constraint Satisfaction [39.869693447362145]
We address the problem of generating a sequence of LEGO brick assembly with high-fidelity structures.
Our method performs a brick structure assessment to predict the next brick position and its confidence by employing a U-shaped sparse 3D convolutional network.
Instead of using handcrafted brick assembly datasets, our model is trained with a large number of 3D objects that allow to create a new high-fidelity structure.
arXiv Detail & Related papers (2022-10-03T15:35:08Z) - Break and Make: Interactive Structural Understanding Using LEGO Bricks [61.01136603613139]
We build a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models.
We take a first step towards solving this problem using sequence-to-sequence models.
arXiv Detail & Related papers (2022-07-27T18:33:09Z) - Brick-by-Brick: Combinatorial Construction with Deep Reinforcement
Learning [52.85981207514049]
We introduce a novel formulation, complex construction, which requires a building agent to assemble unit primitives sequentially.
To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit information to the agent.
We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.
arXiv Detail & Related papers (2021-10-29T01:09:51Z) - Object Detection for Understanding Assembly Instruction Using
Context-aware Data Augmentation and Cascade Mask R-CNN [4.3310896118860445]
We developed a context-aware data augmentation scheme for speech bubble segmentation.
Also, we showed that deep learning can be useful to understand assembly instruction by detecting the essential objects in the instruction.
arXiv Detail & Related papers (2021-01-07T12:10:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.