BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly

• URL: http://arxiv.org/abs/2605.07605v1
• Date: Fri, 08 May 2026 11:30:25 GMT
• Title: BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly
• Abstract summary: BrickCraft is a compositional framework designed for long-horizon and generalizable interlocking brick assembly.<n>It bridges the gap between high-level assembly plans and physical execution through situated manuals.<n>BrickCraft exhibits strong compositional generalization to unseen structures.
• Score: 13.976436316430911
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous robotic assembly of interlocking bricks demands seamless integration of long-horizon task reasoning, spatial grounding, and fine-grained manipulation. This paper presents BrickCraft, a compositional framework designed for long-horizon and generalizable interlocking brick assembly. BrickCraft models the assembly process using a relative formulation, where each step is anchored to a reference brick within the partial structure, thereby decomposing complex tasks into a finite set of reusable primitive skills. BrickCraft bridges the gap between high-level assembly plans and physical execution through situated manuals, which provide explicit spatial guidance for learned visuomotor skills by projecting the assembly intent onto real-time robot observations. Finally, BrickCraft employs a compositional execution pipeline that chains these spatially grounded skills to accomplish long-horizon assembly tasks. Extensive experimental validations demonstrate that BrickCraft acquires proficient assembly skills from a limited set of demonstrations and exhibits strong compositional generalization to unseen structures. The project website is available at https://intelligent-control-lab.github.io/BrickCraft.

Related papers

• Brick-Composer: Using MLLMs for Assembly with Diverse Bricks [64.5380622477211]
  We study whether multimodal large language models (MLLMs) possess the visual grounding and spatial reasoning capabilities required for brick assembly.<n>We introduce BC-Bench, the first benchmark for evaluating MLLMs on assembly with diverse bricks.<n>We propose Brick-Composer, a learning framework that equips MLLMs with assembly skills through three complementary signals.
  arXiv  Detail & Related papers  (2026-06-03T21:08:06Z)
• Prompt-to-Product: Generative Assembly via Bimanual Manipulation [67.08306531337634]
  Prompt-to-Product is an automated pipeline that generates real-world assembly products from natural language prompts.<n>We leverage LEGO bricks as the assembly platform and automate the process of creating brick assembly structures.
  arXiv  Detail & Related papers  (2025-08-28T17:59:05Z)
• Generating Physically Stable and Buildable Brick Structures from Text [63.75381708299733]
  BrickGPT is the first approach for generating physically stable assembly models from text prompts.<n>We release our dataset, StableText2Brick, containing over7,000 3D textured brick structures.
  arXiv  Detail & Related papers  (2025-05-08T17:58:18Z)
• AssemblyComplete: 3D Combinatorial Construction with Deep Reinforcement Learning [4.3507834596906125]
  A critical goal in robotics is to teach robots to adapt to real-world collaborative tasks, particularly in automatic assembly. This paper introduces 3D assembly completion, which is demonstrated using unit primitives (i.e., Lego bricks) We propose a two-part deep reinforcement learning (DRL) framework that tackles teaching the robot to understand the objective of an incomplete assembly and learning a construction policy to complete the assembly.
  arXiv  Detail & Related papers  (2024-10-20T18:51:17Z)
• Learning to Build by Building Your Own Instructions [56.734927320020496]
  We develop a new technique for the recently proposed Break-and-Make problem in LTRON. An agent must learn to build a previously unseen LEGO assembly using a single interactive session. We train these models using online imitation learning which allows the model to learn from its own mistakes.
  arXiv  Detail & Related papers  (2024-10-01T22:39:58Z)
• Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking [6.919208054874144]
  Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications.<n>This paper studies assembly sequence planning (ASP) for physical assembly.<n>We employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object.
  arXiv  Detail & Related papers  (2024-08-19T17:16:35Z)
• BrickPal: Augmented Reality-based Assembly Instructions for Brick Models [20.585631669565295]
  BrickPal is an augmented reality-based system, which visualizes assembly instructions in an augmented reality head-mounted display. It utilizes Natural Language Processing (NLP) techniques to generate plausible assembly sequences, and provide real-time guidance in the AR headset. Our user study demonstrates BrickPal's effectiveness at assisting users in brick assembly compared to traditional assembly methods.
  arXiv  Detail & Related papers  (2023-07-06T17:42:56Z)
• Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction [63.672314717599285]
  We tackle the problem of sequential brick assembly with LEGO bricks to create 3D structures. In particular, the number of assemblable structures increases exponentially as the number of bricks used increases. We propose a new method to predict the scores of the next brick position by employing a U-shaped sparse 3D convolutional neural network.
  arXiv  Detail & Related papers  (2022-10-03T15:35:08Z)
• Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning [52.85981207514049]
  We introduce a novel formulation, complex construction, which requires a building agent to assemble unit primitives sequentially. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit information to the agent. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.
  arXiv  Detail & Related papers  (2021-10-29T01:09:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.