Break and Make: Interactive Structural Understanding Using LEGO Bricks

• URL: http://arxiv.org/abs/2207.13738v1
• Date: Wed, 27 Jul 2022 18:33:09 GMT
• Title: Break and Make: Interactive Structural Understanding Using LEGO Bricks
• Authors: Aaron Walsman, Muru Zhang, Klemen Kotar, Karthik Desingh, Ali Farhadi, Dieter Fox
• Abstract summary: We build a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models. We take a first step towards solving this problem using sequence-to-sequence models.
• Score: 61.01136603613139
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual understanding of geometric structures with complex spatial relationships is a fundamental component of human intelligence. As children, we learn how to reason about structure not only from observation, but also by interacting with the world around us -- by taking things apart and putting them back together again. The ability to reason about structure and compositionality allows us to not only build things, but also understand and reverse-engineer complex systems. In order to advance research in interactive reasoning for part-based geometric understanding, we propose a challenging new assembly problem using LEGO bricks that we call Break and Make. In this problem an agent is given a LEGO model and attempts to understand its structure by interactively inspecting and disassembling it. After this inspection period, the agent must then prove its understanding by rebuilding the model from scratch using low-level action primitives. In order to facilitate research on this problem we have built LTRON, a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models. We pair this simulator with a new dataset of fan-made LEGO creations that have been uploaded to the internet in order to provide complex scenes containing over a thousand unique brick shapes. We take a first step towards solving this problem using sequence-to-sequence models that provide guidance for how to make progress on this challenging problem. Our simulator and data are available at github.com/aaronwalsman/ltron. Additional training code and PyTorch examples are available at github.com/aaronwalsman/ltron-torch-eccv22.

Related papers

• Learning to Build by Building Your Own Instructions [56.734927320020496]
  We develop a new technique for the recently proposed Break-and-Make problem in LTRON. An agent must learn to build a previously unseen LEGO assembly using a single interactive session. We train these models using online imitation learning which allows the model to learn from its own mistakes.
  arXiv  Detail & Related papers  (2024-10-01T22:39:58Z)
• TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
  We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning.
  arXiv  Detail & Related papers  (2024-07-22T14:05:27Z)
• Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks [53.09649785009528]
  In this paper, we explore a paradigm that does not require training to obtain new models. Similar to the birth of CNN inspired by receptive fields in the biological visual system, we propose Model Disassembling and Assembling. For model assembling, we present the alignment padding strategy and parameter scaling strategy to construct a new model tailored for a specific task.
  arXiv  Detail & Related papers  (2022-03-25T05:27:28Z)
• Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning [52.85981207514049]
  We introduce a novel formulation, complex construction, which requires a building agent to assemble unit primitives sequentially. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit information to the agent. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.
  arXiv  Detail & Related papers  (2021-10-29T01:09:51Z)
• Image2Lego: Customized LEGO Set Generation from Images [50.87935634904456]
  We implement a system that generates a LEGO brick model from 2D images. Models are obtained by algorithmic conversion of the 3D voxelized model to bricks. We generate step-by-step building instructions and animations for LEGO models of objects and human faces.
  arXiv  Detail & Related papers  (2021-08-19T03:42:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.