Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

• URL: http://arxiv.org/abs/2606.05445v1
• Date: Wed, 03 Jun 2026 21:08:06 GMT
• Title: Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
• Abstract summary: We study whether multimodal large language models (MLLMs) possess the visual grounding and spatial reasoning capabilities required for brick assembly.<n>We introduce BC-Bench, the first benchmark for evaluating MLLMs on assembly with diverse bricks.<n>We propose Brick-Composer, a learning framework that equips MLLMs with assembly skills through three complementary signals.
• Score: 64.5380622477211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks. As a first step toward this vision, we study whether multimodal large language models (MLLMs) possess the visual grounding and spatial reasoning capabilities required for brick assembly. We formulate brick assembly as a sequential decision-making problem, where each step involves two subtasks: brick selection, identifying the target brick from candidate components, and brick pose estimation, predicting where and how the selected brick should be placed. To support this study, we introduce BC-Bench (Brick Construction Benchmark), the first benchmark for evaluating MLLMs on assembly with diverse bricks. Experiments show that current state-of-the-art MLLMs remain far from reliable builders, struggling with fine-grained brick selection and failing at precise pose estimation. To bridge this gap, we propose Brick-Composer, a learning framework that equips MLLMs with assembly skills through three complementary signals: Human Design Sparks, which provide affordance-rich construction demonstrations; World Feedback, which grounds predicted actions in visual and physical consequences; and Synthetic Experience, which scales learning beyond existing object designs. Brick-Composer improves brick selection accuracy by over three times, substantially reduces pose estimation errors, and raises strict step-level assembly success from less than 1% to around 15%. After training, a Qwen-3-8B can correctly compose up to 42% of the steps for a complete object, suggesting that MLLMs can acquire assembly capabilities through targeted, physically grounded learning.

Related papers

• Brick-DICL: Dynamic In-Context Learning for Automated Brick Schema Classification [5.1315088460488525]
  We propose Brick-DICL, a two-stage dynamic in-context learning framework for automated Brick schema classification.<n> Brick-DICL consists of two primary components: metadata-RAG, which retrieves relevant examples, and class-RAG, which narrows down potential Brick classes.<n>Extensive experiments demonstrate Brick-DICL's effectiveness across diverse building datasets.
  arXiv  Detail & Related papers  (2026-06-16T07:46:51Z)
• BrickCraft: Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly [13.976436316430911]
  BrickCraft is a compositional framework designed for long-horizon and generalizable interlocking brick assembly.<n>It bridges the gap between high-level assembly plans and physical execution through situated manuals.<n>BrickCraft exhibits strong compositional generalization to unseen structures.
  arXiv  Detail & Related papers  (2026-05-08T11:30:25Z)
• Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models [75.45940282834327]
  We introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs.<n>We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs.<n>Our approach employs a two-stage fine-tuning strategy, resulting in significant improvements across multiple tasks.
  arXiv  Detail & Related papers  (2025-11-03T14:27:00Z)
• LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning? [23.90259639381836]
  LEGO-Puzzles consists of 1,100 carefully curated visual question-answering (VQA) samples spanning 11 distinct tasks.<n>Even the most powerful MLLMs can answer only about half of the test cases, whereas human participants achieve over 90% accuracy.<n>Our experiments show that only GPT-4o and Gemini-2.0-Flash exhibit a limited ability to follow these instructions, while other MLLMs either replicate the input image or generate completely irrelevant outputs.
  arXiv  Detail & Related papers  (2025-03-25T18:21:07Z)
• Configurable Foundation Models: Building LLMs from a Modular Perspective [115.63847606634268]
  A growing tendency to decompose LLMs into numerous functional modules allows for inference with part of modules and dynamic assembly of modules to tackle complex tasks. We coin the term brick to represent each functional module, designating the modularized structure as customizable foundation models. We present four brick-oriented operations: retrieval and routing, merging, updating, and growing. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions.
  arXiv  Detail & Related papers  (2024-09-04T17:01:02Z)
• TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
  We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning.
  arXiv  Detail & Related papers  (2024-07-22T14:05:27Z)
• CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [74.22729793816451]
  Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability. We propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems.
  arXiv  Detail & Related papers  (2023-05-23T17:51:52Z)
• Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction [63.672314717599285]
  We tackle the problem of sequential brick assembly with LEGO bricks to create 3D structures. In particular, the number of assemblable structures increases exponentially as the number of bricks used increases. We propose a new method to predict the scores of the next brick position by employing a U-shaped sparse 3D convolutional neural network.
  arXiv  Detail & Related papers  (2022-10-03T15:35:08Z)
• Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning [52.85981207514049]
  We introduce a novel formulation, complex construction, which requires a building agent to assemble unit primitives sequentially. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit information to the agent. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.
  arXiv  Detail & Related papers  (2021-10-29T01:09:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.