Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models

• URL: http://arxiv.org/abs/2510.16344v1
• Date: Sat, 18 Oct 2025 04:13:26 GMT
• Title: Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models
• Abstract summary: We consider connections as first-class primitives in assembly representation, including connector types, specifications, quantities, and placement locations.<n>We present Manual2Skill++, a vision-language framework that automatically extracts structured connection information from assembly manuals.<n>A large-scale vision-language model parses symbolic diagrams and annotations in manuals to instantiate these graphs, leveraging the rich connection knowledge embedded in human-designed instructions.
• Score: 26.61083683414806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Assembly hinges on reliably forming connections between parts; yet most robotic approaches plan assembly sequences and part poses while treating connectors as an afterthought. Connections represent the critical "last mile" of assembly execution, while task planning may sequence operations and motion plan may position parts, the precise establishment of physical connections ultimately determines assembly success or failure. In this paper, we consider connections as first-class primitives in assembly representation, including connector types, specifications, quantities, and placement locations. Drawing inspiration from how humans learn assembly tasks through step-by-step instruction manuals, we present Manual2Skill++, a vision-language framework that automatically extracts structured connection information from assembly manuals. We encode assembly tasks as hierarchical graphs where nodes represent parts and sub-assemblies, and edges explicitly model connection relationships between components. A large-scale vision-language model parses symbolic diagrams and annotations in manuals to instantiate these graphs, leveraging the rich connection knowledge embedded in human-designed instructions. We curate a dataset containing over 20 assembly tasks with diverse connector types to validate our representation extraction approach, and evaluate the complete task understanding-to-execution pipeline across four complex assembly scenarios in simulation, spanning furniture, toys, and manufacturing components with real-world correspondence.

Related papers

• Geometric 2D Scene Graph Generation [4.798313004706817]
  We propose a method for constructing scene graphs to represent and characterize assembly relationships.<n>Our approach does not rely on semantic data and is capable of handling a very small dataset.<n>We validate our method on a study dataset of toy model components which can be assembled into transportation vehicles.
  arXiv  Detail & Related papers  (2026-07-24T14:03:24Z)
• AC-VLA: Robust Out-of-Distribution Action Execution via Compositional Learning [52.47988016454267]
  Vision-Language-Action (VLA) models excel at end-to-end robotic manipulation but struggle with out-of-distribution (OOD) generalization.<n>We identify two mutually reinforcing modes: emphtrajectory overfitting, where models overfit to holistic trajectory patterns rather than compositional sub-skill semantics; and emphperceptual shortcut, where action tokens over-rely on wrist-view textures at the expense of global spatial grounding.<n>We introduce textbfAC-VLA, a plug-and-play Action Compositional learning
  arXiv  Detail & Related papers  (2026-07-17T07:51:03Z)
• AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects [36.01972955466693]
  Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies.<n>We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and part assembly trajectories.<n>We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories.
  arXiv  Detail & Related papers  (2026-05-13T00:44:09Z)
• Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality [40.836596733334254]
  We present an AR-assisted assembly workflow that leverages object recognition and hand tracking.<n>Using object recognition, the system detects and localizes components in real time to create a digital twin of the workspace.<n>A case study with LEGO blocks and custom 3D-printed components demonstrates how the system links digital instructions to physical assembly.
  arXiv  Detail & Related papers  (2025-11-22T22:49:40Z)
• AI Assisted AR Assembly: Object Recognition and Computer Vision for Augmented Reality Assisted Assembly [40.836596733334254]
  We present an AI-assisted Augmented Reality assembly workflow that uses deep learning-based object recognition.<n>For each assembly step, the system displays a bounding box around the corresponding components in the physical space, and where the component should be placed.
  arXiv  Detail & Related papers  (2025-11-07T16:20:53Z)
• Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models [30.640434157219715]
  We present Manual2Skill, a novel framework that enables robots to perform complex assembly tasks guided by high-level manual instructions.<n>Our approach leverages a Vision-Language Model (VLM) to extract structured information from instructional images and then uses this information to construct hierarchical assembly graphs.<n>We demonstrate the effectiveness of Manual2Skill by successfully assembling several real-world IKEA furniture items.
  arXiv  Detail & Related papers  (2025-02-14T11:25:24Z)
• Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following [50.377287115281476]
  We show that learning to associate the representations of current and future states with a temporal loss can improve compositional generalization.<n>We evaluate our approach across diverse robotic manipulation tasks as well as in simulation, showing substantial improvements for tasks specified with either language or goal images.
  arXiv  Detail & Related papers  (2025-02-08T05:26:29Z)
• Manual-PA: Learning 3D Part Assembly from Instruction Diagrams [54.555154845137906]
  We present Manual-PA, a transformer-based instruction Manual-guided 3D Part Assembly framework.<n>Our results show that using the diagrams and the order of the parts lead to significant improvements in assembly performance against the state of the art.
  arXiv  Detail & Related papers  (2024-11-27T03:10:29Z)
• TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly [51.29305265324916]
  We propose a class-agnostic tree-transformer framework to predict the sequential assembly actions from input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning.
  arXiv  Detail & Related papers  (2024-07-22T14:05:27Z)
• Embodied Instruction Following in Unknown Environments [64.57388036567461]
  We propose an embodied instruction following (EIF) method for complex tasks in the unknown environment.<n>We build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller.<n>For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues.
  arXiv  Detail & Related papers  (2024-06-17T17:55:40Z)
• Aligning Step-by-Step Instructional Diagrams to Video Demonstrations [51.67930509196712]
  We consider a novel setting where alignment is between (i) instruction steps that are depicted as assembly diagrams and (ii) video segments from in-the-wild videos.<n>We introduce a novel supervised contrastive learning method that learns to align videos with the subtle details in the assembly diagrams.<n>Experiments on IAW for Ikea assembly in the wild demonstrate superior performances of our approach against alternatives.
  arXiv  Detail & Related papers  (2023-03-24T04:45:45Z)
• IKEA-Manual: Seeing Shape Assembly Step by Step [26.79113677450921]
  We present IKEA-Manual, a dataset consisting of 102 IKEA objects paired with assembly manuals. We provide fine-grained annotations on the IKEA objects and assembly manuals, including assembly parts, assembly plans, manual segmentation, and 2D-3D correspondence between 3D parts and visual manuals.
  arXiv  Detail & Related papers  (2023-02-03T17:32:22Z)
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes [75.5824586200507]
  We present a deep learning network that learns dependencies and transitions across subtasks solely from a set of demonstration videos. We show that these symbols can be learned and predicted directly from image observations. We evaluate our framework on two long horizon tasks: (1) block stacking of puzzle pieces being executed by humans, and (2) a robot manipulation task involving pick and place of objects and sliding a cabinet door with a 7-DoF robot arm.
  arXiv  Detail & Related papers  (2020-06-08T18:07:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.