AI-based worker guidance in assembly and disassembly operations using multimodal ego/exo-centric data capture and structured task knowledge

• URL: http://arxiv.org/abs/2608.22617v1
• Date: Sun, 23 Aug 2026 21:41:33 GMT
• Title: AI-based worker guidance in assembly and disassembly operations using multimodal ego/exo-centric data capture and structured task knowledge
• Abstract summary: Assembly and disassembly processes rely on expert knowledge that is difficult to document, reuse, and transfer.<n>This paper presents a data-centric approach for extracting structured task knowledge from expert demonstrations using egocentric and exocentric recordings.
• Score: 9.167082845109439
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Assembly and disassembly processes rely on expert knowledge that is difficult to document, reuse, and transfer. This paper presents a data-centric approach for extracting structured task knowledge from expert demonstrations using egocentric and exocentric recordings. Temporal and multimodal information from video and narration is jointly encoded to derive structured task representations that enable procedural documentation and context-aware worker guidance. The approach is evaluated on a real-world disassembly case study, demonstrating that video-based representations capture procedural structure and execution context beyond static image-based methods. The results highlight the potential of egocentric video understanding for repair, training, and circular manufacturing applications. Project website: https://indego-assistant.github.io/

Related papers

• Reflecting Process Expertise in Procedural Material Generation [47.15641864882028]
  Procedural material creation underpins applications in digital content creation, visual effects, and 3D asset design.<n>We represent expert synthesis as process traces: textual records of construction steps, parameters, and design intent.<n>We leverage tutorial videos as a source of process knowledge and extract textual traces using automated video analysis tools.
  arXiv  Detail & Related papers  (2026-07-14T22:59:32Z)
• Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance [55.32799307123252]
  We introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets.<n>We propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance.
  arXiv  Detail & Related papers  (2026-03-02T18:46:28Z)
• Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding [12.083424392460406]
  We introduce'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors procedures to what a model can actually see.<n>We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states.<n>Our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction.
  arXiv  Detail & Related papers  (2025-11-25T08:46:11Z)
• Generative Timelines for Instructed Visual Assembly [106.80501761556606]
  The objective of this work is to manipulate visual timelines (e.g. a video) through natural language instructions. We propose the Timeline Assembler, a generative model trained to perform instructed visual assembly tasks.
  arXiv  Detail & Related papers  (2024-11-19T07:26:30Z)
• Attention Where It Matters: Rethinking Visual Document Understanding with Selective Region Concentration [26.408343160223517]
  We propose a novel end-to-end document understanding model called SeRum. SeRum converts image understanding and recognition tasks into a local decoding process of the visual tokens of interest. We show that SeRum achieves state-of-the-art performance on document understanding tasks and competitive results on text spotting tasks.
  arXiv  Detail & Related papers  (2023-09-03T10:14:34Z)
• Chain of Thought Imitation with Procedure Cloning [129.62135987416164]
  We propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations. We show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations.
  arXiv  Detail & Related papers  (2022-05-22T13:14:09Z)
• Shedding Light on Blind Spots: Developing a Reference Architecture to Leverage Video Data for Process Mining [0.0]
  We propose a reference architecture to bridge the gap between computer vision and process mining. A prototype instantiation of the proposed reference architecture is capable of automatically extracting most of the process-relevant events from unstructured video data.
  arXiv  Detail & Related papers  (2020-10-21T20:01:52Z)
• Knowledge-Aware Procedural Text Understanding with Multi-Stage Training [110.93934567725826]
  We focus on the task of procedural text understanding, which aims to comprehend such documents and track entities' states and locations during a process. Two challenges, the difficulty of commonsense reasoning and data insufficiency, still remain unsolved. We propose a novel KnOwledge-Aware proceduraL text understAnding (KOALA) model, which effectively leverages multiple forms of external knowledge.
  arXiv  Detail & Related papers  (2020-09-28T10:28:40Z)
• A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos [126.66212285239624]
  We propose a benchmark of structured procedural knowledge extracted from cooking videos. Our manually annotated open-vocabulary resource includes 356 instructional cooking videos and 15,523 video clip/sentence-level annotations.
  arXiv  Detail & Related papers  (2020-05-02T05:15:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.