DefectTwin: When LLM Meets Digital Twin for Railway Defect Inspection

• URL: http://arxiv.org/abs/2409.06725v1
• Date: Mon, 26 Aug 2024 22:32:31 GMT
• Title: DefectTwin: When LLM Meets Digital Twin for Railway Defect Inspection
• Authors: Rahatara Ferdousi, M. Anwar Hossain, Chunsheng Yang, Abdulmotaleb El Saddik,
• Abstract summary: A Digital Twin (DT) replicates objects, processes, or systems for real-time monitoring, simulation, and predictive maintenance. Recent advancements like Large Language Models (LLMs) have revolutionized traditional AI systems and offer immense potential when combined with DT in industrial applications such as railway defect inspection. We introduce DefectTwin, which employs a multimodal and multi-model (M2) LLM-based AI pipeline to analyze both seen and unseen visual defects in railways.
• Score: 5.601042583221173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A Digital Twin (DT) replicates objects, processes, or systems for real-time monitoring, simulation, and predictive maintenance. Recent advancements like Large Language Models (LLMs) have revolutionized traditional AI systems and offer immense potential when combined with DT in industrial applications such as railway defect inspection. Traditionally, this inspection requires extensive defect samples to identify patterns, but limited samples can lead to overfitting and poor performance on unseen defects. Integrating pre-trained LLMs into DT addresses this challenge by reducing the need for vast sample data. We introduce DefectTwin, which employs a multimodal and multi-model (M^2) LLM-based AI pipeline to analyze both seen and unseen visual defects in railways. This application enables a railway agent to perform expert-level defect analysis using consumer electronics (e.g., tablets). A multimodal processor ensures responses are in a consumable format, while an instant user feedback mechanism (instaUF) enhances Quality-of-Experience (QoE). The proposed M^2 LLM outperforms existing models, achieving high precision (0.76-0.93) across multimodal inputs including text, images, and videos of pre-trained defects, and demonstrates superior zero-shot generalizability for unseen defects. We also evaluate the latency, token count, and usefulness of responses generated by DefectTwin on consumer devices. To our knowledge, DefectTwin is the first LLM-integrated DT designed for railway defect inspection.

Related papers

• Automated Detection of Defects on Metal Surfaces using Vision Transformers [1.6381055567716192]
  The study utilizes deep learning techniques to develop a model for detecting metal surface defects using Vision Transformers (ViTs) The proposed model focuses on the classification and localization of defects using a ViT for feature extraction. Experimental results show that it can be utilized in the process of automated defects detection, improve operational efficiency, and reduce errors in metal manufacturing.
  arXiv  Detail & Related papers  (2024-10-06T10:29:45Z)
• Investigation on domain adaptation of additive manufacturing monitoring systems to enhance digital twin reusability [12.425166883814153]
  Digital twin (DT) using machine learning (ML)-based modeling can be deployed for AM process monitoring and control. Melt pool is one of the most commonly observed physical phenomena for process monitoring. This paper proposes a knowledge transfer pipeline between different AM settings to enhance the reusability of AM DTs.
  arXiv  Detail & Related papers  (2024-09-19T13:54:01Z)
• An Evaluation of Continual Learning for Advanced Node Semiconductor Defect Inspection [0.11184789007828977]
  This work introduces a task-agnostic, meta-learning approach to semiconductor defect inspection. It enables the incremental addition of new defect classes and scales to create a more robust and generalized model. We have benchmarked our approach using real resist-wafer SEM (Scanning Electron Microscopy) datasets for two process steps, ADI and AEI.
  arXiv  Detail & Related papers  (2024-07-17T16:41:22Z)
• Incomplete Multimodal Industrial Anomaly Detection via Cross-Modal Distillation [0.0]
  multimodal industrial anomaly detection (IAD) based on 3D point clouds and RGB images remains a work in progress. Existing quality control processes combine rapid in-line inspections, such as optical and infrared imaging with high-resolution but time-consuming near-line characterization techniques. We propose CMDIAD, a Cross-Modal Distillation framework for IAD to demonstrate the feasibility of a Multi-modal Training, Few-modal Inference pipeline.
  arXiv  Detail & Related papers  (2024-05-22T12:08:56Z)
• Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection [59.41026558455904]
  We focus on multi-modal anomaly detection. Specifically, we investigate early multi-modal approaches that attempted to utilize models pre-trained on large-scale visual datasets. We propose a Local-to-global Self-supervised Feature Adaptation (LSFA) method to finetune the adaptors and learn task-oriented representation toward anomaly detection.
  arXiv  Detail & Related papers  (2024-01-06T07:30:41Z)
• Cal-DETR: Calibrated Detection Transformer [67.75361289429013]
  We propose a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR, UP-DETR and DINO. We develop an uncertainty-guided logit modulation mechanism that leverages the uncertainty to modulate the class logits. Results corroborate the effectiveness of Cal-DETR against the competing train-time methods in calibrating both in-domain and out-domain detections.
  arXiv  Detail & Related papers  (2023-11-06T22:13:10Z)
• Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detection [89.49244928440221]
  We propose a novel large multi-modal model by applying vision experts for industrial anomaly detection (dubbed Myriad) Specifically, we adopt MiniGPT-4 as the base LMM and design an Expert Perception module to embed the prior knowledge from vision experts as tokens which are intelligible to Large Language Models (LLMs) To compensate for the errors and confusions of vision experts, we introduce a domain adapter to bridge the visual representation gaps between generic and industrial images.
  arXiv  Detail & Related papers  (2023-10-29T16:49:45Z)
• Detecting train driveshaft damages using accelerometer signals and Differential Convolutional Neural Networks [67.60224656603823]
  This paper proposes the development of a railway axle condition monitoring system based on advanced 2D-Convolutional Neural Network (CNN) architectures. The resultant system converts the railway axle vibration signals into time-frequency domain representations, i.e., spectrograms, and, thus, trains a two-dimensional CNN to classify them depending on their cracks.
  arXiv  Detail & Related papers  (2022-11-15T15:04:06Z)
• Detecting Faults during Automatic Screwdriving: A Dataset and Use Case of Anomaly Detection for Automatic Screwdriving [80.6725125503521]
  Data-driven approaches, using Machine Learning (ML) for detecting faults have recently gained increasing interest. We present a use case of using ML models for detecting faults during automated screwdriving operations.
  arXiv  Detail & Related papers  (2021-07-05T11:46:00Z)
• Computer Vision and Normalizing Flow Based Defect Detection [0.0]
  We present a two-stage defect detection network based on the object detection model YOLO, and the normalizing flow-based defect detection model DifferNet. Our model has high robustness and performance on defect detection using real-world video clips taken from a production line monitoring system. Our proposed model can learn on a small number of defect-free samples of single or multiple product types.
  arXiv  Detail & Related papers  (2020-12-12T05:38:21Z)
• SUOD: Accelerating Large-Scale Unsupervised Heterogeneous Outlier Detection [63.253850875265115]
  Outlier detection (OD) is a key machine learning (ML) task for identifying abnormal objects from general samples. We propose a modular acceleration system, called SUOD, to address it.
  arXiv  Detail & Related papers  (2020-03-11T00:22:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.