Few-shot Structure-Informed Machinery Part Segmentation with Foundation Models and Graph Neural Networks

• URL: http://arxiv.org/abs/2501.10080v1
• Date: Fri, 17 Jan 2025 09:55:05 GMT
• Title: Few-shot Structure-Informed Machinery Part Segmentation with Foundation Models and Graph Neural Networks
• Authors: Michael Schwingshackl, Fabio Francisco Oberweger, Markus Murschitz,
• Abstract summary: We propose a novel approach to few-shot semantic segmentation for machinery with multiple parts that exhibit spatial and hierarchical relationships.<n>Our method integrates the foundation models CLIPSeg and Segment Anything Model (SAM) with the interest point detector SuperPoint and a graph convolutional network (GCN) to accurately segment machinery parts.<n>Our model, evaluated on a purely synthetic dataset depicting a truck-mounted loading crane, achieves effective segmentation across various levels of detail.
• Score: 1.5293427903448022
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper proposes a novel approach to few-shot semantic segmentation for machinery with multiple parts that exhibit spatial and hierarchical relationships. Our method integrates the foundation models CLIPSeg and Segment Anything Model (SAM) with the interest point detector SuperPoint and a graph convolutional network (GCN) to accurately segment machinery parts. By providing 1 to 25 annotated samples, our model, evaluated on a purely synthetic dataset depicting a truck-mounted loading crane, achieves effective segmentation across various levels of detail. Training times are kept under five minutes on consumer GPUs. The model demonstrates robust generalization to real data, achieving a qualitative synthetic-to-real generalization with a $J\&F$ score of 92.2 on real data using 10 synthetic support samples. When benchmarked on the DAVIS 2017 dataset, it achieves a $J\&F$ score of 71.5 in semi-supervised video segmentation with three support samples. This method's fast training times and effective generalization to real data make it a valuable tool for autonomous systems interacting with machinery and infrastructure, and illustrate the potential of combined and orchestrated foundation models for few-shot segmentation tasks.

Related papers

• Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark [12.231630639022335]
  We propose SAV, a novel framework comprising three core components: a SAM-based encoder-decoder, a vehicle part knowledge graph, and a context sample retrieval encoding module.<n>The knowledge graph explicitly models the spatial and geometric relationships among vehicle parts through a structured ontology, effectively encoding prior structural knowledge.<n>We introduce a new large-scale benchmark dataset for vehicle part segmentation, named VehicleSeg10K, which contains 11,665 high-quality pixel-level annotations.
  arXiv  Detail & Related papers  (2025-08-06T09:46:49Z)
• DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data [61.62554324594797]
  We propose DreamMask, which explores how to generate training data in the open-vocabulary setting, and how to train the model with both real and synthetic data.<n>In general, DreamMask significantly simplifies the collection of large-scale training data, serving as a plug-and-play enhancement for existing methods.<n>For instance, when trained on COCO and tested on ADE20K, the model equipped with DreamMask outperforms the previous state-of-the-art by a substantial margin of 2.1% mIoU.
  arXiv  Detail & Related papers  (2025-01-03T19:00:00Z)
• Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
  We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals. Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars. Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
  arXiv  Detail & Related papers  (2023-12-18T18:59:51Z)
• Zero-Shot Refinement of Buildings' Segmentation Models using SAM [6.110856077714895]
  We present a novel approach to adapt foundation models to address existing models' generalization dropback. Among several models, our focus centers on the Segment Anything Model (SAM) SAM does not offer recognition abilities and thus fails to classify and tag localized objects. This novel approach augments SAM with recognition abilities, a first of its kind.
  arXiv  Detail & Related papers  (2023-10-03T07:19:59Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Part-Based Models Improve Adversarial Robustness [57.699029966800644]
  We show that combining human prior knowledge with end-to-end learning can improve the robustness of deep neural networks. Our model combines a part segmentation model with a tiny classifier and is trained end-to-end to simultaneously segment objects into parts. Our experiments indicate that these models also reduce texture bias and yield better robustness against common corruptions and spurious correlations.
  arXiv  Detail & Related papers  (2022-09-15T15:41:47Z)
• Segmenting Moving Objects via an Object-Centric Layered Representation [100.26138772664811]
  We introduce an object-centric segmentation model with a depth-ordered layer representation. We introduce a scalable pipeline for generating synthetic training data with multiple objects. We evaluate the model on standard video segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-07-05T17:59:43Z)
• MSeg: A Composite Dataset for Multi-domain Semantic Segmentation [100.17755160696939]
  We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains. We reconcile the generalization and bring the pixel-level annotations into alignment by relabeling more than 220,000 object masks in more than 80,000 images. A model trained on MSeg ranks first on the WildDash-v1 leaderboard for robust semantic segmentation, with no exposure to WildDash data during training.
  arXiv  Detail & Related papers  (2021-12-27T16:16:35Z)
• Chair Segments: A Compact Benchmark for the Study of Object Segmentation [12.16129964498819]
  ChairSegments is a novel and compact semi-synthetic dataset for object segmentation. We show empirical findings in transfer learning that mirror recent findings for image classification.
  arXiv  Detail & Related papers  (2020-12-02T14:54:03Z)
• Semantic Segmentation With Multi Scale Spatial Attention For Self Driving Cars [2.7317088388886384]
  We present a novel neural network using multi scale feature fusion at various scales for accurate and efficient semantic image segmentation. We used ResNet based feature extractor, dilated convolutional layers in downsampling part, atrous convolutional layers in the upsampling part and used concat operation to merge them. A new attention module is proposed to encode more contextual information and enhance the receptive field of the network.
  arXiv  Detail & Related papers  (2020-06-30T20:19:09Z)
• Unsupervised Learning Consensus Model for Dynamic Texture Videos Segmentation [12.462608802359936]
  We present an effective unsupervised learning consensus model for the segmentation of dynamic texture (ULCM) In the proposed model, the set of values of the requantized local binary patterns (LBP) histogram around the pixel to be classified are used as features. Experiments conducted on the challenging SynthDB dataset show that ULCM is significantly faster, easier to code, simple and has limited parameters.
  arXiv  Detail & Related papers  (2020-06-29T16:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.