PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation

• URL: http://arxiv.org/abs/2604.09701v1
• Date: Tue, 07 Apr 2026 08:29:11 GMT
• Title: PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation
• Abstract summary: PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer feature spaces.<n>Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation.<n>While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 3.5% 6 IoU) segmentation performance in the industrial and agricultural domain.
• Score: 4.14197005718384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting unseen anomalies in unstructured environments presents a critical challenge for industrial and agricultural applications such as material recycling and weeding. Existing perception systems frequently fail to satisfy the strict operational requirements of these domains, specifically real-time processing, pixel-level segmentation precision, and robust accuracy, due to their reliance on exhaustively annotated datasets. To address these limitations, we propose a weakly supervised pipeline for object segmentation and classification using weak image-level supervision called 'Patch Aggregation for Segmentation of Targets and Anomalies' (PASTA). By comparing an observed scene with a nominal reference, PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer (ViT) feature spaces. Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation. Evaluations on a custom steel scrap recycling dataset and a plant dataset demonstrate a 75.8% training time reduction of our approach to domain-specific baselines. While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 63.5% IoU) segmentation performance in the industrial and agricultural domain.

Related papers

• Grounding Agentic VLMs with Dedicated Segmentation for Fine-Grained Vehicle Damage Assessment [0.0]
  Vision-language models (VLMs) are increasingly deployed as reasoning agents in real-world visual assessment pipelines.<n>We show that a state-of-the-art VLM (Qwen-VL) achieves strong semantic classification accuracy (87.3%) on this task but is systematically ungrounded at the spatial level.<n>We propose TinyDamage, a hybrid architecture that delegates spatial grounding to a dedicated multi-task segmentation model.<n>We show that this grounding reduces the report hallucination rate from 92% (text-only) and 78% (image-only) to 31% in a controlled evaluation on 100 human-verified reports
  arXiv  Detail & Related papers  (2026-08-03T16:37:49Z)
• Seek to Segment: Active Perception for Panoramic Referring Segmentation [76.39419654494505]
  Existing referring segmentation models passively process static images captured from fixed perspectives.<n>We propose PanoSeeker, a memory-augmented agent for efficient Active Panoramic Referring.<n>Experiments on our newly established APRS benchmark demonstrate that PanoSeeker achieves superior search efficiency and segmentation accuracy.
  arXiv  Detail & Related papers  (2026-07-02T17:56:49Z)
• Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Unstructured Outdoor Environments by Leveraging Synthetic Data [20.795406392781647]
  Existing vision-based traversability estimation methods rely on robot-specific annotations or semantic class mappings.<n>We propose a transformer-based architecture that jointly performs class-specific semantic segmentation and class-agnostic terrain segmentation.<n>We present the EXTerra dataset, providing real-world images annotated with both class-specific and class-agnostic terrain labels.
  arXiv  Detail & Related papers  (2026-05-26T20:04:19Z)
• FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios [58.34124792457706]
  The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
  arXiv  Detail & Related papers  (2026-04-08T12:23:27Z)
• Prompt-Driven Lightweight Foundation Model for Instance Segmentation-Based Fault Detection in Freight Trains [6.6492664858930475]
  We propose a lightweight self-prompted instance segmentation framework tailored for freight train fault detection.<n>Our method leverages the Segment Anything Model by introducing a self-prompt generation module that automatically produces task-specific prompts.<n>In addition, we adopt a Tiny Vision Transformer backbone to reduce computational cost, making the framework suitable for real-time deployment on edge devices in railway monitoring systems.
  arXiv  Detail & Related papers  (2026-03-13T03:56:37Z)
• Edge-Optimized Vision-Language Models for Underground Infrastructure Assessment [1.5124107808802705]
  This paper presents a novel two-stage pipeline for end-to-end summarization of underground deficiencies.<n>It combines our lightweight RAPID-SCAN segmentation model with a fine-tuned Vision-Language Model deployed on edge computing platform.<n>Our results show the potential of edge-deployable integrated AI systems to bridge the gap between automated defect detection and actionable insights for infrastructure maintenance.
  arXiv  Detail & Related papers  (2026-02-03T17:03:46Z)
• Instance-Guided Unsupervised Domain Adaptation for Robotic Semantic Segmentation [4.556831252263135]
  Unsupervised Domain Adaptation (UDA) adapts the network to the robot's target environment without external supervision.<n>We propose a method that starts from a 3D map to generate multi-view consistent pseudo-labels.<n>We refine these labels using the zero-shot instance segmentation capabilities of a foundation model, enforcing instance-level coherence.
  arXiv  Detail & Related papers  (2026-02-01T18:49:03Z)
• SOPSeg: Prompt-based Small Object Instance Segmentation in Remote Sensing Imagery [19.743431031185736]
  We propose SOPSeg, a prompt-based framework specifically designed for small object segmentation in remote sensing imagery.<n>It incorporates a region-adaptive magnification strategy to preserve fine-grained details, and employs a customized decoder that integrates edge prediction and progressive refinement for accurate boundary delineation.<n> SOPSeg outperforms existing methods in small object segmentation and facilitates efficient dataset construction for remote sensing tasks.
  arXiv  Detail & Related papers  (2025-09-03T04:25:03Z)
• Stanceformer: Target-Aware Transformer for Stance Detection [59.69858080492586]
  Stance Detection involves discerning the stance expressed in a text towards a specific subject or target. Prior works have relied on existing transformer models that lack the capability to prioritize targets effectively. We introduce Stanceformer, a target-aware transformer model that incorporates enhanced attention towards the targets during both training and inference.
  arXiv  Detail & Related papers  (2024-10-09T17:24:28Z)
• Fully and Weakly Supervised Referring Expression Segmentation with End-to-End Learning [50.40482222266927]
  Referring Expression (RES) is aimed at localizing and segmenting the target according to the given language expression. We propose a parallel position- kernel-segmentation pipeline to better isolate and then interact with the localization and segmentation steps. Our method is simple but surprisingly effective, outperforming all previous state-of-the-art RES methods on fully- and weakly-supervised settings.
  arXiv  Detail & Related papers  (2022-12-17T08:29:33Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• ISTR: End-to-End Instance Segmentation with Transformers [147.14073165997846]
  We propose an instance segmentation Transformer, termed ISTR, which is the first end-to-end framework of its kind. ISTR predicts low-dimensional mask embeddings, and matches them with ground truth mask embeddings for the set loss. Benefiting from the proposed end-to-end mechanism, ISTR demonstrates state-of-the-art performance even with approximation-based suboptimal embeddings.
  arXiv  Detail & Related papers  (2021-05-03T06:00:09Z)
• Unsupervised Instance Segmentation in Microscopy Images via Panoptic Domain Adaptation and Task Re-weighting [86.33696045574692]
  We propose a Cycle Consistency Panoptic Domain Adaptive Mask R-CNN (CyC-PDAM) architecture for unsupervised nuclei segmentation in histopathology images. We first propose a nuclei inpainting mechanism to remove the auxiliary generated objects in the synthesized images. Secondly, a semantic branch with a domain discriminator is designed to achieve panoptic-level domain adaptation.
  arXiv  Detail & Related papers  (2020-05-05T11:08:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.