From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection

• URL: http://arxiv.org/abs/2608.07770v1
• Date: Fri, 07 Aug 2026 21:31:20 GMT
• Title: From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection
• Abstract summary: We evaluate 19 unsupervised anomaly detection models on the BowTie dataset.<n>We observe that model performance is less stable than typically reported on standard benchmarks.<n>We deploy a unified human-in-the-loop framework for manufactured-part inspection.
• Score: 28.67239775192971
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated anomaly detection methods often report strong performance on curated academic benchmarks, but their behavior under real-world industrial conditions is less clear. In this work, we evaluate 19 unsupervised anomaly detection models on the BowTie dataset, a challenging manufacturing dataset with reflective surfaces, subtle defects, and profile-specific variation. In contrast to benchmark results, we observe that model performance is less stable than typically reported on standard benchmarks such as MVTec AD, highly sensitive to preprocessing, and inconsistent across conditions, with no single approach emerging as uniformly robust; a consensus audit further indicates that nominal-data quality affects deployment. Motivated by these findings, we developed and initially deployed a unified human-in-the-loop framework for manufactured-part inspection that combines image annotation, AI-assisted defect detection, and an integrated validation engine, replacing a prior manual visual inspection and documentation workflow. The system supports heatmap-guided defect review, SAM-refined candidate regions for inspector acceptance, rejection, or boundary adjustment, mask evaluation where annotations exist, and review history for inspector consistency and onboarding. Together, the results highlight the gap between benchmark performance and deployment reality, and provide a practical framework for addressing it.

Related papers

• From Signals to Transfer: A Factorised Study of Probe-Based Uncertainty Estimation in Large Language Models [51.470637128942336]
  We propose a factorised study of probe-based uncertainty estimation (UE) under matched conditions.<n>Our results show that raw hidden states and attention features are difficult to outperform in-domain.<n>We train benchmark-based pretrained probes that transfer reasonably well to open-ended factual generation.
  arXiv  Detail & Related papers  (2026-06-26T03:23:15Z)
• Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks [0.36748639131154315]
  This work investigates the robustness of established benchmarking setups and examines how to measure intrinsic biases.<n>Our experiments uncover significant discrepancies in benchmark behaviors when evaluation setups are altered.<n>We observe model-specific instabilities, demonstrating a clear need for more robust and comprehensive safety evaluation frameworks.
  arXiv  Detail & Related papers  (2026-05-11T14:27:39Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• K$α$LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks [4.297070083645049]
  We propose K$$LOS, a unified meta-algorithm that generalizes the "Localization First" principle to dataset quality evaluation.<n>By resolving spatial correspondence before assessing agreement, our framework transforms complex-categorical problems into nominal reliability.
  arXiv  Detail & Related papers  (2026-03-28T08:54:05Z)
• ProbeLLM: Automating Principled Diagnosis of LLM Failures [89.44131968886184]
  We propose ProbeLLM, a benchmark-agnostic automated probing framework that elevates weakness discovery from individual failures to structured failure modes.<n>By restricting probing to verifiable test cases and leveraging tool-augmented generation and verification, ProbeLLM grounds failure discovery in reliable evidence.
  arXiv  Detail & Related papers  (2026-02-13T14:33:13Z)
• Toward Faithful Explanations in Acoustic Anomaly Detection [21.487734134424187]
  Interpretability is essential for user trust in real-world anomaly detection applications.<n>We compare a standard autoencoder (AE) with a mask autoencoder (MAE) in terms of detection performance and interpretability.<n>MAE consistently provides more faithful and temporally precise explanations, suggesting a better alignment with true anomalies.
  arXiv  Detail & Related papers  (2026-01-19T02:16:37Z)
• Correcting False Alarms from Unseen: Adapting Graph Anomaly Detectors at Test Time [60.341117019125214]
  We propose a lightweight and plug-and-play Test-time adaptation framework for correcting Unseen Normal pattErns in graph anomaly detection (GAD)<n>To address semantic confusion, a graph aligner is employed to align the shifted data to the original one at the graph attribute level.<n>Extensive experiments on 10 real-world datasets demonstrate that TUNE significantly enhances the generalizability of pre-trained GAD models to both synthetic and real unseen normal patterns.
  arXiv  Detail & Related papers  (2025-11-10T12:10:05Z)
• ISP-AD: A Large-Scale Real-World Dataset for Advancing Industrial Anomaly Detection with Synthetic and Real Defects [0.0]
  TheISP-AD is the largest publicly available industrial dataset to date, including both synthetic and real defects collected directly from the factory floor.<n>Experiments show that even a small amount of injected, weakly labeled real defects improves generalization.
  arXiv  Detail & Related papers  (2025-03-06T21:56:31Z)
• AUPIMO: Redefining Visual Anomaly Detection Benchmarks with High Speed and Low Tolerance [0.562479170374811]
  Per-IMage Overlap (PIMO) is a novel metric that addresses the shortcomings of AUROC and AUPRO. measuring recall per image simplifies computation and is more robust to noisy annotations. Our experiments demonstrate that PIMO offers practical advantages and nuanced performance insights.
  arXiv  Detail & Related papers  (2024-01-03T21:24:44Z)
• An Outlier Exposure Approach to Improve Visual Anomaly Detection Performance for Mobile Robots [76.36017224414523]
  We consider the problem of building visual anomaly detection systems for mobile robots. Standard anomaly detection models are trained using large datasets composed only of non-anomalous data. We tackle the problem of exploiting these data to improve the performance of a Real-NVP anomaly detection model.
  arXiv  Detail & Related papers  (2022-09-20T15:18:13Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.