Benchmark AUC Is Not Deployable Reliability: A Cross-Dataset Audit of Off-the-Shelf Features for Surveillance Video Anomaly Detection

• URL: http://arxiv.org/abs/2606.29506v1
• Date: Sun, 28 Jun 2026 17:08:02 GMT
• Title: Benchmark AUC Is Not Deployable Reliability: A Cross-Dataset Audit of Off-the-Shelf Features for Surveillance Video Anomaly Detection
• Abstract summary: Automated "suspicious behavior" flagging is a headline promise of AI surveillance.<n>Field reports high frame-level ROC-AUC on standard video anomaly detection benchmarks.<n>We audit what happens when that assumption is dropped.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated "suspicious behavior" flagging is a headline promise of AI surveillance, and the field reports high frame-level ROC-AUC on standard video anomaly detection benchmarks. Those numbers are measured by training and testing on the same camera and scene. We audit what happens when that assumption is dropped. We build an unsupervised normality model from the all-normal training frames of one dataset, using frozen off-the-shelf embeddings (CLIP, DINOv2, ResNet-50, EfficientNet-B0) and a nearest-neighbour distance, and score the test frames of the same and of other datasets. Across 4 real datasets (UCSD Ped1, UCSD Ped2, CUHK Avenue, ShanghaiTech) and 4 backbones, same-dataset AUC averages 0.704 but cross-dataset AUC averages 0.499, which is chance: a detector calibrated on one scene is no better than a coin flip on another, and in several pairs it is below chance. The strongest backbone makes this worse, not better: DINOv2 has the best same-dataset AUC (up to 0.901 on Ped2) and the largest cross-dataset drop. The collapse is not an artefact of the scoring rule: replacing the nearest-neighbour detector with a PaDiM-style Mahalanobis detector reproduces it almost exactly (cross-dataset gap 0.202 versus 0.208). Even at a favourable operating point the false-alarm rate is on the order of 31,931 per hour. We conclude that the benchmark numbers quoted for surveillance anomaly detection describe a calibrated laboratory setting and overstate deployable reliability by a wide margin, and we release the code that reproduces every number.

Related papers

• Provenance, Not Behaviour: A Serialisation Artifact in Edge-IIoTset and a Leakage-Free Benchmark for Precision-Agriculture Intrusion Detection [0.0]
  Edge-IIoTset is the benchmark for machine-learning intrusion detection in the Internet of Things.<n>We show that much of that performance is not intrusion detection.<n>We produce AgriEdge: 1,276,122 rows, five devices with full attribution, and no column separating the classes above 0.0288.
  arXiv  Detail & Related papers  (2026-08-16T14:30:45Z)
• Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection [0.0]
  We ask, independently and adversarially, whether replacement metrics resist no-skill detectors on real benchmarks.<n>We find the answer turns entirely on one overlooked variable: N, the number of random attempts an adversary reports the best of.<n>We recommend reporting single-run scores or disclosing N and preferring PR-based metrics, which resist best-of-N inflation on nearly every benchmark.
  arXiv  Detail & Related papers  (2026-07-12T22:33:07Z)
• RGLD: Randomized Global-Local Density Estimation for Tabular Anomaly Detection [10.363495989074226]
  Unsupervised tabular anomaly detection requires methods that are robust across heterogeneous datasets.<n>We propose RGLD, a randomized global-local density estimator for efficient unsupervised anomaly detection.<n> RGLD achieves the strongest dataset-level AUROC performance, ranking 1st in dataset wins, and ranks 2nd in AUPRC wins.
  arXiv  Detail & Related papers  (2026-06-27T15:20:09Z)
• Anticipating the Optimism Gap: Predicting Distribution-Shift Degradation of RF-Impairment Detectors from In-Distribution Statistics [0.0]
  Detectors for radio-frequency impairments are usually reported with a single AUC measured on the distribution they were tuned on.<n>We ask whether this optimism can be predicted before any out-of-distribution data is seen.
  arXiv  Detail & Related papers  (2026-06-20T14:10:59Z)
• The Distributed Detectability Band Against Marginal-Preserving Attacks [0.0]
  AI-control monitors score individual agent actions to detect misbehavior.<n>Real harm can be distributed across many benign-looking steps, each individually below any per-step alarm.<n>We construct a marginal-preserving, correlation-encoded distributed-sabotage attack.
  arXiv  Detail & Related papers  (2026-06-09T06:04:30Z)
• How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures [0.0]
  We find that VLA architectures fail in fundamentally different, predictable ways at the motor-command level.<n>We quantify consequence of the well-known discrete/continuous VLA distinction: the two families produce qualitatively different failure monitor works.
  arXiv  Detail & Related papers  (2026-05-27T16:44:55Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Benchmarking Fraud Detectors on Private Graph Data [70.4654745317714]
  Currently, many types of fraud are managed in part by automated detection algorithms that operate over graphs.<n>We consider the scenario where a data holder wishes to outsource development of fraud detectors to third parties.<n>Third parties submit their fraud detectors to the data holder, who evaluates these algorithms on a private dataset and then publicly communicates the results.<n>We propose a realistic privacy attack on this system that allows an adversary to de-anonymize individuals' data based only on the evaluation results.
  arXiv  Detail & Related papers  (2025-07-30T03:20:15Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Discriminative-Generative Dual Memory Video Anomaly Detection [81.09977516403411]
  Recently, people tried to use a few anomalies for video anomaly detection (VAD) instead of only normal data during the training process. We propose a DiscRiminative-gEnerative duAl Memory (DREAM) anomaly detection model to take advantage of a few anomalies and solve data imbalance.
  arXiv  Detail & Related papers  (2021-04-29T15:49:01Z)
• Real-Time Anomaly Detection in Edge Streams [49.26098240310257]
  We propose MIDAS, which focuses on detecting microcluster anomalies, or suddenly arriving groups of suspiciously similar edges. We further propose MIDAS-F, to solve the problem by which anomalies are incorporated into the algorithm's internal states. Experiments show that MIDAS-F has significantly higher accuracy than MIDAS.
  arXiv  Detail & Related papers  (2020-09-17T17:59:27Z)
• Detection in Crowded Scenes: One Proposal, Multiple Predictions [79.28850977968833]
  We propose a proposal-based object detector, aiming at detecting highly-overlapped instances in crowded scenes. The key of our approach is to let each proposal predict a set of correlated instances rather than a single one in previous proposal-based frameworks. Our detector can obtain 4.9% AP gains on challenging CrowdHuman dataset and 1.0% $textMR-2$ improvements on CityPersons dataset.
  arXiv  Detail & Related papers  (2020-03-20T09:48:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.