Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection

• URL: http://arxiv.org/abs/2607.11969v2
• Date: Sun, 19 Jul 2026 06:38:29 GMT
• Title: Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection
• Abstract summary: We ask, independently and adversarially, whether replacement metrics resist no-skill detectors on real benchmarks.<n>We find the answer turns entirely on one overlooked variable: N, the number of random attempts an adversary reports the best of.<n>We recommend reporting single-run scores or disclosing N and preferring PR-based metrics, which resist best-of-N inflation on nearly every benchmark.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Point-adjustment (PA), for years the default scoring protocol in time-series anomaly detection (TSAD), was shown by Kim et al. (2022) to award near-perfect F1 to random anomaly scores. The field adopted a suite of replacement metrics (PA%K, range-based precision/recall, affiliation precision/recall, and Volume-Under-the-Surface, VUS, ROC/PR). We ask, independently and adversarially, whether these resist no-skill detectors on real benchmarks, and find the answer turns entirely on one overlooked variable: N, the number of random attempts an adversary reports the best of. Under a single honest run (N=1), not one replacement metric is gameable on any of six benchmarks (UCR, SMD, SMAP, MSL, NAB, PSM): a random detector reaches 90% of the best real detector's score on at most 11% of series for affiliation-F1, 5% for the ROC family, and 2% for the PR-based metrics and PA%K. But under best-of-N reporting, the seed-shopping endemic to ML, the metrics split sharply. affiliation-F1 and every ROC-based metric inflate steeply, affiliation crossing gameable (25% of series) by N=3 and reaching 0.98 at the full pool (N=41), the ROC family crossing by N=9-11; the PR-based metrics and PA%K stay near-flat at every N, floored near the anomaly prevalence (the lone exception is NAB at large N). A paired test finds VUS-ROC inflated on 131 series where its sibling VUS-PR is not, and never the reverse. The ROC-vs-PR split follows from the order-statistic behaviour of AUC under extreme class imbalance (a random PR-AUC is floored at prevalence); affiliation inflates by a second route, its extreme single-run leniency (already fragile at N=1). We release a pip-installable stress-test harness, and recommend reporting single-run scores or disclosing N and preferring PR-based metrics, which resist best-of-N inflation on nearly every benchmark.

Related papers

• Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection [66.44527094471619]
  We present RECAL, an unsupervised framework using relation-balanced masked graph learning to better capture rare interaction patterns.<n>On three DARPA E3 datasets, RECAL achieves F1 scores of 99.99%, 99.93%, and 99.99%, outperforming the best baseline on each dataset by 0.88, 0.82, and 0.42 percentage points, respectively.
  arXiv  Detail & Related papers  (2026-09-15T00:31:01Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy [17.48959208263029]
  We introduce emphmean-to-score (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map.<n>For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly.
  arXiv  Detail & Related papers  (2026-07-23T14:37:30Z)
• Benchmark AUC Is Not Deployable Reliability: A Cross-Dataset Audit of Off-the-Shelf Features for Surveillance Video Anomaly Detection [0.0]
  Automated "suspicious behavior" flagging is a headline promise of AI surveillance.<n>Field reports high frame-level ROC-AUC on standard video anomaly detection benchmarks.<n>We audit what happens when that assumption is dropped.
  arXiv  Detail & Related papers  (2026-06-28T17:08:02Z)
• Beyond ROC-AUC: Operating-Point Performance Reporting for Biometric Verification [0.0]
  A biometric verifier is often deployed with a strict false match budget, so only a narrow, low false match rate (FMR) slice of the score range is used.<n>In practice, a single area under the receiver operating characteristic curve (ROC-AUC) is still reported as the resolution.<n>A system that looks stronger on full ROC-AUC is shown to be significantly worse at FMR = 10-3.
  arXiv  Detail & Related papers  (2026-06-13T15:28:54Z)
• Precision Is Not Faithfulness: Coverage-Aware Evaluation of Grounded Generation with a Complete Oracle [0.0]
  Reference-free faithfulness metrics verify each atomic claim a model makes against ground truth.<n>We show they share a blind spot: they measure only precision, are the stated claims supported?<n>We make this measurable using Formula 1 telemetry, a domain where strategic ground truth is derived deterministically.
  arXiv  Detail & Related papers  (2026-06-08T11:56:25Z)
• Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference [48.78219918881965]
  We propose a principled framework for deriving task-specific strictly proper scoring rules by matching the local curvature of the downstream error metric.<n>We apply this to the Average Treatment Effect (ATE) estimation, deriving a closed-form loss and its corresponding canonical probability mapping.
  arXiv  Detail & Related papers  (2026-06-02T08:41:25Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score [14.579552536669217]
  Unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation Code.<n>We present the first study of reliability in multimodal unlearning.<n>We introduce the Unified Quality Score (UQS), a metric with weights derived from each metric's Spearman correlation with the metric distance.
  arXiv  Detail & Related papers  (2026-05-04T04:13:00Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• Target before Shooting: Accurate Anomaly Detection and Localization under One Millisecond via Cascade Patch Retrieval [49.45246833329707]
  We re-examine the "matching" nature of Anomaly Detection (AD) We propose a new AD framework that simultaneously enjoys new records of AD accuracy and dramatically high running speed.
  arXiv  Detail & Related papers  (2023-08-13T11:49:05Z)
• Detection Transformer with Stable Matching [48.963171068785435]
  We show that the most important design is to use and only use positional metrics to supervise classification scores of positive examples. Under the principle, we propose two simple yet effective modifications by integrating positional metrics to DETR's classification loss and matching cost. We achieve 50.4 and 51.5 AP on the COCO detection benchmark using ResNet-50 backbones under 12 epochs and 24 epochs training settings.
  arXiv  Detail & Related papers  (2023-04-10T17:55:37Z)
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details [107.2722027807328]
  We find that the default implementation of AP is neither category independent, nor does it directly reward properly calibrated detectors. We show that the default implementation produces a gameable metric, where a simple, nonsensical re-ranking policy can improve AP by a large margin. We benchmark recent advances in large-vocabulary detection and find that many reported gains do not translate to improvements under our new per-class independent evaluation.
  arXiv  Detail & Related papers  (2021-02-01T18:56:02Z)
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times [51.61278695776151]
  Federated Learning (FL) is well known for its privacy protection when training machine learning models among distributed clients collaboratively. Recent studies have pointed out that the naive FL is susceptible to gradient leakage attacks. Differential Privacy (DP) emerges as a promising countermeasure to defend against gradient leakage attacks.
  arXiv  Detail & Related papers  (2021-01-11T19:43:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.