Prior-matched evaluation of operational Earth-observation classifiers: a three-number reporting method demonstrated on Sentinel-1 internal-wave detection

• URL: http://arxiv.org/abs/2607.07146v1
• Date: Wed, 08 Jul 2026 08:38:40 GMT
• Title: Prior-matched evaluation of operational Earth-observation classifiers: a three-number reporting method demonstrated on Sentinel-1 internal-wave detection
• Abstract summary: A model that scores 0.794 balanced-test precision scores 0.192 in real operation.<n>We show the mismatch to be an evaluation problem in the costume of a training one at a fixed recall.<n>We show a prior-matched reporting method based on three figures: balanced-test, operational-prior, and real post-deployment.
• Score: 1.2599533416395765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Internal Waves Service screens the Sentinel-1 Wave-mode archive for internal solitary waves, routing detections to experts whose adjudication time is the resource the effort exists to conserve. Because attention is the cost of error, precision leads. Its classifier was trained and reported at a one-to-one class balance, fixed before the operational rate could be known. That rate has since emerged at roughly one scene in twenty, and a balanced-test score badly overstates the precision a validator meets. A model that scores 0.794 balanced-test precision scores 0.192 in real operation: the gap is a systematic artefact of reporting at the wrong prior, invisible to the metric most work quotes. We show the mismatch to be an evaluation problem in the costume of a training one at a fixed recall, prior correction and calibration cannot move precision, and answer it with a prior-matched reporting method based on three figures: balanced-test, operational-prior, and real post-deployment, whose contrast is the honest measure. A precision-first, leakage-controlled development cycle then improves the classifier lever by lever, each promoted only against a pre-registered margin; added capacity not clearing it, calibration inert, feature aggregation the one real lift, so the honest negatives are as much a result as the gain. Holding recall at a floor of 0.80 and certifying against a sealed, single-read lockbox, the promoted model reports 0.927 precision at the operational prior; an out-of-time check confirms discrimination transfers to unseen periods while a fixed operating point does not. Prior-matched reporting, begin balanced, then move to the prior as the stream reveals it, transfers to any operational Earth-observation service bootstrapping a rare-event detector under a prior it has yet to discover.

Related papers

• A Bayesian Correlated Equilibrium for Early Insider-Threat Detection [11.643788980306235]
  We model insider threat detection as a dynamic Bayesian game in which a platform coordinates a committee of strategic certifiers to sustain equilibrium among honest users.<n>On CERT r6.2 our mechanism achieves up to 28.3% pre-exfiltration detection with false positives below 1.6%, while both a transformer baseline and a streaming provenance approximation achieve near-zero pre-exfiltration detection.
  arXiv  Detail & Related papers  (2026-09-02T19:12:56Z)
• Excitation-Supervised Closed-Loop Self-Calibration and Target Seeking for an Unknown-Pose Range-Bearing Relay [0.0]
  This paper supplies the closed-loop layer that governs identifiability.<n>An excitation-supervised algorithm is used to supervise an excitation-reset controller.<n>A decay-rate sweep shows that supervision matters when a fixed schedule's decay outruns the unknown time-to-adequate-excitation.
  arXiv  Detail & Related papers  (2026-08-12T19:00:07Z)
• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Anytime-Valid Confirmation of Label-Shift Corrections [0.6599344783327052]
  In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation.<n>We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value.
  arXiv  Detail & Related papers  (2026-06-12T02:03:21Z)
• When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories [20.332123003765837]
  Early failure alerting requires deciding, while a dialog or agent trajectory is still unfolding, whether to flag it as likely to fail.<n>This is challenging because supervision is typically available only as a trajectory-level success/failure label while alerts must be raised from partial interactions.<n>We introduce a two-stage approach that learns from this sparse evidence structure and uses the resulting risk estimates for controllable early alerting.
  arXiv  Detail & Related papers  (2026-06-03T20:28:27Z)
• Anomaly-Informed Confidence Calibration for Vision-Based Safety Prediction [1.0981335059631014]
  We identify a critical perception-dynamics gap in existing anomaly signals.<n>We propose an Anomaly-Informed Online approach that fuses two anomaly scores extracted from a world model.
  arXiv  Detail & Related papers  (2026-05-20T12:43:09Z)
• The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation [67.26315138466312]
  We propose a calibration-aware OPD framework, CaOPD, that estimates empirical confidence from model rollouts.<n>Our findings highlight that capability distillation does not imply confidence, and that confidence should be treated as an essential objective in post-training.
  arXiv  Detail & Related papers  (2026-04-18T04:43:40Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Temperature Scaling Attack Disrupting Model Confidence in Federated Learning [4.863985119779627]
  We present the Temperature Scaling Attack (TSA), a training-time attack that degrades calibration while preserving accuracy.<n>Our results establish calibration integrity as a critical attack surface in federated learning.
  arXiv  Detail & Related papers  (2026-02-06T12:01:54Z)
• Predicting Overtakes in Trucks Using CAN Data [51.28632782308621]
  We investigate the detection of truck overtakes from CAN data. Our analysis covers up to 10 seconds before the overtaking event. We observe that the prediction scores of the overtake class tend to increase as we approach the overtake trigger.
  arXiv  Detail & Related papers  (2024-04-08T17:58:22Z)
• Deployment Prior Injection for Run-time Calibratable Object Detection [58.636806402337776]
  We introduce an additional graph input to the detector, where the graph represents the deployment context prior. During the test phase, any suitable deployment context prior can be injected into the detector via graph edits. Even if the deployment prior is unknown, the detector can self-calibrate using deployment prior approximated using its own predictions.
  arXiv  Detail & Related papers  (2024-02-27T04:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.