OCRR: A Benchmark for Online Correction Recovery under Distribution Shift

• URL: http://arxiv.org/abs/2605.03153v1
• Date: Mon, 04 May 2026 20:51:07 GMT
• Title: OCRR: A Benchmark for Online Correction Recovery under Distribution Shift
• Abstract summary: OCRR (Online Correction Recovery Rate) is a benchmark that streams a corpus through a classification system and applies oracle or corrections to wrong predictions.<n>It beats the next-best published continual-learning baseline by 32.6 percentage points at equal memory budget.<n>It also beats LoRA-on-DeBERTa-v3-large by 84.6 percentage points on retention.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Static benchmarks measure a model frozen at training time. Real systems face distribution shift: new categories, paraphrased queries, drift: and must recover online via user corrections. No existing benchmark measures recovery speed under correction streams. We introduce OCRR (Online Correction Recovery Rate): a benchmark that streams a corpus through a classification system, applies oracle or stochastic corrections to wrong predictions, and reports two curves: novel-class accuracy and original-distribution accuracy versus correction count. We evaluate the substrate alongside nine baseline algorithms from five families plus seven bounded-storage variants of the substrate for the Pareto sweep, including standard online-learning baselines (river), continual-learning methods (EWC, A-GEM, LwF), retrieval/parametric hybrids (kNN-LM), parameter-efficient fine-tuning of a 1.5 B-parameter encoder (LoRA on DeBERTa-v3-large), and a hash-chained append-only substrate (Substrate). On Banking77 and CLINC150, under oracle and sparse correction policies, the substrate is the only system that simultaneously recovers novel-class accuracy (88.7 +/- 2.9 %) and retains original-distribution accuracy (95.4 +/- 0.8 %) beating the next-best published continual-learning baseline by 32.6 percentage points at equal memory budget, and beating LoRA-on-DeBERTa-v3-large by 84.6 percentage points on retention. We further find that classification accuracy remains stable at 99 % even as approximate-nearest-neighbour recall@5 degrades from 0.69 to 0.23 across 10 k to 10 M corpus scales, suggesting the substrate's margin-band majority vote is robust to retrieval imperfection in a way that pure top-k recall metrics do not predict. Code and data are available at https://github.com/adriangrassi/ocrr-benchmark.

Related papers

• SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning [70.71795707610961]
  method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
  arXiv  Detail & Related papers  (2026-08-29T21:34:47Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Leakage-Robust Evaluation and Data-Scale Sensitivity of Attention-Enhanced Multi-Task Learning for Joint Fault Diagnosis and Remaining Useful Life Estimation [0.0]
  Multi-task deep learning models can be affected by how sliding-window sequences are split into training and test sets.<n>We show that naive splitting can inflate classification accuracy from a genuine 20-60 percent to 99.9 percent.<n>We introduce a chunk-based, leakage-audited splitting protocol and evaluate all models using five seeds, one-way ANOVA, and Tukey HSD tests.
  arXiv  Detail & Related papers  (2026-07-17T20:33:36Z)
• GCE-MIL: Faithful and Recoverable Evidence for Multiple Instance Learning in Whole-Slide Imaging [4.001315295826918]
  Multiple instance learning (MIL) is the standard approach for whole-slide image (WSI) classification and survival prediction.<n>GCE-MIL is a backbone-agnostic wrapper implemented through three injection modes and three evidence components.
  arXiv  Detail & Related papers  (2026-05-17T13:56:30Z)
• Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuning [6.908972852063454]
  Uncertainty reliability can degrade earlier and more sharply than top-1 performance.<n>We study this empirically by measuring conformal coverage and calibration error on sequentially fine-tuned models.<n>We propose calibration replay, a lightweight post-hoc procedure that maintains a task-specific buffer and refits a task-specific conformal threshold.
  arXiv  Detail & Related papers  (2026-04-27T03:03:38Z)
• Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark [1.9537983097153042]
  Behaviour-Driven Development suites accumulate step-duplicate duplication with documented maintenance cost.<n>We release the largest cross-organisational BDD step corpus to date.
  arXiv  Detail & Related papers  (2026-04-22T11:44:05Z)
• Efficient Punctuation Restoration via Weighted Lookahead Scoring Method for Streaming ASR Systems [4.703098868120046]
  Punctuation restoration improves ASR (Automatic Speech Recognition) readability.<n>Streaming ASR requires online decisions with limited future context.<n>This paper proposes a non-autoregressive scoring method that preserves the input transcript and makes a decision at each word boundary.
  arXiv  Detail & Related papers  (2026-04-18T01:03:33Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning [43.24339861841546]
  This paper investigates fine-tuning of a decoder-style language model (phi/phi-2 with LoRA) on a CVE-linked dataset.<n>We evaluate eight continual learning strategies, including window-only and cumulative training, replay-based baselines and regularisation-based variants.<n>Hybrid-CASR reduces training time per window by about 17 percent compared to the baseline, whereas cumulative training delivers only a minor F1 increase (0.661) at a 15.9-fold computational cost.
  arXiv  Detail & Related papers  (2026-02-27T09:13:23Z)
• Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steering [3.7758197704962835]
  We introduce CORAL, a regularized-time steering method that captures correctness signals from model internal activations using weight-decay probes.<n>CORAL consistently improves accuracy by 10% and expected calibration error (ECE) by 50% on average.<n>Our results support the hypothesis that distributed information in model internals can be extracted using regularized probes when individual neurons are insufficient.
  arXiv  Detail & Related papers  (2026-02-05T18:55:56Z)
• Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods [0.0]
  We study model-agnostic post-hoc calibration methods to improve probabilistic predictions in supervised binary classification.<n>We benchmark 21 widely used classifiers, including linear models, SVMs, tree ensembles (CatBoost, XGBoost, LightGBM)<n>We find that commonly used calibration procedures, most notably Platt scaling and isotonic regression, can systematically degrade proper scoring performance.
  arXiv  Detail & Related papers  (2026-01-19T18:23:36Z)
• Adaptive Set-Mass Calibration with Conformal Prediction [60.47079469141295]
  We develop a new calibration procedure that starts with conformal prediction to obtain a set of labels that gives the desired coverage.<n>We then instantiate two simple post-hoc calibrators: a mass normalization and a temperature scaling-based rule, tuned to the conformal constraint.
  arXiv  Detail & Related papers  (2025-05-21T12:18:15Z)
• Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right? [135.71855998537347]
  We revisit the common practice of evaluating adaptation of Online Continual Learning (OCL) algorithms through the metric of online accuracy. We show that this metric is unreliable, as even vacuous blind classifiers can achieve unrealistically high online accuracy. Existing OCL algorithms can also achieve high online accuracy, but perform poorly in retaining useful information.
  arXiv  Detail & Related papers  (2023-05-16T08:29:33Z)
• Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking [57.42241521034744]
  We propose the concept of certified error control of candidate set pruning for relevance ranking. Our method successfully prunes the first-stage retrieved candidate sets to improve the second-stage reranking speed.
  arXiv  Detail & Related papers  (2022-05-19T16:00:13Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.