Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting

• URL: http://arxiv.org/abs/2609.13840v1
• Date: Sat, 12 Sep 2026 09:51:30 GMT
• Title: Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting
• Abstract summary: We benchmarked 38 forecasting methods spanning classical, intermittent-demand, machine-learning, deep-learning, and pretrained foundation models.<n>Forecast-accuracy rank and order-service rank are negatively correlated on the industrial panel, at -0.555, across methods evaluated on 20,330 real multi-item orders.<n>For intermittent demand, the lowest-error forecast need not deliver the highest service, which can be reduced without retraining.
• Score: 4.1751058176413105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters are selected based on line-level forecast accuracy. This disconnect matters when demand is intermittent and lumpy, histories are short, and lead times span months. We benchmarked 38 forecasting methods spanning classical, intermittent-demand, machine-learning, deep-learning, and pretrained foundation models. A common decision-aware protocol evaluates them on an industrial panel drawn from a live contract and two public datasets. Forecast-accuracy rank and order-service rank are negatively correlated on the industrial panel, at -0.555, across methods evaluated on 20,330 real multi-item orders. Service is more closely associated with the direction of cumulative forecast bias, including over-prediction during zero-demand periods, than with point accuracy. Examining bias in Chronos-2's instance normalization yields a training-free correction that lifts the per-material fill proxy from 77.5% to 92.0% (14.5 percentage points) at the 90% policy target and raises the complete-order fill rate from 54% to 63%. For reproducibility, we release RUF (Regenerate-Until-Fidelity), a method for generating fidelity-certified synthetic panels on which the findings reproduce. For intermittent demand, the lowest-error forecast need not deliver the highest service. Bias direction helps explain this gap, which can be reduced without retraining.

Related papers

• Hidden in Rounds: Predicting the Time Cost of 802.11 Contention in Federated Learning [2.313256061695585]
  Federated learning over IEEE802.11 shares the wireless channel among clients that send model updates.<n>We use ns-3 to measure the frame-delivery ratio and saturation throughput for different client densities and offered loads.<n>A separate FedAvg trainer uses the frame-delivery ratio as a first-order proxy for the update-admission probability and uses an equation to estimate communication time.
  arXiv  Detail & Related papers  (2026-09-11T14:30:26Z)
• Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledge [7.4253318607352155]
  We present Wnuan, a three-stage pipeline that constructs task-oriented supervision from documents.<n>On the 707-question WnuanBench, the primary 32B route raises acceptable-answer rate (AAR) from 52.76% before adaptation to 80.06% after SFT and 91.51% after RL.
  arXiv  Detail & Related papers  (2026-08-03T08:11:33Z)
• RAVEN: A Regime-Aware Variable-context Expert Network for Financial Time Series Forecasting [83.06074370551887]
  A fixed context window is mismatched to the time-varying optimal look-back of non-stationary price processes.<n>We propose a Mixture-of-Experts framework designed to adaptively determine the temporal context for each input sample.<n> Experiments on cumulative log-return prediction (HS300, S&P500) and fund sales forecasting demonstrate that RAVEN achieves SOTA performances, improves Pearson correlation by 9.2% on HS300 and 20.2% on S&P500, and reduces MSE by 18.2% on fund sales forecasting.
  arXiv  Detail & Related papers  (2026-06-23T02:11:04Z)
• ReacTOD: Bounded Neuro-Symbolic Agentic NLU for Zero-Shot Dialogue State Tracking [2.2573512203799626]
  We propose ReacTOD, a bounded neuro-symbolic architecture that reformulates NLU as discrete tool calls within a self-correcting ReAct loop governed by deterministic validation.<n>A bounded ReAct loop enables iterative self-correction, improving accuracy by up to 9.3 percentage points over single-pass inference on MultiWOZ.
  arXiv  Detail & Related papers  (2026-05-18T20:06:04Z)
• Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents? [0.7199239000118145]
  We introduce a forced-injection framework that provides ground-truth clarifications at controlled points in the agent's trajectory.<n>We find that the value of clarification depends sharply on what information is missing.<n>A complementary study of 300 unscripted sessions reveals that no current frontier model asks within the empirically optimal window.
  arXiv  Detail & Related papers  (2026-05-08T16:08:03Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• F-FOMAML: GNN-Enhanced Meta-Learning for Peak Period Demand Forecasting with Proxy Data [65.6499834212641]
  We formulate the demand prediction as a meta-learning problem and develop the Feature-based First-Order Model-Agnostic Meta-Learning (F-FOMAML) algorithm. By considering domain similarities through task-specific metadata, our model improved generalization, where the excess risk decreases as the number of training tasks increases. Compared to existing state-of-the-art models, our method demonstrates a notable improvement in demand prediction accuracy, reducing the Mean Absolute Error by 26.24% on an internal vending machine dataset and by 1.04% on the publicly accessible JD.com dataset.
  arXiv  Detail & Related papers  (2024-06-23T21:28:50Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• Making forecasting self-learning and adaptive -- Pilot forecasting rack [0.0]
  This paper presents our findings based on a proactive pilot exercise to explore ways to help retailers to improve forecast accuracy for such product categories. We evaluated opportunities for algorithmic interventions to improve forecast accuracy based on a sample product category, Knitwear. Our outcomes show an increase in the accuracy of demand forecast for Knitwear product category by 20%, taking the overall accuracy to 80%.
  arXiv  Detail & Related papers  (2023-06-12T03:26:11Z)
• Debiased Fine-Tuning for Vision-language Models by Prompt Regularization [56.48290708901531]
  We present a new paradigm for fine-tuning large-scale vision pre-trained models on downstream task, dubbed Prompt Regularization (ProReg)<n>ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning.<n>We show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-01-29T11:53:55Z)
• Approaching sales forecasting using recurrent neural networks and transformers [57.43518732385863]
  We develop three alternatives to tackle the problem of forecasting the customer sales at day/store/item level using deep learning techniques. Our empirical results show how good performance can be achieved by using a simple sequence to sequence architecture with minimal data preprocessing effort. The proposed solution achieves a RMSLE of around 0.54, which is competitive with other more specific solutions to the problem proposed in the Kaggle competition.
  arXiv  Detail & Related papers  (2022-04-16T12:03:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.