The cost of ensembling: is it always worth combining?

• URL: http://arxiv.org/abs/2506.04677v2
• Date: Wed, 09 Jul 2025 12:32:09 GMT
• Title: The cost of ensembling: is it always worth combining?
• Abstract summary: Trade-off between forecast accuracy and computational cost is emerging as an extremely relevant topic.<n>We evaluated ten base models and eight ensemble configurations across two large-scale retail datasets.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Given the continuous increase in dataset sizes and the complexity of forecasting models, the trade-off between forecast accuracy and computational cost is emerging as an extremely relevant topic, especially in the context of ensemble learning for time series forecasting. To asses it, we evaluated ten base models and eight ensemble configurations across two large-scale retail datasets (M5 and VN1), considering both point and probabilistic accuracy under varying retraining frequencies. We showed that ensembles consistently improve forecasting performance, particularly in probabilistic settings. However, these gains come at a substantial computational cost, especially for larger, accuracy-driven ensembles. We found that reducing retraining frequency significantly lowers costs, with minimal impact on accuracy, particularly for point forecasts. Moreover, efficiency-driven ensembles offer a strong balance, achieving competitive accuracy with considerably lower costs compared to accuracy-optimized combinations. Most importantly, small ensembles of two or three models are often sufficient to achieve near-optimal results. These findings provide practical guidelines for deploying scalable and cost-efficient forecasting systems, supporting the broader goals of sustainable AI in forecasting. Overall, this work shows that careful ensemble design and retraining strategy selection can yield accurate, robust, and cost-effective forecasts suitable for real-world applications.

Related papers

• Diversity is the Strength of the AI Crowd [4.101606888483043]
  Given a fixed number of samples, which off-the-shelf model forecasts should be combined to maximize accuracy?<n>We find that individual accuracy is not enough: many frontier LLMs make highly correlated predictions, limiting the value of additional forecasts from the same or similar models.<n>The strongest ensembles combine accurate but diverse forecasters, with models such as modelGrok 4 contributing disproportionately because their predictions are less correlated with other frontier LLMs.
  arXiv  Detail & Related papers  (2026-06-29T00:09:49Z)
• A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs [64.8510381475827]
  Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently.<n>SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized.<n>We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set.
  arXiv  Detail & Related papers  (2026-02-23T15:11:16Z)
• Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking [51.56484100374058]
  We evaluate whether a fully automatic, purely feedback-driven ESN can serve as a competitive alternative to widely used statistical forecasting methods.<n>Forecast accuracy is measured using MASE and sMAPE and benchmarked against simple benchmarks like drift and seasonal naive and statistical models.
  arXiv  Detail & Related papers  (2026-02-03T16:01:22Z)
• Demystifying Data-Driven Probabilistic Medium-Range Weather Forecasting [63.8116386935854]
  We demonstrate that state-of-the-art probabilistic skill requires neither intricate architectural constraints nor specialized trainings.<n>We introduce a scalable framework for learning multi-scale atmospheric dynamics by combining a directly downsampled latent space with a history-conditioned local projector.<n>We find that our framework design is robust to the choice of probabilistic estimators, seamlessly supporting interpolants, diffusion models, and CRPS-based ensemble training.
  arXiv  Detail & Related papers  (2026-01-26T03:52:16Z)
• Comparative Analysis of Modern Machine Learning Models for Retail Sales Forecasting [0.0]
  When forecasts underestimate the level of sales, firms experience lost sales, shortages, and impact on the reputation of the retailer in their relevant market.<n>This study provides an exhaustive assessment of the forecasting models applied to a high-resolution brick-and-mortar retail dataset.
  arXiv  Detail & Related papers  (2025-06-06T10:08:17Z)
• Do global forecasting models require frequent retraining? [0.0]
  We show that less frequent retraining strategies maintain the forecast accuracy while reducing the computational costs.<n>We also found that machine learning models are a marginally better choice to reduce the costs of forecasting when coupled with less frequent model retraining strategies.
  arXiv  Detail & Related papers  (2025-05-01T07:00:29Z)
• Forecasting Company Fundamentals [19.363166648866066]
  We evaluate 24 deterministic and probabilistic company fundamentals forecasting models on real company data.<n>We find that deep learning models provide superior forecasting performance to classical models.<n>We show how these high-quality forecasts can benefit automated stock allocation.
  arXiv  Detail & Related papers  (2024-10-21T14:21:43Z)
• Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity [89.81738321188391]
  This study investigates the relationship between task complexity and optimal sparsity in SMoE models.<n>We show that the optimal sparsity lies between minimal activation (1-2 experts) and full activation, with the exact number scaling proportionally to task complexity.
  arXiv  Detail & Related papers  (2024-10-17T18:40:48Z)
• Learning Graph Structures and Uncertainty for Accurate and Calibrated Time-series Forecasting [65.40983982856056]
  We introduce STOIC, that leverages correlations between time-series to learn underlying structure between time-series and to provide well-calibrated and accurate forecasts. Over a wide-range of benchmark datasets STOIC provides 16% more accurate and better-calibrated forecasts.
  arXiv  Detail & Related papers  (2024-07-02T20:14:32Z)
• When Rigidity Hurts: Soft Consistency Regularization for Probabilistic Hierarchical Time Series Forecasting [69.30930115236228]
  Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting. Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions. We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
  arXiv  Detail & Related papers  (2023-10-17T20:30:16Z)
• Ensemble Modeling for Time Series Forecasting: an Adaptive Robust Optimization Approach [3.7565501074323224]
  This paper proposes a new methodology for building robust ensembles of time series forecasting models. We demonstrate the effectiveness of our method through a series of synthetic experiments and real-world applications.
  arXiv  Detail & Related papers  (2023-04-09T20:30:10Z)
• When Rigidity Hurts: Soft Consistency Regularization for Probabilistic Hierarchical Time Series Forecasting [69.30930115236228]
  Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting. Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions. We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
  arXiv  Detail & Related papers  (2022-06-16T06:13:53Z)
• Functional Ensemble Distillation [18.34081591772928]
  We investigate how to best distill an ensemble's predictions using an efficient model. We find that learning the distilled model via a simple augmentation scheme in the form of mixup augmentation significantly boosts the performance.
  arXiv  Detail & Related papers  (2022-06-05T14:07:17Z)
• Sparse MoEs meet Efficient Ensembles [49.313497379189315]
  We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs) We present Efficient Ensemble of Experts (E$3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble.
  arXiv  Detail & Related papers  (2021-10-07T11:58:35Z)
• Generalizable Mixed-Precision Quantization via Attribution Rank Preservation [90.26603048354575]
  We propose a generalizable mixed-precision quantization (GMPQ) method for efficient inference. Our method obtains competitive accuracy-complexity trade-off compared with the state-of-the-art mixed-precision networks.
  arXiv  Detail & Related papers  (2021-08-05T16:41:57Z)
• Deep Learning for Post-Processing Ensemble Weather Forecasts [14.622977874836298]
  We propose a mixed model that uses only a subset of the original weather trajectories combined with a post-processing step using deep neural networks. We show that our post-processing can use fewer trajectories to achieve comparable results to the full ensemble.
  arXiv  Detail & Related papers  (2020-05-18T14:23:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.