The Challenger: When Do New Data Sources Justify Switching Machine Learning Models?

• URL: http://arxiv.org/abs/2512.18390v1
• Date: Sat, 20 Dec 2025 15:03:40 GMT
• Title: The Challenger: When Do New Data Sources Justify Switching Machine Learning Models?
• Authors: Vassilis Digalakis, Christophe Pérignon, Sébastien Saurin, Flore Sentenac,
• Abstract summary: We study the problem of deciding whether an organization should replace a trained incumbent model with a challenger relying on newly available features.<n>We develop a unified economic and statistical framework that links learning-curve dynamics, data-acquisition and retraining costs, and discounting of future gains.
• Score: 2.7998963147546143
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the problem of deciding whether, and when an organization should replace a trained incumbent model with a challenger relying on newly available features. We develop a unified economic and statistical framework that links learning-curve dynamics, data-acquisition and retraining costs, and discounting of future gains. First, we characterize the optimal switching time in stylized settings and derive closed-form expressions that quantify how horizon length, learning-curve curvature, and cost differentials shape the optimal decision. Second, we propose three practical algorithms: a one-shot baseline, a greedy sequential method, and a look-ahead sequential method. Using a real-world credit-scoring dataset with gradually arriving alternative data, we show that (i) optimal switching times vary systematically with cost parameters and learning-curve behavior, and (ii) the look-ahead sequential method outperforms other methods and is able to approach in value an oracle with full foresight. Finally, we establish finite-sample guarantees, including conditions under which the sequential look-ahead method achieve sublinear regret relative to that oracle. Our results provide an operational blueprint for economically sound model transitions as new data sources become available.

Related papers

• LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimization [7.8639568562295965]
  We consider small-data, large-scale decision problems in which a firm must make many operational decisions simultaneously.<n>We propose a pretrain-then-finetune approach built on a designed Transformer model to address this challenge.
  arXiv  Detail & Related papers  (2026-02-03T16:08:33Z)
• Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning [35.359482937263145]
  We propose a novel Differential-Informed Sample Selection (DISSect) method, which accurately and efficiently discriminates the noisy correspondence for training acceleration.<n>Specifically, we rethink the impact of noisy correspondence on contrastive learning and propose that the differential between the predicted correlation of the current model and that of a historical model is more informative to characterize sample quality.
  arXiv  Detail & Related papers  (2025-07-17T11:13:44Z)
• Shifting AI Efficiency From Model-Centric to Data-Centric Compression [67.45087283924732]
  We argue that the focus of research for AI is shifting from model-centric compression to data-centric compression.<n>Data-centric compression improves AI efficiency by directly compressing the volume of data processed during model training or inference.<n>Our work aims to provide a novel perspective on AI efficiency, synthesize existing efforts, and catalyze innovation to address the challenges posed by ever-increasing context lengths.
  arXiv  Detail & Related papers  (2025-05-25T13:51:17Z)
• Online Decision-Focused Learning [74.3205104323777]
  Decision-focused learning (DFL) is an increasingly popular paradigm for training models whose predictive outputs are used in decision-making tasks.<n>In this paper, we regularize the objective function to make it different and investigate how to overcome nonoptimality function.<n>We also showcase the effectiveness of our algorithms on a knapsack experiment, where they outperform two standard benchmarks.
  arXiv  Detail & Related papers  (2025-05-19T10:40:30Z)
• OPO: Making Decision-Focused Data Acquisition Decisions [0.0]
  We propose a model for making data acquisition decisions for variables in contextual optimisation problems.<n>We solve the data acquisition problem with well-defined constraints by learning a surrogate linear objective function.<n>We ablate the problem with a number of training modalities and demonstrate that the differentiable optimisation approach outperforms random search strategies.
  arXiv  Detail & Related papers  (2025-04-21T12:41:35Z)
• Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework [10.317740844867913]
  We build a simulator based on 472 language model pre-training runs with varying data compositions from the SlimPajama dataset.<n>We observe that even simple acquisition functions can enable principled training decisions across training models from 20M to 1B kernels.
  arXiv  Detail & Related papers  (2025-03-26T22:19:47Z)
• When Does Visual Prompting Outperform Linear Probing for Vision-Language Models? A Likelihood Perspective [57.05315507519704]
  We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of visual prompting and linear probing. Our measure attains up to a 100-fold reduction in run time compared to full training, while achieving prediction accuracies up to 91%.
  arXiv  Detail & Related papers  (2024-09-03T12:03:45Z)
• Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment [65.15914284008973]
  We propose to leverage an Inverse Reinforcement Learning (IRL) technique to simultaneously build an reward model and a policy model. We show that the proposed algorithms converge to the stationary solutions of the IRL problem. Our results indicate that it is beneficial to leverage reward learning throughout the entire alignment process.
  arXiv  Detail & Related papers  (2024-05-28T07:11:05Z)
• Adaptive Retention & Correction: Test-Time Training for Continual Learning [114.5656325514408]
  A common problem in continual learning is the classification layer's bias towards the most recent task.<n>We name our approach Adaptive Retention & Correction (ARC)<n>ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
  arXiv  Detail & Related papers  (2024-05-23T08:43:09Z)
• Learning to Maximize Mutual Information for Dynamic Feature Selection [13.821253491768168]
  We consider the dynamic feature selection (DFS) problem where a model sequentially queries features based on the presently available information. We explore a simpler approach of greedily selecting features based on their conditional mutual information. The proposed method is shown to recover the greedy policy when trained to optimality, and it outperforms numerous existing feature selection methods in our experiments.
  arXiv  Detail & Related papers  (2023-01-02T08:31:56Z)
• Smoothed Online Learning is as Easy as Statistical Learning [77.00766067963195]
  We provide the first oracle-efficient, no-regret algorithms in this setting. We show that if a function class is learnable in the classical setting, then there is an oracle-efficient, no-regret algorithm for contextual bandits.
  arXiv  Detail & Related papers  (2022-02-09T19:22:34Z)
• A Scalable Inference Method For Large Dynamic Economic Systems [19.757929782329892]
  We present a novel Variational Bayesian Inference approach to incorporate a time-varying parameter auto-regressive model. Our model is applied to a large blockchain dataset containing prices, transactions of individual actors, analyzing transactional flows and price movements. We further improve the simple state-space modelling by introducing non-linearities in the forward model with the help of machine learning architectures.
  arXiv  Detail & Related papers  (2021-10-27T10:52:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.