Simulating classification models to evaluate Predict-Then-Optimize methods

• URL: http://arxiv.org/abs/2509.02191v1
• Date: Tue, 02 Sep 2025 11:05:27 GMT
• Title: Simulating classification models to evaluate Predict-Then-Optimize methods
• Authors: Pieter Smet,
• Abstract summary: Uncertainty in optimization is often represented as parameters in the optimization model.<n>In Predict-Then-trivial approaches, predictions of a machine learning model are used as values for such parameters.<n>We show that the relationship between prediction error and how close solutions are to the actual is non-experimental.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Uncertainty in optimization is often represented as stochastic parameters in the optimization model. In Predict-Then-Optimize approaches, predictions of a machine learning model are used as values for such parameters, effectively transforming the stochastic optimization problem into a deterministic one. This two-stage framework is built on the assumption that more accurate predictions result in solutions that are closer to the actual optimal solution. However, providing evidence for this assumption in the context of complex, constrained optimization problems is challenging and often overlooked in the literature. Simulating predictions of machine learning models offers a way to (experimentally) analyze how prediction error impacts solution quality without the need to train real models. Complementing an algorithm from the literature for simulating binary classification, we introduce a new algorithm for simulating predictions of multiclass classifiers. We conduct a computational study to evaluate the performance of these algorithms, and show that classifier performance can be simulated with reasonable accuracy, although some variability is observed. Additionally, we apply these algorithms to assess the performance of a Predict-Then-Optimize algorithm for a machine scheduling problem. The experiments demonstrate that the relationship between prediction error and how close solutions are to the actual optimum is non-trivial, highlighting important considerations for the design and evaluation of decision-making systems based on machine learning predictions.

Related papers

• The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong [1.973144426163543]
  We highlight methodological issues that frequently occur in the community and should be addressed when evaluating algorithm selection approaches.<n>We show that non-informative features and meta-models can achieve high accuracy, which should not be the case with a well-designed evaluation framework.
  arXiv  Detail & Related papers  (2025-05-12T16:57:45Z)
• Algorithms with Calibrated Machine Learning Predictions [9.18151868060576]
  A field of algorithms with predictions incorporates machine learning advice in the design of online algorithms to improve real-world performance.<n>A central consideration is the extent to which predictions can be trusted -- while existing approaches often require users to specify an aggregate trust level, modern machine learning models can provide estimates of prediction-level uncertainty.<n>We propose calibration as a principled and practical tool to bridge this gap, demonstrating the benefits of calibrated advice through two case studies.
  arXiv  Detail & Related papers  (2025-02-05T03:41:18Z)
• Forecasting Outside the Box: Application-Driven Optimal Pointwise Forecasts for Stochastic Optimization [0.0]
  We show that, under mild assumptions, the problem can be solved with just one scenario, which we call an optimal scenario''<n>Finding an optimal scenario in general might be hard, but we show that the result can be particularly useful in the case of optimization problems with contextual information.
  arXiv  Detail & Related papers  (2024-11-05T21:54:50Z)
• Comparative study of regression vs pairwise models for surrogate-based heuristic optimisation [1.2535250082638645]
  This paper addresses the formulation of surrogate problems as both regression models that approximate fitness (surface surrogate models) and a novel way to connect classification models (pairwise surrogate models) The performance of the overall search, when using online machine learning-based surrogate models, depends not only on the accuracy of the predictive model but also on the kind of bias towards positive or negative cases.
  arXiv  Detail & Related papers  (2024-10-04T13:19:06Z)
• Learning Joint Models of Prediction and Optimization [56.04498536842065]
  Predict-Then-Then framework uses machine learning models to predict unknown parameters of an optimization problem from features before solving. This paper proposes an alternative method, in which optimal solutions are learned directly from the observable features by joint predictive models.
  arXiv  Detail & Related papers  (2024-09-07T19:52:14Z)
• End-to-End Learning for Fair Multiobjective Optimization Under Uncertainty [55.04219793298687]
  The Predict-Then-Forecast (PtO) paradigm in machine learning aims to maximize downstream decision quality. This paper extends the PtO methodology to optimization problems with nondifferentiable Ordered Weighted Averaging (OWA) objectives. It shows how optimization of OWA functions can be effectively integrated with parametric prediction for fair and robust optimization under uncertainty.
  arXiv  Detail & Related papers  (2024-02-12T16:33:35Z)
• Predict-Then-Optimize by Proxy: Learning Joint Models of Prediction and Optimization [59.386153202037086]
  Predict-Then- framework uses machine learning models to predict unknown parameters of an optimization problem from features before solving. This approach can be inefficient and requires handcrafted, problem-specific rules for backpropagation through the optimization step. This paper proposes an alternative method, in which optimal solutions are learned directly from the observable features by predictive models.
  arXiv  Detail & Related papers  (2023-11-22T01:32:06Z)
• You Shall Pass: Dealing with the Zero-Gradient Problem in Predict and Optimize for Convex Optimization [1.98873083514863]
  Predict and optimize is an increasingly popular decision-making paradigm that employs machine learning to predict unknown parameters of optimization problems. The key challenge to train such models is the computation of the Jacobian of the solution of the optimization problem with respect to its parameters. This paper demonstrates that the zero-gradient problem appears in the non-linear case as well -- the Jacobian can have a sizeable null space, thereby causing the training process to get stuck in suboptimal points.
  arXiv  Detail & Related papers  (2023-07-30T19:14:05Z)
• Integrated Optimization of Predictive and Prescriptive Tasks [0.0]
  We propose a new framework directly integrating predictive tasks under prescriptive tasks. We train the parameters of predictive algorithm within a prescription problem via bilevel optimization techniques.
  arXiv  Detail & Related papers  (2021-01-02T02:43:10Z)
• An AI-Assisted Design Method for Topology Optimization Without Pre-Optimized Training Data [68.8204255655161]
  An AI-assisted design method based on topology optimization is presented, which is able to obtain optimized designs in a direct way. Designs are provided by an artificial neural network, the predictor, on the basis of boundary conditions and degree of filling as input data.
  arXiv  Detail & Related papers  (2020-12-11T14:33:27Z)
• Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
  We show that common optimization methods lead to poor variational approximations if the problem is moderately large. Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
  arXiv  Detail & Related papers  (2020-09-01T19:12:11Z)
• Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [55.94450542785096]
  Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values. Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making. We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
  arXiv  Detail & Related papers  (2020-06-18T19:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.