Learning to Spend: Model Predictive Control for Budgeting under Non-Stationary Returns

• URL: http://arxiv.org/abs/2604.27186v1
• Date: Wed, 29 Apr 2026 20:39:25 GMT
• Title: Learning to Spend: Model Predictive Control for Budgeting under Non-Stationary Returns
• Abstract summary: We study finite-horizon budget allocation as a closed-loop economic control problem.<n>We evaluate receding-horizon Model Predictive Control (MPC) relative to reactive budgeting policies.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study finite-horizon budget allocation as a closed-loop economic control problem and evaluate receding-horizon Model Predictive Control (MPC) relative to reactive budgeting policies. Budgets are allocated periodically under execution noise and operational constraints, while return efficiency may evolve over time. Using a controlled simulation framework motivated by digital marketing, we compare reactive pacing to MPC across environments with increasing degrees of non-stationarity. Our results show that non-stationarity alone does not justify predictive control. When return dynamics are stationary or evolve through unpredictable stochastic drift, MPC offers no systematic advantage over reactive baselines. By contrast, when return efficiency exhibits predictable structure over the planning horizon, that is captured through an underlying model, MPC consistently outperforms reactive budgeting by exploiting intertemporal trade-offs.

Related papers

• Learning-enabled Acceleration of Scenario-based Model Predictive Control [1.838884769821944]
  This paper presents a learning-accelerated Alternating Direction Method of Multipliers (ADMM) algorithm for efficiently solving scenario-based model predictive control (SBMPC) problems.<n>We reformulate the SBMPC problems into consensus forms that can be decomposed via ADMM, separating the scenario-dependent dynamics from non-anticipativity constraints.<n>Building on this decomposition, we utilize existing learning-to-optimize schemes, which leverages Moreau envelope learning of the cost function to accelerate the primal update in ADMM.
  arXiv  Detail & Related papers  (2026-07-14T13:48:48Z)
• Evaluating AI Investment Strategies [0.0]
  We study the problem of auditing a black-box algorithmic decision-maker from observable inputs and outputs alone.<n>Under precisely characterized conditions, the cumulative emphregret of a dynamic policy equals the sum of per-period covariances between the cost vector and the policy's decision.<n>We prove the identity holds exactly under i.i.d. costs and mean-unbiased Markov policies, derive closed-form bias corrections for non-stationary and time-varying cases, and establish the discounted-horizon analog.
  arXiv  Detail & Related papers  (2026-06-07T19:16:50Z)
• Ratio-Variance Regularized Policy Optimization [64.95520246570446]
  We show that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints.<n>We introduce $bf R2bf VPO$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework.
  arXiv  Detail & Related papers  (2026-05-26T09:53:42Z)
• Optimal sequential decision-making for error propagation mitigation in digital twins [0.0]
  We develop a Markov Decision Process (MDP) in which the inferred regimes serve as states, corrective interventions serve as actions, and a scalar reward takes into consideration the cost-benefit tradeoff between system fidelity and maintenance expense.<n>We then extend the formulation to a Partially Observable MDP (POMDP) that accounts for the imperfect nature of regime classification by maintaining a belief updated via Bayesian filtering.<n>Both formulations are solved via dynamic programming and simulation. We then benchmark two model-free reinforcement learning algorithms, Q-learning and REINFORCE, to assess whether effective policies can be learned without explicit model knowledge.
  arXiv  Detail & Related papers  (2026-04-24T02:36:23Z)
• $V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts [81.48669089692189]
  Generalist Value Models (such as $V_0.5$) achieve pre-trained value estimation by explicitly encoding model capabilities in-context.<n>We propose $V_0.5$, which adaptively fuses the baseline predicted by such value model with the empirical mean derived from sparse rollouts.<n>$V_0.5$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
  arXiv  Detail & Related papers  (2026-03-11T14:57:41Z)
• Conformal Thinking: Risk Control for Reasoning on a Compute Budget [60.65072883773352]
  Reasoning Large Language Models (LLMs) enable test-time scaling, with dataset-level accuracy improving as the token budget increases.<n>We re-frame the budget setting problem as risk control, limiting the error rate while minimizing compute.<n>Our framework introduces an upper threshold that stops reasoning when the model is confident and a novel lower threshold that preemptively stops unsolvable instances.
  arXiv  Detail & Related papers  (2026-02-03T18:17:22Z)
• $V_0$: A Generalist Value Model for Any Policy at State Zero [80.7505802128501]
  Policy methods rely on a baseline to measure the relative advantage of an action.<n>This baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself.<n>We propose a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts.
  arXiv  Detail & Related papers  (2026-02-03T14:35:23Z)
• Anytime Incremental $ρ$POMDP Planning in Continuous Spaces [5.767643556541711]
  We present an anytime solver that dynamically refines belief representations, with formal guarantees of improvement over time.<n>We demonstrate its effectiveness for common entropy estimators, reducing computational cost by orders of magnitude.<n> Experimental results show that $rho$POMCPOW outperforms state-of-the-art solvers in both efficiency and solution quality.
  arXiv  Detail & Related papers  (2025-02-04T18:19:40Z)
• Improved Monte Carlo Planning via Causal Disentanglement for Structurally-Decomposed Markov Decision Processes [0.9768138268100163]
  We introduce the Structurally Decomposed MDP (SD-MDP), which leverages causal disentanglement to partition an MDP's temporal causal graph into independent components.<n>We demonstrate superior policy performance over benchmarks across various logistics and finance domains.
  arXiv  Detail & Related papers  (2024-06-23T16:22:40Z)
• End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control [45.84205238554709]
  We present a method for reinforcement learning of Koopman surrogate models for optimal performance as part of (e)NMPC. We show that the end-to-end trained models outperform those trained using system identification in (e)NMPC.
  arXiv  Detail & Related papers  (2023-08-03T10:21:53Z)
• Provably Efficient Model-Free Algorithms for Non-stationary CMDPs [10.930095238723327]
  We study model-free reinforcement learning algorithms in episodic non-stationary constrained Markov Decision Processes. In the non-stationary environment, reward, utility functions, and transition kernels can vary arbitrarily over time as long as the cumulative variations do not exceed certain variation budgets. We propose the first model-free, simulator-free RL algorithms with sublinear regret and zero constraint violation for non-stationary CMDPs.
  arXiv  Detail & Related papers  (2023-03-10T06:33:38Z)
• Blending MPC & Value Function Approximation for Efficient Reinforcement Learning [42.429730406277315]
  Model-Predictive Control (MPC) is a powerful tool for controlling complex, real-world systems. We present a framework for improving on MPC with model-free reinforcement learning (RL) We show that our approach can obtain performance comparable with MPC with access to true dynamics.
  arXiv  Detail & Related papers  (2020-12-10T11:32:01Z)
• Stein Variational Model Predictive Control [130.60527864489168]
  Decision making under uncertainty is critical to real-world, autonomous systems. Model Predictive Control (MPC) methods have demonstrated favorable performance in practice, but remain limited when dealing with complex distributions. We show that this framework leads to successful planning in challenging, non optimal control problems.
  arXiv  Detail & Related papers  (2020-11-15T22:36:59Z)
• Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting [91.43582419264763]
  We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty. LqgOpt efficiently explores the system dynamics, estimates the model parameters up to their confidence interval, and deploys the controller of the most optimistic model.
  arXiv  Detail & Related papers  (2020-03-12T19:56:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.