Foresighted Online Policy Optimization with Interference

• URL: http://arxiv.org/abs/2510.15273v1
• Date: Fri, 17 Oct 2025 03:32:06 GMT
• Title: Foresighted Online Policy Optimization with Interference
• Abstract summary: We introduce the foresighted online policy with interference (FRONT), that innovatively considers the long-term impact of the current decision on subsequent decisions and rewards.<n>FRONT attains sublinear regret under two distinct definitions, capturing both the immediate and consequential impacts of decisions.<n>We show that FRONT attains sublinear regret through extensive simulations and a real-world application to urban hotel profits.
• Score: 7.075948680027319
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contextual bandits, which leverage the baseline features of sequentially arriving individuals to optimize cumulative rewards while balancing exploration and exploitation, are critical for online decision-making. Existing approaches typically assume no interference, where each individual's action affects only their own reward. Yet, such an assumption can be violated in many practical scenarios, and the oversight of interference can lead to short-sighted policies that focus solely on maximizing the immediate outcomes for individuals, which further results in suboptimal decisions and potentially increased regret over time. To address this significant gap, we introduce the foresighted online policy with interference (FRONT) that innovatively considers the long-term impact of the current decision on subsequent decisions and rewards. The proposed FRONT method employs a sequence of exploratory and exploitative strategies to manage the intricacies of interference, ensuring robust parameter inference and regret minimization. Theoretically, we establish a tail bound for the online estimator and derive the asymptotic distribution of the parameters of interest under suitable conditions on the interference network. We further show that FRONT attains sublinear regret under two distinct definitions, capturing both the immediate and consequential impacts of decisions, and we establish these results with and without statistical inference. The effectiveness of FRONT is further demonstrated through extensive simulations and a real-world application to urban hotel profits.

Related papers

• Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning [63.24624171679711]
  Large language model (LLM)-based agents often make suboptimal tool-use decisions.<n>We propose TRUST, which incorporates uncertainty quantification into reward design as a repulsive force for maintaining uncertainty separation.
  arXiv  Detail & Related papers  (2026-06-05T07:08:34Z)
• OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents [74.20327254615854]
  Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks.<n>Existing inference-time adaptation methods for LLM agents mainly rely on prompting or retrieval.<n>We propose OLIVIA, an inference-time action adaptation framework for ReAct-style agents.
  arXiv  Detail & Related papers  (2026-05-11T19:28:20Z)
• Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time [52.230936493691985]
  We propose SITAlign, an inference framework that addresses the multifaceted nature of alignment by maximizing a primary objective while satisfying threshold-based constraints on secondary criteria.<n>We provide theoretical insights by deriving sub-optimality bounds of our satisficing based inference alignment approach.
  arXiv  Detail & Related papers  (2025-05-29T17:56:05Z)
• Uncertainty Quantification and Causal Considerations for Off-Policy Decision Making [4.514386953429771]
  Off-policy evaluation (OPE) seeks to assess the performance of a new policy using data collected under a different policy.<n>Existing OPE methodologies suffer from several limitations arising from statistical uncertainty as well as causal considerations.<n>We introduce the Marginal Ratio (MR) estimator, a novel OPE method that reduces variance by focusing on the marginal distribution of outcomes.<n>Next, we propose Conformal Off-Policy Prediction (COPP), a principled approach for uncertainty quantification in OPE.<n>Finally, we address causal unidentifiability in off-policy decision-making by developing novel bounds for sequential decision settings
  arXiv  Detail & Related papers  (2025-02-09T20:05:19Z)
• Online Experimental Design With Estimation-Regret Trade-off Under Network Interference [7.080131271060764]
  We introduce a unified interference-aware framework for online experimental design.<n>Compared to existing studies, we extend the definition of arm space by utilizing the statistical concept of exposure mapping.<n>We also propose an algorithmic implementation and discuss its generalization across different learning settings and network topology.
  arXiv  Detail & Related papers  (2024-12-04T21:45:35Z)
• Linear Contextual Bandits with Interference [14.835167982538053]
  We introduce a systematic framework to address interference in Linear CB (LinCB) We propose a series of algorithms that explicitly quantify the interference effect in the reward modeling process. The effectiveness of our approach is demonstrated through simulations and a synthetic data generated based on MovieLens data.
  arXiv  Detail & Related papers  (2024-09-24T02:51:00Z)
• Reduced-Rank Multi-objective Policy Learning and Optimization [57.978477569678844]
  In practice, causal researchers do not have a single outcome in mind a priori. In government-assisted social benefit programs, policymakers collect many outcomes to understand the multidimensional nature of poverty. We present a data-driven dimensionality-reduction methodology for multiple outcomes in the context of optimal policy learning.
  arXiv  Detail & Related papers  (2024-04-29T08:16:30Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• Online Learning under Budget and ROI Constraints via Weak Adaptivity [57.097119428915796]
  Existing primal-dual algorithms for constrained online learning problems rely on two fundamental assumptions. We show how such assumptions can be circumvented by endowing standard primal-dual templates with weakly adaptive regret minimizers. We prove the first best-of-both-worlds no-regret guarantees which hold in absence of the two aforementioned assumptions.
  arXiv  Detail & Related papers  (2023-02-02T16:30:33Z)
• Online Statistical Inference in Decision-Making with Matrix Context [5.2071564436846245]
  We propose an online procedure to conduct statistical inference with adaptively collected data.<n>Standard low-rank estimators are biased and cannot be obtained in a sequential manner.<n>Existing approaches in sequential decision-making algorithms fail to account for the low-rankness and are also biased.
  arXiv  Detail & Related papers  (2022-12-21T22:03:06Z)
• Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes [93.61202366677526]
  We study the offline reinforcement learning (RL) in the face of unmeasured confounders. We propose various policy learning methods with the finite-sample suboptimality guarantee of finding the optimal in-class policy.
  arXiv  Detail & Related papers  (2022-09-18T22:03:55Z)
• Reinforcement Learning with a Terminator [80.34572413850186]
  We learn the parameters of the TerMDP and leverage the structure of the estimation problem to provide state-wise confidence bounds. We use these to construct a provably-efficient algorithm, which accounts for termination, and bound its regret.
  arXiv  Detail & Related papers  (2022-05-30T18:40:28Z)
• Off-Policy Evaluation with Policy-Dependent Optimization Response [90.28758112893054]
  We develop a new framework for off-policy evaluation with a textitpolicy-dependent linear optimization response. We construct unbiased estimators for the policy-dependent estimand by a perturbation method. We provide a general algorithm for optimizing causal interventions.
  arXiv  Detail & Related papers  (2022-02-25T20:25:37Z)
• False Correlation Reduction for Offline Reinforcement Learning [115.11954432080749]
  We propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL)
  arXiv  Detail & Related papers  (2021-10-24T15:34:03Z)
• An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes [0.0]
  Exploitation vs Caution (EvC) is a paradigm that elegantly incorporates model uncertainty abiding by the Bayesian formalism. We validate EvC with state-of-the-art approaches in different discrete, yet simple, environments offering a fair variety of MDP classes. In the tested scenarios EvC manages to select robust policies and hence stands out as a useful tool for practitioners.
  arXiv  Detail & Related papers  (2021-05-27T20:12:20Z)
• Online Optimization and Ambiguity-based Learning of Distributionally Uncertain Dynamic Systems [1.6709415233613623]
  This paper proposes a novel approach to construct data-driven online solutions to optimization problems (P) subject to a class of distributionally uncertain dynamical systems. The introduced framework allows for the simultaneous learning of distributional system uncertainty via a parameterized, control-dependent ambiguity set. We also introduce an online version of Nesterov's accelerated-gradient algorithm, and analyze its performance to solve this class of problems via dissipativity theory.
  arXiv  Detail & Related papers  (2021-02-18T01:49:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.