Fitted Occupancy-Ratio Evaluation without Bellman Completeness

• URL: http://arxiv.org/abs/2607.05375v1
• Date: Mon, 06 Jul 2026 17:53:32 GMT
• Title: Fitted Occupancy-Ratio Evaluation without Bellman Completeness
• Abstract summary: Occupancy ratios correct distribution shift in offline reinforcement learning.<n>We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio.<n>For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error.
• Score: 40.322273308230606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.

Related papers

• Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning [40.322273308230606]
  We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method.<n>Method corrects estimate's scale and shape by applying fitted occupancy-ratio evaluation (FORE) over a one-dimensional class of nondecreasing transformations.<n>We derive a calibration-refinement bound showing that any fitted ratio with small calibration error performs nearly as well as the best post-processing.
  arXiv  Detail & Related papers  (2026-08-25T17:44:02Z)
• Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL [7.803284875836737]
  bootstrapping can amplify estimation errors in offline reinforcement learning.<n>We propose Convex Hull Neighborhood Smooth Dual Generalization, which expresses the Bellman backup as an in-sample value target plus a CHN-local correction.<n> Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation.
  arXiv  Detail & Related papers  (2026-08-04T04:30:02Z)
• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• Distributional Soft Bellman Operator under the Cramér Geometry [19.762345326138536]
  This paper studies the Cramér-geometric Bellman fixed point associated with the policy-evaluation step of DSPI.<n>It provides a reference point for studying approximate critics, evaluation error, and critic-loss design in DSPI-style algorithms.
  arXiv  Detail & Related papers  (2026-07-20T12:44:30Z)
• Quotient-Categorical Representations for Bellman-Compatible Average-Reward Distributional Reinforcement Learning [8.988097534171995]
  We introduce a quotient-space formulation in which state-indexed bias laws are identified up to a common translation.<n>We show that synchronous exact updates are gain-independent at the quotient-law level.
  arXiv  Detail & Related papers  (2026-05-11T22:17:09Z)
• Fitted Q Evaluation Without Bellman Completeness via Stationary Weighting [40.322273308230606]
  We show the need for this assumption stems from a fundamental norm mismatch.<n>We propose a simple fix: reweight each regression step using an estimate of the stationary density ratio.<n>This enables strong evaluation guarantees in the absence of realizability or Bellman completeness.
  arXiv  Detail & Related papers  (2025-12-29T19:04:40Z)
• Stability and Generalization for Bellman Residuals [8.250374560598493]
  Bellman residual minimization (BRM) has emerged as an attractive remedy for offline reinforcement learning.<n>This paper introduces a single Lyapunov potential that couples SGDA runs on neighbouring datasets and yields an O (1/n) on-average argument-stability bound.<n>The results hold for standard neural-network parameterizations and minibatch SGD.
  arXiv  Detail & Related papers  (2025-08-26T07:15:36Z)
• To bootstrap or to rollout? An optimal and adaptive interpolation [4.755935781862859]
  We introduce a class of Bellman operators that interpolate between bootstrapping and rollout methods.<n>Our estimator combines the strengths of the bootstrapping-based temporal difference (TD) estimator and the rollout-based Monte Carlo (MC) methods.
  arXiv  Detail & Related papers  (2024-11-14T19:00:00Z)
• Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
  Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs.<n>We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint.<n>We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
  arXiv  Detail & Related papers  (2024-06-05T13:36:38Z)
• Optimal variance-reduced stochastic approximation in Banach spaces [114.8734960258221]
  We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. We establish non-asymptotic bounds for both the operator defect and the estimation error.
  arXiv  Detail & Related papers  (2022-01-21T02:46:57Z)
• Bayesian Bellman Operators [55.959376449737405]
  We introduce a novel perspective on Bayesian reinforcement learning (RL) Our framework is motivated by the insight that when bootstrapping is introduced, model-free approaches actually infer a posterior over Bellman operators, not value functions.
  arXiv  Detail & Related papers  (2021-06-09T12:20:46Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.