Abstract: In a sequential decision-making problem, off-policy evaluation estimates the
expected cumulative reward of a target policy using logged trajectory data
generated from a different behavior policy, without execution of the target
policy. Reinforcement learning in high-stake environments, such as healthcare
and education, is often limited to off-policy settings due to safety or ethical
concerns, or inability of exploration. Hence it is imperative to quantify the
uncertainty of the off-policy estimate before deployment of the target policy.
In this paper, we propose a novel framework that provides robust and optimistic
cumulative reward estimates using one or multiple logged trajectories data.
Leveraging methodologies from distributionally robust optimization, we show
that with proper selection of the size of the distributional uncertainty set,
these estimates serve as confidence bounds with non-asymptotic and asymptotic
guarantees under stochastic or adversarial environments. Our results are also
generalized to batch reinforcement learning and are supported by empirical