Abstract: Finding the minimal structural assumptions that empower sample-efficient
learning is one of the most important research directions in Reinforcement
Learning (RL). This paper advances our understanding of this fundamental
question by introducing a new complexity measure -- Bellman Eluder (BE)
dimension. We show that the family of RL problems of low BE dimension is
remarkably rich, which subsumes a vast majority of existing tractable RL
problems including but not limited to tabular MDPs, linear MDPs, reactive
POMDPs, low Bellman rank problems as well as low Eluder dimension problems.
This paper further designs a new optimization-based algorithm -- GOLF, and
reanalyzes a hypothesis elimination-based algorithm -- OLIVE (proposed in Jiang
et al. (2017)). We prove that both algorithms learn the near-optimal policies
of low BE dimension problems in a number of samples that is polynomial in all
relevant parameters, but independent of the size of state-action space. Our
regret and sample complexity results match or improve the best existing results
for several well-known subclasses of low BE dimension problems.