Abstract: In stochastic contextual bandit (SCB) problems, an agent selects an action
based on certain observed context to maximize the cumulative reward over
iterations. Recently there have been a few studies using a deep neural network
(DNN) to predict the expected reward for an action, and the DNN is trained by a
stochastic gradient based method. However, convergence analysis has been
greatly ignored to examine whether and where these methods converge. In this
work, we formulate the SCB that uses a DNN reward function as a non-convex
stochastic optimization problem, and design a stage-wise stochastic gradient
descent algorithm to optimize the problem and determine the action policy. We
prove that with high probability, the action sequence chosen by this algorithm
converges to a greedy action policy respecting a local optimal reward function.
Extensive experiments have been performed to demonstrate the effectiveness and
efficiency of the proposed algorithm on multiple real-world datasets.