Abstract: Active learning is usually applied to acquire labels of informative data
points in supervised learning, to maximize accuracy in a sample-efficient way.
However, maximizing the accuracy is not the end goal when the results are used
for decision-making, for example in personalized medicine or economics. We
argue that when acquiring samples sequentially, separating learning and
decision-making is sub-optimal, and we introduce a novel active learning
strategy which takes the down-the-line decision problem into account.
Specifically, we introduce a novel active learning criterion which maximizes
the expected information gain on the posterior distribution of the optimal
decision. We compare our decision-making-aware active learning strategy to
existing alternatives on both simulated and real data, and show improved
performance in decision-making accuracy.