Abstract: It is often remarked that neural networks fail to increase their uncertainty
when predicting on data far from the training distribution. Yet naively using
softmax confidence as a proxy for uncertainty achieves modest success in tasks
exclusively testing for this, e.g., out-of-distribution (OOD) detection. This
paper investigates this contradiction, identifying two implicit biases that do
encourage softmax confidence to correlate with epistemic uncertainty: 1)
Approximately optimal decision boundary structure, and 2) Filtering effects of
deep networks. It describes why low-dimensional intuitions about softmax
confidence are misleading. Diagnostic experiments quantify reasons softmax
confidence can fail, finding that extrapolations are less to blame than overlap
between training and OOD data in final-layer representations.
Pre-trained/fine-tuned networks reduce this overlap.