Abstract: We present a statistical testing framework to detect if a given machine
learning classifier fails to satisfy a wide range of group fairness notions.
The proposed test is a flexible, interpretable, and statistically rigorous tool
for auditing whether exhibited biases are intrinsic to the algorithm or due to
the randomness in the data. The statistical challenges, which may arise from
multiple impact criteria that define group fairness and which are discontinuous
on model parameters, are conveniently tackled by projecting the empirical
measure onto the set of group-fair probability models using optimal transport.
This statistic is efficiently computed using linear programming and its
asymptotic distribution is explicitly obtained. The proposed framework can also
be used to test for testing composite fairness hypotheses and fairness with
multiple sensitive attributes. The optimal transport testing formulation
improves interpretability by characterizing the minimal covariate perturbations
that eliminate the bias observed in the audit.