Abstract: Despite the success of machine learning applications in science, industry,
and society in general, many approaches are known to be non-robust, often
relying on spurious correlations to make predictions. Spuriousness occurs when
some features correlate with labels but are not causal; relying on such
features prevents models from generalizing to unseen environments where such
correlations break. In this work, we focus on image classification and propose
two data generation processes to reduce spuriousness. Given human annotations
of the subset of the features responsible (causal) for the labels (e.g.
bounding boxes), we modify this causal set to generate a surrogate image that
no longer has the same label (i.e. a counterfactual image). We also alter
non-causal features to generate images still recognized as the original labels,
which helps to learn a model invariant to these features. In several
challenging datasets, our data generations outperform state-of-the-art methods
in accuracy when spurious correlations break, and increase the saliency focus
on causal features providing better explanations.