Abstract: Many machine learning applications can benefit from simulated data for
systematic validation - in particular if real-life data is difficult to obtain
or annotate. However, since simulations are prone to domain shift w.r.t.
real-life data, it is crucial to verify the transferability of the obtained
results. We propose a novel framework consisting of a generative label-to-image
synthesis model together with different transferability measures to inspect to
what extent we can transfer testing results of semantic segmentation models
from synthetic data to equivalent real-life data. With slight modifications,
our approach is extendable to, e.g., general multi-class classification tasks.
Grounded on the transferability analysis, our approach additionally allows for
extensive testing by incorporating controlled simulations. We validate our
approach empirically on a semantic segmentation task on driving scenes.
Transferability is tested using correlation analysis of IoU and a learned
discriminator. Although the latter can distinguish between real-life and
synthetic tests, in the former we observe surprisingly strong correlations of
0.7 for both cars and pedestrians.