Abstract: We propose a self-supervised framework to learn scene representations from
video that are automatically delineated into background, characters, and their
animations. Our method capitalizes on moving characters being equivariant with
respect to their transformation across frames and the background being constant
with respect to that same transformation. After training, we can manipulate
image encodings in real time to create unseen combinations of the delineated
components. As far as we know, we are the first method to perform unsupervised
extraction and synthesis of interpretable background, character, and animation.
We demonstrate results on three datasets: Moving MNIST with backgrounds, 2D
video game sprites, and Fashion Modeling.