On the Importance of Asymmetry for Siamese Representation Learning
- URL: http://arxiv.org/abs/2204.00613v1
- Date: Fri, 1 Apr 2022 17:57:24 GMT
- Title: On the Importance of Asymmetry for Siamese Representation Learning
- Authors: Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, Xinlei Chen
- Abstract summary: Siamese networks are conceptually symmetric with two parallel encoders.
We study the importance of asymmetry by explicitly distinguishing the two encoders within the network.
We find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones.
- Score: 53.86929387179092
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Many recent self-supervised frameworks for visual representation learning are
based on certain forms of Siamese networks. Such networks are conceptually
symmetric with two parallel encoders, but often practically asymmetric as
numerous mechanisms are devised to break the symmetry. In this work, we conduct
a formal study on the importance of asymmetry by explicitly distinguishing the
two encoders within the network -- one produces source encodings and the other
targets. Our key insight is keeping a relatively lower variance in target than
source generally benefits learning. This is empirically justified by our
results from five case studies covering different variance-oriented designs,
and is aligned with our preliminary theoretical analysis on the baseline.
Moreover, we find the improvements from asymmetric designs generalize well to
longer training schedules, multiple other frameworks and newer backbones.
Finally, the combined effect of several asymmetric designs achieves a
state-of-the-art accuracy on ImageNet linear probing and competitive results on
downstream transfer. We hope our exploration will inspire more research in
exploiting asymmetry for Siamese representation learning.
Related papers
Err
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.