Learning to Transform for Generalizable Instance-wise Invariance

• URL: http://arxiv.org/abs/2309.16672v3
• Date: Thu, 15 Feb 2024 19:00:07 GMT
• Title: Learning to Transform for Generalizable Instance-wise Invariance
• Authors: Utkarsh Singhal and Carlos Esteves and Ameesh Makadia and Stella X. Yu
• Abstract summary: Given any image, we use a normalizing flow to predict a distribution over transformations and average the predictions over them. This normalizing flow is trained end-to-end and can learn a much larger range of transformations than Augerino and InstaAug. When used as data augmentation, our method shows accuracy and robustness gains on CIFAR 10, CIFAR10-LT, and TinyImageNet.
• Score: 48.647925994707855
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer vision research has long aimed to build systems that are robust to spatial transformations found in natural data. Traditionally, this is done using data augmentation or hard-coding invariances into the architecture. However, too much or too little invariance can hurt, and the correct amount is unknown a priori and dependent on the instance. Ideally, the appropriate invariance would be learned from data and inferred at test-time. We treat invariance as a prediction problem. Given any image, we use a normalizing flow to predict a distribution over transformations and average the predictions over them. Since this distribution only depends on the instance, we can align instances before classifying them and generalize invariance across classes. The same distribution can also be used to adapt to out-of-distribution poses. This normalizing flow is trained end-to-end and can learn a much larger range of transformations than Augerino and InstaAug. When used as data augmentation, our method shows accuracy and robustness gains on CIFAR 10, CIFAR10-LT, and TinyImageNet.

Related papers

• Improving Equivariance in State-of-the-Art Supervised Depth and Normal Predictors [29.562054614079788]
  We find that state-of-the-art depth and normal predictors, despite having strong performances, surprisingly do not respect equivariance. To remedy this, we propose an equivariant regularization technique, consisting of an averaging procedure and a self-consistency loss. Our approach can be applied to both CNN and Transformer architectures, does not incur extra cost during testing, and notably improves the supervised and semi-supervised learning performance.
  arXiv  Detail & Related papers  (2023-09-28T17:51:05Z)
• In What Ways Are Deep Neural Networks Invariant and How Should We Measure This? [5.757836174655293]
  We introduce a family of invariance and equivariance metrics that allows us to quantify these properties in a way that disentangles them from other metrics such as loss or accuracy. We draw a range of conclusions about invariance and equivariance in deep learning models, ranging from whether initializing a model with pretrained weights has an effect on a trained model's invariance, to the extent to which invariance learned via training can generalize to out-of-distribution data.
  arXiv  Detail & Related papers  (2022-10-07T18:43:21Z)
• The Lie Derivative for Measuring Learned Equivariance [84.29366874540217]
  We study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. We find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities. For example, transformers can be more equivariant than convolutional neural networks after training.
  arXiv  Detail & Related papers  (2022-10-06T15:20:55Z)
• Learning Instance-Specific Augmentations by Capturing Local Invariances [62.70897571389785]
  InstaAug is a method for automatically learning input-specific augmentations from data. We empirically demonstrate that InstaAug learns meaningful input-dependent augmentations for a wide range of transformation classes.
  arXiv  Detail & Related papers  (2022-05-31T18:38:06Z)
• Do Deep Networks Transfer Invariances Across Classes? [123.84237389985236]
  We show how a generative approach for learning the nuisance transformations can help transfer invariances across classes. Our results provide one explanation for why classifiers generalize poorly on unbalanced and longtailed distributions.
  arXiv  Detail & Related papers  (2022-03-18T04:38:18Z)
• Regularising for invariance to data augmentation improves supervised learning [82.85692486314949]
  We show that using multiple augmentations per input can improve generalisation. We propose an explicit regulariser that encourages this invariance on the level of individual model predictions.
  arXiv  Detail & Related papers  (2022-03-07T11:25:45Z)
• Grounding inductive biases in natural images:invariance stems from variations in data [20.432568247732206]
  We study the factors of variation in a real dataset, ImageNet. We show standard augmentation relies on a precise combination of translation and scale. We find that the main factors of variation in ImageNet mostly relate to appearance.
  arXiv  Detail & Related papers  (2021-06-09T14:58:57Z)
• Rotating spiders and reflecting dogs: a class conditional approach to learning data augmentation distributions [0.0]
  We introduce a method by which we can learn class conditional distributions on augmentation transformations. We give a number of examples where our methods learn different non-meaningful transformations depending on class. Our method can be used as a tool to probe the symmetries intrinsic to a potentially complex dataset.
  arXiv  Detail & Related papers  (2021-06-07T23:36:24Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.