Group Orthogonalization Regularization For Vision Models Adaptation and Robustness

• URL: http://arxiv.org/abs/2306.10001v2
• Date: Sun, 18 Feb 2024 17:01:06 GMT
• Title: Group Orthogonalization Regularization For Vision Models Adaptation and Robustness
• Authors: Yoav Kurtz, Noga Bar, Raja Giryes
• Abstract summary: We propose a computationally efficient regularization technique that encourages orthonormality between groups of filters within the same layer. Our experiments show that when incorporated into recent adaptation methods for diffusion models and vision transformers (ViTs), this regularization improves performance on downstream tasks.
• Score: 31.43307762723943
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As neural networks become deeper, the redundancy within their parameters increases. This phenomenon has led to several methods that attempt to reduce the correlation between convolutional filters. We propose a computationally efficient regularization technique that encourages orthonormality between groups of filters within the same layer. Our experiments show that when incorporated into recent adaptation methods for diffusion models and vision transformers (ViTs), this regularization improves performance on downstream tasks. We further show improved robustness when group orthogonality is enforced during adversarial training. Our code is available at https://github.com/YoavKurtz/GOR.

Related papers

• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Adaptive aggregation of Monte Carlo augmented decomposed filters for efficient group-equivariant convolutional neural network [0.36122488107441414]
  Group-equivariant convolutional neural networks (G-CNN) heavily rely on parameter sharing to increase CNN's data efficiency and performance. We propose a non- parameter-sharing approach for group equivariant neural networks. The proposed methods adaptively aggregate a diverse range of filters by a weighted sum of decomposedally augmented filters.
  arXiv  Detail & Related papers  (2023-05-17T10:18:02Z)
• LD-GAN: Low-Dimensional Generative Adversarial Network for Spectral Image Generation with Variance Regularization [72.4394510913927]
  Deep learning methods are state-of-the-art for spectral image (SI) computational tasks. GANs enable diverse augmentation by learning and sampling from the data distribution. GAN-based SI generation is challenging since the high-dimensionality nature of this kind of data hinders the convergence of the GAN training yielding to suboptimal generation. We propose a statistical regularization to control the low-dimensional representation variance for the autoencoder training and to achieve high diversity of samples generated with the GAN.
  arXiv  Detail & Related papers  (2023-04-29T00:25:02Z)
• Orthogonal SVD Covariance Conditioning and Latent Disentanglement [65.67315418971688]
  Inserting an SVD meta-layer into neural networks is prone to make the covariance ill-conditioned. We propose Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) Experiments on visual recognition demonstrate that our methods can simultaneously improve covariance conditioning and generalization.
  arXiv  Detail & Related papers  (2022-12-11T20:31:31Z)
• Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality [65.67315418971688]
  Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) are proposed. Experiments on visual recognition demonstrate that our methods can simultaneously improve the covariance conditioning and generalization.
  arXiv  Detail & Related papers  (2022-07-05T15:39:29Z)
• Dense Unsupervised Learning for Video Segmentation [49.46930315961636]
  We present a novel approach to unsupervised learning for video object segmentation (VOS) Unlike previous work, our formulation allows to learn dense feature representations directly in a fully convolutional regime. Our approach exceeds the segmentation accuracy of previous work despite using significantly less training data and compute power.
  arXiv  Detail & Related papers  (2021-11-11T15:15:11Z)
• Distribution Mismatch Correction for Improved Robustness in Deep Neural Networks [86.42889611784855]
  normalization methods increase the vulnerability with respect to noise and input corruptions. We propose an unsupervised non-parametric distribution correction method that adapts the activation distribution of each layer. In our experiments, we empirically show that the proposed method effectively reduces the impact of intense image corruptions.
  arXiv  Detail & Related papers  (2021-10-05T11:36:25Z)
• Robust Learning with Frequency Domain Regularization [1.370633147306388]
  We introduce a new regularization method by constraining the frequency spectra of the filter of the model. We demonstrate the effectiveness of our regularization by (1) defensing to adversarial perturbations; (2) reducing the generalization gap in different architecture; and (3) improving the generalization ability in transfer learning scenario without fine-tune.
  arXiv  Detail & Related papers  (2020-07-07T07:29:20Z)
• DCT-Conv: Coding filters in convolutional networks with Discrete Cosine Transform [0.0]
  We analyze how switching off selected components of the spectra, thereby reducing the number of trained weights of the network, affects its performance. Experiments show that coding the filters with trained DCT parameters leads to improvement over traditional convolution.
  arXiv  Detail & Related papers  (2020-01-23T13:58:17Z)
• Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters [28.54654866641997]
  We introduce an implicit self-regularization into OR to push the mean and variance of filter angles in a network towards 90 and 0 simultaneously. Our regularization can be implemented as an architectural plug-in and integrated with an arbitrary network.
  arXiv  Detail & Related papers  (2020-01-05T17:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.