Role of Orthogonality Constraints in Improving Properties of Deep
Networks for Image Classification
- URL: http://arxiv.org/abs/2009.10762v1
- Date: Tue, 22 Sep 2020 18:46:05 GMT
- Title: Role of Orthogonality Constraints in Improving Properties of Deep
Networks for Image Classification
- Authors: Hongjun Choi, Anirudh Som, Pavan Turaga
- Abstract summary: We propose an Orthogonal Sphere (OS) regularizer that emerges from physics-based latent-representations under simplifying assumptions.
Under further simplifying assumptions, the OS constraint can be written in closed-form as a simple orthonormality term and be used along with the cross-entropy loss function.
We demonstrate the effectiveness of the proposed OS regularization by providing quantitative and qualitative results on four benchmark datasets.
- Score: 8.756814963313804
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Standard deep learning models that employ the categorical cross-entropy loss
are known to perform well at image classification tasks. However, many standard
models thus obtained often exhibit issues like feature redundancy, low
interpretability, and poor calibration. A body of recent work has emerged that
has tried addressing some of these challenges by proposing the use of new
regularization functions in addition to the cross-entropy loss. In this paper,
we present some surprising findings that emerge from exploring the role of
simple orthogonality constraints as a means of imposing physics-motivated
constraints common in imaging. We propose an Orthogonal Sphere (OS) regularizer
that emerges from physics-based latent-representations under simplifying
assumptions. Under further simplifying assumptions, the OS constraint can be
written in closed-form as a simple orthonormality term and be used along with
the cross-entropy loss function. The findings indicate that orthonormality loss
function results in a) rich and diverse feature representations, b) robustness
to feature sub-selection, c) better semantic localization in the class
activation maps, and d) reduction in model calibration error. We demonstrate
the effectiveness of the proposed OS regularization by providing quantitative
and qualitative results on four benchmark datasets - CIFAR10, CIFAR100, SVHN
and tiny ImageNet.
Related papers
- Enhancing Fine-Grained Visual Recognition in the Low-Data Regime Through Feature Magnitude Regularization [23.78498670529746]
We introduce a regularization technique to ensure that the magnitudes of the extracted features are evenly distributed.
Despite its apparent simplicity, our approach has demonstrated significant performance improvements across various fine-grained visual recognition datasets.
arXiv Detail & Related papers (2024-09-03T07:32:46Z) - On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics.
The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
arXiv Detail & Related papers (2023-12-13T02:11:07Z) - A Fair Loss Function for Network Pruning [93.0013343535411]
We introduce the performance weighted loss function, a simple modified cross-entropy loss function that can be used to limit the introduction of biases during pruning.
Experiments using biased classifiers for facial classification and skin-lesion classification tasks demonstrate that the proposed method is a simple and effective tool.
arXiv Detail & Related papers (2022-11-18T15:17:28Z) - Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold.
We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples.
We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
arXiv Detail & Related papers (2022-06-23T14:16:30Z) - On the Optimization Landscape of Neural Collapse under MSE Loss: Global
Optimality with Unconstrained Features [38.05002597295796]
Collapselayers collapse to the vertices of a Simplex Equiangular Tight Frame (ETF)
An intriguing empirical phenomenon has been widely observed in the last-layers and features of deep neural networks for tasks.
arXiv Detail & Related papers (2022-03-02T17:00:18Z) - Learning Discriminative Shrinkage Deep Networks for Image Deconvolution [122.79108159874426]
We propose an effective non-blind deconvolution approach by learning discriminative shrinkage functions to implicitly model these terms.
Experimental results show that the proposed method performs favorably against the state-of-the-art ones in terms of efficiency and accuracy.
arXiv Detail & Related papers (2021-11-27T12:12:57Z) - Shaping Deep Feature Space towards Gaussian Mixture for Visual
Classification [74.48695037007306]
We propose a Gaussian mixture (GM) loss function for deep neural networks for visual classification.
With a classification margin and a likelihood regularization, the GM loss facilitates both high classification performance and accurate modeling of the feature distribution.
The proposed model can be implemented easily and efficiently without using extra trainable parameters.
arXiv Detail & Related papers (2020-11-18T03:32:27Z) - Deep Variational Network Toward Blind Image Restoration [60.45350399661175]
Blind image restoration is a common yet challenging problem in computer vision.
We propose a novel blind image restoration method, aiming to integrate both the advantages of them.
Experiments on two typical blind IR tasks, namely image denoising and super-resolution, demonstrate that the proposed method achieves superior performance over current state-of-the-arts.
arXiv Detail & Related papers (2020-08-25T03:30:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.