Impact of Regularization on Calibration and Robustness: from the Representation Space Perspective

• URL: http://arxiv.org/abs/2410.03999v1
• Date: Sat, 5 Oct 2024 02:09:03 GMT
• Title: Impact of Regularization on Calibration and Robustness: from the Representation Space Perspective
• Authors: Jonghyun Park, Juyeop Kim, Jong-Seok Lee,
• Abstract summary: Recent studies have shown that regularization techniques using soft labels enhance image classification accuracy and improve model calibration and robustness against adversarial attacks. In this paper, we offer a novel explanation from the perspective of the representation space. Our investigation first reveals that the decision regions in the representation space form cone-like shapes around the origin after training regardless of the presence of regularization.
• Score: 16.123727386404312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies have shown that regularization techniques using soft labels, e.g., label smoothing, Mixup, and CutMix, not only enhance image classification accuracy but also improve model calibration and robustness against adversarial attacks. However, the underlying mechanisms of such improvements remain underexplored. In this paper, we offer a novel explanation from the perspective of the representation space (i.e., the space of the features obtained at the penultimate layer). Our investigation first reveals that the decision regions in the representation space form cone-like shapes around the origin after training regardless of the presence of regularization. However, applying regularization causes changes in the distribution of features (or representation vectors). The magnitudes of the representation vectors are reduced and subsequently the cosine similarities between the representation vectors and the class centers (minimal loss points for each class) become higher, which acts as a central mechanism inducing improved calibration and robustness. Our findings provide new insights into the characteristics of the high-dimensional representation space in relation to training and regularization using soft labels.

Related papers

• Unifying Perplexing Behaviors in Modified BP Attributions through Alignment Perspective [61.5509267439999]
  We present a unified theoretical framework for methods like GBP, RectGrad, LRP, and DTD. We demonstrate that they achieve input alignment by combining the weights of activated neurons. This alignment improves the visualization quality and reduces sensitivity to weight randomization.
  arXiv  Detail & Related papers  (2025-03-14T07:58:26Z)
• Spatial regularisation for improved accuracy and interpretability in keypoint-based registration [5.286949071316761]
  Recent approaches based on unsupervised keypoint detection stand out as very promising for interpretability. Here, we propose a three-fold loss to regularise the spatial distribution of the features. Our loss considerably improves the interpretability of the features, which now correspond to precise and anatomically meaningful landmarks.
  arXiv  Detail & Related papers  (2025-03-06T14:48:25Z)
• Analysis of Spatial augmentation in Self-supervised models in the purview of training and test distributions [38.77816582772029]
  We present an empirical study of typical spatial augmentation techniques used in self-supervised representation learning methods. Our contributions are: (a) we dissociate random cropping into two separate augmentations, overlap and patch, and provide a detailed analysis on the effect of area of overlap and patch size to the accuracy on down stream tasks. We offer an insight into why cutout augmentation does not learn good representation, as reported in earlier literature.
  arXiv  Detail & Related papers  (2024-09-26T19:18:36Z)
• Understanding Imbalanced Semantic Segmentation Through Neural Collapse [81.89121711426951]
  We show that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes. We introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure. Our method ranks 1st and sets a new record on the ScanNet200 test leaderboard.
  arXiv  Detail & Related papers  (2023-01-03T13:51:51Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)
• Adaptive Local-Component-aware Graph Convolutional Network for One-shot Skeleton-based Action Recognition [54.23513799338309]
  We present an Adaptive Local-Component-aware Graph Convolutional Network for skeleton-based action recognition. Our method provides a stronger representation than the global embedding and helps our model reach state-of-the-art.
  arXiv  Detail & Related papers  (2022-09-21T02:33:07Z)
• SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimation [77.88624073105768]
  Category-level pose estimation is a challenging problem due to intra-class shape variations. We propose an end-to-end trainable network SSP-Pose for category-level pose estimation. SSP-Pose produces superior performance compared with competitors with a real-time inference speed at about 25Hz.
  arXiv  Detail & Related papers  (2022-08-13T14:37:31Z)
• Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification [3.6954802719347413]
  This paper presents an end-to-end framework that preserves alignment and uniformity properties for representations on both seen and unseen classes. Experiments show that our method significantly outperforms SoTA by relative improvements of 28.1% on UCF101 and 27.0% on HMDB51.
  arXiv  Detail & Related papers  (2022-03-29T09:21:22Z)
• Learning Where to Learn in Cross-View Self-Supervised Learning [54.14989750044489]
  Self-supervised learning (SSL) has made enormous progress and largely narrowed the gap with supervised ones. Current methods simply adopt uniform aggregation of pixels for embedding. We present a new approach, Learning Where to Learn (LEWEL), to adaptively aggregate spatial information of features.
  arXiv  Detail & Related papers  (2022-03-28T17:02:42Z)
• Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection [11.091890625685298]
  We study adapting trained object detectors to unseen domains manifesting significant variations of object appearance, viewpoints and backgrounds. We propose to leverage model predictive uncertainty to strike the right balance between adversarial feature alignment and class-level alignment.
  arXiv  Detail & Related papers  (2021-10-01T08:10:00Z)
• Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation [25.070027668717422]
  Generalized zero-shot semantic segmentation (GZS3) predicts pixel-wise semantic labels for seen and unseen classes. Most GZS3 methods adopt a generative approach that synthesizes visual features of unseen classes from corresponding semantic ones. We propose a discriminative approach to address limitations in a unified framework.
  arXiv  Detail & Related papers  (2021-08-14T13:33:58Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.