SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning

• URL: http://arxiv.org/abs/2408.13351v1
• Date: Fri, 23 Aug 2024 19:55:13 GMT
• Title: SeA: Semantic Adversarial Augmentation for Last Layer Features from Unsupervised Representation Learning
• Authors: Qi Qian, Yuanhong Xu, Juhua Hu,
• Abstract summary: Deep features extracted from certain layers of a pre-trained deep model show superior performance over the conventional hand-crafted features. We propose a novel semantic adversarial augmentation (SeA) in the feature space for optimization. Our method is $2%$ better than the deep features without SeA on average.
• Score: 13.779858242220724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep features extracted from certain layers of a pre-trained deep model show superior performance over the conventional hand-crafted features. Compared with fine-tuning or linear probing that can explore diverse augmentations, \eg, random crop/flipping, in the original input space, the appropriate augmentations for learning with fixed deep features are more challenging and have been less investigated, which degenerates the performance. To unleash the potential of fixed deep features, we propose a novel semantic adversarial augmentation (SeA) in the feature space for optimization. Concretely, the adversarial direction implied by the gradient will be projected to a subspace spanned by other examples to preserve the semantic information. Then, deep features will be perturbed with the semantic direction, and augmented features will be applied to learn the classifier. Experiments are conducted on $11$ benchmark downstream classification tasks with $4$ popular pre-trained models. Our method is $2\%$ better than the deep features without SeA on average. Moreover, compared to the expensive fine-tuning that is expected to give good performance, SeA shows a comparable performance on $6$ out of $11$ tasks, demonstrating the effectiveness of our proposal in addition to its efficiency. Code is available at \url{https://github.com/idstcv/SeA}.

Related papers

• No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations [30.9134119244757]
  FUNGI is a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio.
  arXiv  Detail & Related papers  (2024-07-15T17:58:42Z)
• Beyond Prototypes: Semantic Anchor Regularization for Better Representation Learning [82.29761875805369]
  One of the ultimate goals of representation learning is to achieve compactness within a class and well-separability between classes. We propose a novel perspective to use pre-defined class anchors serving as feature centroid to unidirectionally guide feature learning. The proposed Semantic Anchor Regularization (SAR) can be used in a plug-and-play manner in the existing models.
  arXiv  Detail & Related papers  (2023-12-19T05:52:38Z)
• Enhancing Few-shot CLIP with Semantic-Aware Fine-Tuning [61.902254546858465]
  Methods based on Contrastive Language-Image Pre-training have exhibited promising performance in few-shot adaptation tasks. We propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics.
  arXiv  Detail & Related papers  (2023-11-08T05:18:57Z)
• DASA: Difficulty-Aware Semantic Augmentation for Speaker Verification [55.306583814017046]
  We present a novel difficulty-aware semantic augmentation (DASA) approach for speaker verification. DASA generates diversified training samples in speaker embedding space with negligible extra computing cost. The best result achieves a 14.6% relative reduction in EER metric on CN-Celeb evaluation set.
  arXiv  Detail & Related papers  (2023-10-18T17:07:05Z)
• Dynamic Perceiver for Efficient Visual Recognition [87.08210214417309]
  We propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features.
  arXiv  Detail & Related papers  (2023-06-20T03:00:22Z)
• Toward Adversarial Training on Contextualized Language Representation [78.39805974043321]
  This paper investigates adversarial training (AT) from the perspective of the contextualized language representation outputted by PLM encoders. We propose textitContextualized representation-Adversarial Training (CreAT) in which the attack is explicitly optimized to deviate the contextualized representation of the encoder. CreAT produces consistent performance gains on a wider range of tasks and is proven to be more effective for language pre-training where only the encoder part is kept for downstream tasks.
  arXiv  Detail & Related papers  (2023-05-08T08:56:51Z)
• Improved Regret for Efficient Online Reinforcement Learning with Linear Function Approximation [69.0695698566235]
  We study reinforcement learning with linear function approximation and adversarially changing cost functions. We present a computationally efficient policy optimization algorithm for the challenging general setting of unknown dynamics and bandit feedback.
  arXiv  Detail & Related papers  (2023-01-30T17:26:39Z)
• Hyperbolic Disentangled Representation for Fine-Grained Aspect Extraction [5.545062009366532]
  HDAE is a hyperbolic disentangled aspect extractor for user reviews. It achieves average F1 performance gains of 18.2% and 24.1% on Amazon product review and restaurant review datasets.
  arXiv  Detail & Related papers  (2021-12-16T21:47:28Z)
• Dense Prediction with Attentive Feature Aggregation [26.205279570906473]
  We introduce Attentive Feature Aggregation (AFA) to fuse different network layers with more expressive non-linear operations. AFA exploits both spatial and channel attention to compute weighted average of the layer activations. Our experiments show consistent and significant improvements on challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-11-01T08:44:45Z)
• The Geometry of Sign Gradient Descent [29.8753797565422]
  We show a close connection between separable smoothness and $ell_infty$-smoothness and argue that the latter is the weaker and more natural assumption. We then proceed to study the smoothness constant with respect to the $ell_infty$-norm and thereby isolate geometric properties of the objective function. In short, we find sign-based methods to be preferable over gradient descent if (i) the Hessian is to some degree concentrated on its diagonal, and (ii) its maximal eigenvalue is much larger than the average eigenvalue.
  arXiv  Detail & Related papers  (2020-02-19T08:45:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.