Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection

• URL: http://arxiv.org/abs/2302.03857v5
• Date: Thu, 26 Oct 2023 09:15:14 GMT
• Title: Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection
• Authors: Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, Mohan Kankanhalli
• Abstract summary: Adversarial contrast learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks. ACL needs tremendous running time to generate the adversarial variants of all training data. This paper proposes a robustness-aware coreset selection (RCS) method to speed up ACL.
• Score: 59.77647907277523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial contrastive learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks and also generalizes to a wide range of downstream tasks. However, ACL needs tremendous running time to generate the adversarial variants of all training data, which limits its scalability to large datasets. To speed up ACL, this paper proposes a robustness-aware coreset selection (RCS) method. RCS does not require label information and searches for an informative subset that minimizes a representational divergence, which is the distance of the representation between natural data and their virtual adversarial variants. The vanilla solution of RCS via traversing all possible subsets is computationally prohibitive. Therefore, we theoretically transform RCS into a surrogate problem of submodular maximization, of which the greedy search is an efficient solution with an optimality guarantee for the original problem. Empirically, our comprehensive results corroborate that RCS can speed up ACL by a large margin without significantly hurting the robustness transferability. Notably, to the best of our knowledge, we are the first to conduct ACL efficiently on the large-scale ImageNet-1K dataset to obtain an effective robust representation via RCS. Our source code is at https://github.com/GodXuxilie/Efficient_ACL_via_RCS.

Related papers

• L^2CL: Embarrassingly Simple Layer-to-Layer Contrastive Learning for Graph Collaborative Filtering [33.165094795515785]
  Graph neural networks (GNNs) have recently emerged as an effective approach to model neighborhood signals in collaborative filtering. We propose L2CL, a principled Layer-to-Layer Contrastive Learning framework that contrasts representations from different layers. We find that L2CL, using only one-hop contrastive learning paradigm, is able to capture intrinsic semantic structures and improve the quality of node representation.
  arXiv  Detail & Related papers  (2024-07-19T12:45:21Z)
• Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control [66.78146440275093]
  Learned retrieval (LSR) is a family of neural methods that encode queries and documents into sparse lexical vectors. We explore the application of LSR to the multi-modal domain, with a focus on text-image retrieval. Current approaches like LexLIP and STAIR require complex multi-step training on massive datasets. Our proposed approach efficiently transforms dense vectors from a frozen dense model into sparse lexical vectors.
  arXiv  Detail & Related papers  (2024-02-27T14:21:56Z)
• RecDCL: Dual Contrastive Learning for Recommendation [65.6236784430981]
  We propose a dual contrastive learning recommendation framework -- RecDCL. In RecDCL, the FCL objective is designed to eliminate redundant solutions on user-item positive pairs. The BCL objective is utilized to generate contrastive embeddings on output vectors for enhancing the robustness of the representations.
  arXiv  Detail & Related papers  (2024-01-28T11:51:09Z)
• Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach [58.57026686186709]
  We introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR) CFSR inherits the advantages of both convolution-based and transformer-based approaches. Experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance.
  arXiv  Detail & Related papers  (2024-01-11T03:08:00Z)
• Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization [59.77647907277523]
  Adversarial contrastive learning (ACL) is a technique that enhances standard contrastive learning (SCL) In this paper, we propose adversarial invariant regularization (AIR) to enforce independence from style factors.
  arXiv  Detail & Related papers  (2023-04-30T03:12:21Z)
• TransCL: Transformer Makes Strong and Flexible Compressive Learning [11.613886854794133]
  Compressive learning (CL) is an emerging framework that integrates signal acquisition via compressed sensing (CS) and machine learning for inference tasks directly on a small number of measurements. Previous attempts on CL are not only limited to a fixed CS ratio, but also limited to MNIST/CIFAR-like datasets and do not scale to complex real-world high-resolution (HR) data or vision tasks. In this paper, a novel transformer-based compressive learning framework on large-scale images with arbitrary CS ratios, dubbed TransCL, is proposed.
  arXiv  Detail & Related papers  (2022-07-25T08:21:48Z)
• Decoupled Contrastive Learning [23.25775900388382]
  We identify a noticeable negative-positive-coupling (NPC) effect in the widely used cross-entropy (InfoNCE) loss. By properly addressing the NPC effect, we reach a decoupled contrastive learning (DCL) objective function. Our approach achieves $66.9%$ ImageNet top-1 accuracy using batch size 256 within 200 epochs pre-training, outperforming its baseline SimCLR by $5.1%$.
  arXiv  Detail & Related papers  (2021-10-13T16:38:43Z)
• Adversarial Representation Learning With Closed-Form Solvers [29.933607957877335]
  Existing methods learn model parameters iteratively through descent-ascent gradient, which is often unstable and unreliable in practice. We model them as kernel ridge regressors and analytically determine an upper-bound on the optimal dimensionality of representation. Our solution, dubbed OptNet-ARL, reduces to a stable one-shot optimization problem that can be solved reliably and efficiently.
  arXiv  Detail & Related papers  (2021-09-12T15:12:23Z)
• On Coresets for Support Vector Machines [61.928187390362176]
  A coreset is a small, representative subset of the original data points. We show that our algorithm can be used to extend the applicability of any off-the-shelf SVM solver to streaming, distributed, and dynamic data settings.
  arXiv  Detail & Related papers  (2020-02-15T23:25:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.