Localizing Memorization in SSL Vision Encoders

• URL: http://arxiv.org/abs/2409.19069v2
• Date: Tue, 29 Oct 2024 20:13:18 GMT
• Title: Localizing Memorization in SSL Vision Encoders
• Authors: Wenhao Wang, Adam Dziedzic, Michael Backes, Franziska Boenisch,
• Abstract summary: We propose two metrics for localizing memorization in SSL encoders on a per-layer (layermem) and per-unit basis (unitmem) We find that while SSL memorization increases with layer depth, highly memorizing units are distributed across the entire encoder.
• Score: 24.681788021239118
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained on millions of images, they still memorize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this gap, we propose two metrics for localizing memorization in SSL encoders on a per-layer (layermem) and per-unit basis (unitmem). Our localization methods are independent of the downstream task, do not require any label information, and can be performed in a forward pass. By localizing memorization in various encoder architectures (convolutional and transformer-based) trained on diverse datasets with contrastive and non-contrastive SSL frameworks, we find that (1) while SSL memorization increases with layer depth, highly memorizing units are distributed across the entire encoder, (2) a significant fraction of units in SSL encoders experiences surprisingly high memorization of individual data points, which is in contrast to models trained under supervision, (3) atypical (or outlier) data points cause much higher layer and unit memorization than standard data points, and (4) in vision transformers, most memorization happens in the fully-connected layers. Finally, we show that localizing memorization in SSL has the potential to improve fine-tuning and to inform pruning strategies.

Related papers

• Memorization in Self-Supervised Learning Improves Downstream Generalization [49.42010047574022]
  Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data. We propose SSLMem, a framework for defining memorization within SSL.
  arXiv  Detail & Related papers  (2024-01-19T11:32:47Z)
• Exploring Memorization in Fine-tuned Language Models [53.52403444655213]
  We conduct the first comprehensive analysis to explore language models' memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.
  arXiv  Detail & Related papers  (2023-10-10T15:41:26Z)
• Speech Self-Supervised Representation Benchmarking: Are We Doing it Right? [24.354848095744536]
  Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on speech tasks. benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.
  arXiv  Detail & Related papers  (2023-06-01T08:51:18Z)
• Transfer-Free Data-Efficient Multilingual Slot Labeling [82.02076369811402]
  Slot labeling is a core component of task-oriented dialogue (ToD) systems. To mitigate the inherent data scarcity issue, current research on multilingual ToD assumes that sufficient English-language annotated data are always available. We propose a two-stage slot labeling approach (termed TWOSL) which transforms standard multilingual sentence encoders into effective slot labelers.
  arXiv  Detail & Related papers  (2023-05-22T22:47:32Z)
• Do SSL Models Have D\'ej\`a Vu? A Case of Unintended Memorization in Self-supervised Learning [47.46863155263094]
  Self-supervised learning (SSL) algorithms can produce useful image representations by learning to associate different parts of natural images with one another. SSL models can unintendedly memorize specific parts in individual training samples rather than learning semantically meaningful associations. We show that given the trained model and a crop of a training image containing only the background, it is possible to infer the foreground object with high accuracy.
  arXiv  Detail & Related papers  (2023-04-26T22:29:49Z)
• De-coupling and De-positioning Dense Self-supervised Learning [65.56679416475943]
  Dense Self-Supervised Learning (SSL) methods address the limitations of using image-level feature representations when handling images with multiple objects. We show that they suffer from coupling and positional bias, which arise from the receptive field increasing with layer depth and zero-padding. We demonstrate the benefits of our method on COCO and on a new challenging benchmark, OpenImage-MINI, for object classification, semantic segmentation, and object detection.
  arXiv  Detail & Related papers  (2023-03-29T18:07:25Z)
• A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends [82.64268080902742]
  Self-supervised learning (SSL) aims to learn discriminative features from unlabeled data without relying on human-annotated labels. SSL has garnered significant attention recently, leading to the development of numerous related algorithms. This paper presents a review of diverse SSL methods, encompassing algorithmic aspects, application domains, three key trends, and open research questions.
  arXiv  Detail & Related papers  (2023-01-13T14:41:05Z)
• Joint Encoder-Decoder Self-Supervised Pre-training for ASR [0.0]
  Self-supervised learning has shown tremendous success in various speech-related downstream tasks. In this paper, we propose a new paradigm that exploits the power of a decoder during self-supervised learning.
  arXiv  Detail & Related papers  (2022-06-09T12:45:29Z)
• Sound and Visual Representation Learning with Multiple Pretraining Tasks [104.11800812671953]
  Self-supervised tasks (SSL) reveal different features from the data. This work aims to combine Multiple SSL tasks (Multi-SSL) that generalizes well for all downstream tasks. Experiments on sound representations demonstrate that Multi-SSL via incremental learning (IL) of SSL tasks outperforms single SSL task models.
  arXiv  Detail & Related papers  (2022-01-04T09:09:38Z)
• Generalized Continual Zero-Shot Learning [7.097782028036196]
  zero-shot learning (ZSL) aims to classify unseen classes by transferring the knowledge from seen classes to unseen classes based on the class description. We propose a more general and practical setup for ZSL, where classes arrive sequentially in the form of a task. We use knowledge distillation and storing and replay the few samples from previous tasks using a small episodic memory.
  arXiv  Detail & Related papers  (2020-11-17T08:47:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.