Unveiling Multiple Descents in Unsupervised Autoencoders

• URL: http://arxiv.org/abs/2406.11703v2
• Date: Tue, 18 Feb 2025 19:00:26 GMT
• Title: Unveiling Multiple Descents in Unsupervised Autoencoders
• Authors: Kobi Rahimi, Yehonathan Refael, Tom Tirer, Ofir Lindenbaum,
• Abstract summary: We show for the first time that double and triple descent can be observed with nonlinear unsupervised autoencoders. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent.
• Score: 13.180761892449736
• License:
• Abstract: The phenomenon of double descent has challenged the traditional bias-variance trade-off in supervised learning but remains unexplored in unsupervised learning, with some studies arguing for its absence. In this study, we first demonstrate analytically that double descent does not occur in linear unsupervised autoencoders (AEs). In contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the importance of bottleneck size in influencing the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. Our findings indicate that over-parameterized models not only improve reconstruction but also enhance performance in downstream tasks such as anomaly detection and domain adaptation, highlighting their practical value in complex real-world scenarios.

Related papers

• Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearning [19.229039345631406]
  The impact of data duplication on the unlearning process remains largely unexplored. We propose an adversary who duplicates a subset of the target model's training set and incorporates it into the training set. We then examine their impacts on the unlearning process when de-duplication techniques are applied.
  arXiv  Detail & Related papers  (2025-01-28T02:52:51Z)
• Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection [59.41026558455904]
  We focus on multi-modal anomaly detection. Specifically, we investigate early multi-modal approaches that attempted to utilize models pre-trained on large-scale visual datasets. We propose a Local-to-global Self-supervised Feature Adaptation (LSFA) method to finetune the adaptors and learn task-oriented representation toward anomaly detection.
  arXiv  Detail & Related papers  (2024-01-06T07:30:41Z)
• Generating and Reweighting Dense Contrastive Patterns for Unsupervised Anomaly Detection [59.34318192698142]
  We introduce a prior-less anomaly generation paradigm and develop an innovative unsupervised anomaly detection framework named GRAD. PatchDiff effectively expose various types of anomaly patterns. experiments on both MVTec AD and MVTec LOCO datasets also support the aforementioned observation.
  arXiv  Detail & Related papers  (2023-12-26T07:08:06Z)
• Video Anomaly Detection via Spatio-Temporal Pseudo-Anomaly Generation : A Unified Approach [49.995833831087175]
  This work proposes a novel method for generating generic Video-temporal PAs by inpainting a masked out region of an image. In addition, we present a simple unified framework to detect real-world anomalies under the OCC setting. Our method performs on par with other existing state-of-the-art PAs generation and reconstruction based methods under the OCC setting.
  arXiv  Detail & Related papers  (2023-11-27T13:14:06Z)
• A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning [68.76846801719095]
  We show that double descent appears exactly when and where it occurs, and that its location is not inherently tied to the threshold p=n. This provides a resolution to tensions between double descent and statistical intuition.
  arXiv  Detail & Related papers  (2023-10-29T12:05:39Z)
• Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space [12.907949196758565]
  Double descent presents a counter-intuitive aspect within the machine learning domain. We argue that double descent arises in imperfect models trained with noisy data.
  arXiv  Detail & Related papers  (2023-10-20T15:10:16Z)
• Learning from few examples with nonlinear feature maps [68.8204255655161]
  We explore the phenomenon and reveal key relationships between dimensionality of AI model's feature space, non-degeneracy of data distributions, and the model's generalisation capabilities. The main thrust of our present analysis is on the influence of nonlinear feature transformations mapping original data into higher- and possibly infinite-dimensional spaces on the resulting model's generalisation capabilities.
  arXiv  Detail & Related papers  (2022-03-31T10:36:50Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• Analytic Study of Double Descent in Binary Classification: The Impact of Loss [34.100845063076534]
  We show that the DD phenomenon persists, but we also identify several differences compared to logistic loss. We further study the dependence of DD curves on the size of the training set.
  arXiv  Detail & Related papers  (2020-01-30T21:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.