LDReg: Local Dimensionality Regularized Self-Supervised Learning

• URL: http://arxiv.org/abs/2401.10474v2
• Date: Thu, 14 Mar 2024 04:49:05 GMT
• Title: LDReg: Local Dimensionality Regularized Self-Supervised Learning
• Authors: Hanxun Huang, Ricardo J. G. B. Campello, Sarah Monazam Erfani, Xingjun Ma, Michael E. Houle, James Bailey,
• Abstract summary: Dimensional collapse also known as the "underfilling" phenomenon is one of the major causes of degraded performance on downstream tasks. Previous work has investigated the dimensional collapse problem of SSL at a global level. We propose a method called $textitlocal dimensionality regularization (LDReg)
• Score: 31.0201280709395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representations learned via self-supervised learning (SSL) can be susceptible to dimensional collapse, where the learned representation subspace is of extremely low dimensionality and thus fails to represent the full data distribution and modalities. Dimensional collapse also known as the "underfilling" phenomenon is one of the major causes of degraded performance on downstream tasks. Previous work has investigated the dimensional collapse problem of SSL at a global level. In this paper, we demonstrate that representations can span over high dimensional space globally, but collapse locally. To address this, we propose a method called $\textit{local dimensionality regularization (LDReg)}$. Our formulation is based on the derivation of the Fisher-Rao metric to compare and optimize local distance distributions at an asymptotically small radius for each data point. By increasing the local intrinsic dimensionality, we demonstrate through a range of experiments that LDReg improves the representation quality of SSL. The results also show that LDReg can regularize dimensionality at both local and global levels.

Related papers

• HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms [14.10876324116018]
  We propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. We show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.
  arXiv  Detail & Related papers  (2024-10-30T16:49:59Z)
• Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalization [61.51372812489661]
  We show that LDHD generalization is generally unattainable without exploiting prior knowledge to provide appropriate inductive bias. Applying the insights from LDHD generalization to length generalization, we explain the effectiveness of CoT as changing the structure latent space. We also propose a principle for position embedding design to handle both the inherent LDHD generalization and the nuisances such as the data format.
  arXiv  Detail & Related papers  (2024-10-11T15:18:43Z)
• Scalable manifold learning by uniform landmark sampling and constrained locally linear embedding [0.6144680854063939]
  We propose a scalable manifold learning (scML) method that can manipulate large-scale and high-dimensional data in an efficient manner. We empirically validated the effectiveness of scML on synthetic datasets and real-world benchmarks of different types. scML scales well with increasing data sizes and embedding dimensions, and exhibits promising performance in preserving the global structure.
  arXiv  Detail & Related papers  (2024-01-02T08:43:06Z)
• Sample Complexity of Neural Policy Mirror Descent for Policy Optimization on Low-Dimensional Manifolds [75.51968172401394]
  We study the sample complexity of the neural policy mirror descent (NPMD) algorithm with deep convolutional neural networks (CNN) In each iteration of NPMD, both the value function and the policy can be well approximated by CNNs. We show that NPMD can leverage the low-dimensional structure of state space to escape from the curse of dimensionality.
  arXiv  Detail & Related papers  (2023-09-25T07:31:22Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• Geometric Graph Representation Learning via Maximizing Rate Reduction [73.6044873825311]
  Learning node representations benefits various downstream tasks in graph analysis such as community detection and node classification. We propose Geometric Graph Representation Learning (G2R) to learn node representations in an unsupervised manner. G2R maps nodes in distinct groups into different subspaces, while each subspace is compact and different subspaces are dispersed.
  arXiv  Detail & Related papers  (2022-02-13T07:46:24Z)
• A Local Similarity-Preserving Framework for Nonlinear Dimensionality Reduction with Neural Networks [56.068488417457935]
  We propose a novel local nonlinear approach named Vec2vec for general purpose dimensionality reduction. To train the neural network, we build the neighborhood similarity graph of a matrix and define the context of data points. Experiments of data classification and clustering on eight real datasets show that Vec2vec is better than several classical dimensionality reduction methods in the statistical hypothesis test.
  arXiv  Detail & Related papers  (2021-03-10T23:10:47Z)
• LDLE: Low Distortion Local Eigenmaps [77.02534963571597]
  We present Low Distortion Local Eigenmaps (LDLE), a manifold learning technique which constructs a set of low distortion local views of a dataset in lower dimension and registers them to obtain a global embedding. The local views are constructed using the global eigenvectors of the graph Laplacian and are registered using Procrustes analysis.
  arXiv  Detail & Related papers  (2021-01-26T19:55:05Z)
• Learning a Deep Part-based Representation by Preserving Data Distribution [21.13421736154956]
  Unsupervised dimensionality reduction is one of the commonly used techniques in the field of high dimensional data recognition problems. In this paper, by preserving the data distribution, a deep part-based representation can be learned, and the novel algorithm is called Distribution Preserving Network Embedding. The experimental results on the real-world data sets show that the proposed algorithm has good performance in terms of cluster accuracy and AMI.
  arXiv  Detail & Related papers  (2020-09-17T12:49:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.