URLOST: Unsupervised Representation Learning without Stationarity or Topology

• URL: http://arxiv.org/abs/2310.04496v2
• Date: Fri, 21 Mar 2025 17:59:54 GMT
• Title: URLOST: Unsupervised Representation Learning without Stationarity or Topology
• Authors: Zeyu Yun, Juexiao Zhang, Yann LeCun, Yubei Chen,
• Abstract summary: We introduce a novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology.<n>Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder.<n>We evaluate its effectiveness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions.
• Score: 26.010647961403148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised representation learning has seen tremendous progress. However, it is constrained by its reliance on domain specific stationarity and topology, a limitation not found in biological intelligence systems. For instance, unlike computer vision, human vision can process visual signals sampled from highly irregular and non-stationary sensors. We introduce a novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology. Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder (MAE). We evaluate its effectiveness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions. Compared to state-of-the-art unsupervised learning methods like SimCLR and MAE, our model excels at learning meaningful representations across diverse modalities without knowing their stationarity or topology. It also outperforms other methods that are not dependent on these factors, setting a new benchmark in the field. We position this work as a step toward unsupervised learning methods capable of generalizing across diverse high-dimensional data modalities.

Related papers

• Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems [80.62854148838359]
  Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. We use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data. Our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.
  arXiv  Detail & Related papers  (2024-03-23T22:32:06Z)
• Joint-Embedding Masked Autoencoder for Self-supervised Learning of Dynamic Functional Connectivity from the Human Brain [18.165807360855435]
  Graph Neural Networks (GNNs) have shown promise in learning dynamic functional connectivity for distinguishing phenotypes from human brain networks. We introduce the Spatio-Temporal Joint Embedding Masked Autoencoder (ST-JEMA), drawing inspiration from the Joint Embedding Predictive Architecture (JEPA) in computer vision.
  arXiv  Detail & Related papers  (2024-03-11T04:49:41Z)
• Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning [58.93724285214628]
  We present Self-guided Masked Autoencoders (SMA), a fully domain-agnostic masked modeling method. SMA trains an attention based model using a masked modeling objective, by learning masks to sample without any domain-specific assumptions. We evaluate SMA on three self-supervised learning benchmarks in protein biology, chemical property prediction, and particle physics.
  arXiv  Detail & Related papers  (2024-02-22T18:46:22Z)
• Towards Unsupervised Representation Learning: Learning, Evaluating and Transferring Visual Representations [1.8130068086063336]
  We contribute to the field of unsupervised (visual) representation learning from three perspectives. We design unsupervised, backpropagation-free Convolutional Self-Organizing Neural Networks (CSNNs) We build upon the widely used (non-)linear evaluation protocol to define pretext- and target-objective-independent metrics. We contribute CARLANE, the first 3-way sim-to-real domain adaptation benchmark for 2D lane detection, and a method based on self-supervised learning.
  arXiv  Detail & Related papers  (2023-11-30T15:57:55Z)
• Homological Convolutional Neural Networks [4.615338063719135]
  We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations. We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
  arXiv  Detail & Related papers  (2023-08-26T08:48:51Z)
• Free-ATM: Exploring Unsupervised Learning on Diffusion-Generated Images with Free Attention Masks [64.67735676127208]
  Text-to-image diffusion models have shown great potential for benefiting image recognition. Although promising, there has been inadequate exploration dedicated to unsupervised learning on diffusion-generated images. We introduce customized solutions by fully exploiting the aforementioned free attention masks.
  arXiv  Detail & Related papers  (2023-08-13T10:07:46Z)
• TS-MoCo: Time-Series Momentum Contrast for Self-Supervised Physiological Representation Learning [8.129782272731397]
  We propose a novel encoding framework that relies on self-supervised learning with momentum contrast to learn representations from various physiological domains without needing labels. We show that our self-supervised learning approach can indeed learn discriminative features which can be exploited in downstream classification tasks.
  arXiv  Detail & Related papers  (2023-06-10T21:17:42Z)
• ProtoVAE: Prototypical Networks for Unsupervised Disentanglement [1.6114012813668934]
  We introduce a novel deep generative VAE-based model, ProtoVAE, that leverages a deep metric learning Prototypical network trained using self-supervision. Our model is completely unsupervised and requires no priori knowledge of the dataset, including the number of factors. We evaluate our proposed model on the benchmark dSprites, 3DShapes, and MPI3D disentanglement datasets.
  arXiv  Detail & Related papers  (2023-05-16T01:29:26Z)
• Do Neural Networks Trained with Topological Features Learn Different Internal Representations? [1.418465438044804]
  We investigate whether a model trained with topological features learns internal representations of data that are fundamentally different than those learned by a model trained with the original raw data. We find that structurally, the hidden representations of models trained and evaluated on topological features differ substantially compared to those trained and evaluated on the corresponding raw data. We conjecture that this means that neural networks trained on raw data may extract some limited topological features in the process of making predictions.
  arXiv  Detail & Related papers  (2022-11-14T19:19:04Z)
• Bridging the Gap to Real-World Object-Centric Learning [66.55867830853803]
  We show that reconstructing features from models trained in a self-supervised manner is a sufficient training signal for object-centric representations to arise in a fully unsupervised way. Our approach, DINOSAUR, significantly out-performs existing object-centric learning models on simulated data.
  arXiv  Detail & Related papers  (2022-09-29T15:24:47Z)
• Stochastic Coherence Over Attention Trajectory For Continuous Learning In Video Streams [64.82800502603138]
  This paper proposes a novel neural-network-based approach to progressively and autonomously develop pixel-wise representations in a video stream. The proposed method is based on a human-like attention mechanism that allows the agent to learn by observing what is moving in the attended locations. Our experiments leverage 3D virtual environments and they show that the proposed agents can learn to distinguish objects just by observing the video stream.
  arXiv  Detail & Related papers  (2022-04-26T09:52:31Z)
• Neurosymbolic hybrid approach to driver collision warning [64.02492460600905]
  There are two main algorithmic approaches to autonomous driving systems. Deep learning alone has achieved state-of-the-art results in many areas. But sometimes it can be very difficult to debug if the deep learning model doesn't work.
  arXiv  Detail & Related papers  (2022-03-28T20:29:50Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Self-Supervised Graph Representation Learning for Neuronal Morphologies [75.38832711445421]
  We present GraphDINO, a data-driven approach to learn low-dimensional representations of 3D neuronal morphologies from unlabeled datasets. We show, in two different species and across multiple brain areas, that this method yields morphological cell type clusterings on par with manual feature-based classification by experts. Our method could potentially enable data-driven discovery of novel morphological features and cell types in large-scale datasets.
  arXiv  Detail & Related papers  (2021-12-23T12:17:47Z)
• Multi-Modal Anomaly Detection for Unstructured and Uncertain Environments [5.677685109155077]
  Modern robots require the ability to detect and recover from anomalies and failures with minimal human supervision. We propose a deep learning neural network: supervised variational autoencoder (SVAE), for failure identification in unstructured and uncertain environments. Our experiments on real field robot data demonstrate superior failure identification performance than baseline methods, and that our model learns interpretable representations.
  arXiv  Detail & Related papers  (2020-12-15T21:59:58Z)
• A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention [96.77554122595578]
  We introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference. Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning mechanism with small computational cost.
  arXiv  Detail & Related papers  (2020-06-22T08:35:58Z)
• Manifolds for Unsupervised Visual Anomaly Detection [79.22051549519989]
  Unsupervised learning methods that don't necessarily encounter anomalies in training would be immensely useful. We develop a novel hyperspherical Variational Auto-Encoder (VAE) via stereographic projections with a gyroplane layer. We present state-of-the-art results on visual anomaly benchmarks in precision manufacturing and inspection, demonstrating real-world utility in industrial AI scenarios.
  arXiv  Detail & Related papers  (2020-06-19T20:41:58Z)
• Evolving Losses for Unsupervised Video Representation Learning [91.2683362199263]
  We present a new method to learn video representations from large-scale unlabeled video data. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods.
  arXiv  Detail & Related papers  (2020-02-26T16:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.