Self-Organising Neural Discrete Representation Learning à la Kohonen

• URL: http://arxiv.org/abs/2302.07950v2
• Date: Mon, 8 Jul 2024 19:47:40 GMT
• Title: Self-Organising Neural Discrete Representation Learning à la Kohonen
• Authors: Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber,
• Abstract summary: We study an alternative Vector Quantisation (VQ) algorithm based on Kohonen's learning rule for the Self-Organising Map (KSOM) In our experiments, the speed-up compared to well-configured EMA-VQ is only observable at the beginning of training.
• Score: 42.710124929514066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unsupervised learning of discrete representations in neural networks (NNs) from continuous ones is essential for many modern applications. Vector Quantisation (VQ) has become popular for this, in particular in the context of generative models, such as Variational Auto-Encoders (VAEs), where the exponential moving average-based VQ (EMA-VQ) algorithm is often used. Here, we study an alternative VQ algorithm based on Kohonen's learning rule for the Self-Organising Map (KSOM; 1982). EMA-VQ is a special case of KSOM. KSOM is known to offer two potential benefits: empirically, it converges faster than EMA-VQ, and KSOM-generated discrete representations form a topological structure on the grid whose nodes are the discrete symbols, resulting in an artificial version of the brain's topographic map. We revisit these properties by using KSOM in VQ-VAEs for image processing. In our experiments, the speed-up compared to well-configured EMA-VQ is only observable at the beginning of training, but KSOM is generally much more robust, e.g., w.r.t. the choice of initialisation schemes.

Related papers

• SOM-VQ: Topology-Aware Tokenization for Interactive Generative Models [41.99844472131922]
  We introduce SOM-VQ, a tokenization method that combines vector quantization with Self-Organizing Maps to learn discrete codebooks.<n> SOM-VQ produces more learnable token sequences while providing an explicit geometry in code space.<n>We focus on human motion generation - a domain where kinematic structure, smooth temporal continuity, and interactive use cases make topology-aware control especially natural.
  arXiv  Detail & Related papers  (2026-02-24T17:29:04Z)
• HyperVQ: MLR-based Vector Quantization in Hyperbolic Space [56.4245885674567]
  We study the use of hyperbolic spaces for vector quantization (HyperVQ) We show that hyperVQ performs comparably in reconstruction and generative tasks while outperforming VQ in discriminative tasks and learning a highly disentangled latent space.
  arXiv  Detail & Related papers  (2024-03-18T03:17:08Z)
• Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps [56.827895559823126]
  Self-organizing map (SOM) is a neural model often used in clustering and dimensionality reduction. We propose a generalization of the SOM, the continual SOM, which is capable of online unsupervised learning under a low memory budget. Our results, on benchmarks including MNIST, Kuzushiji-MNIST, and Fashion-MNIST, show almost a two times increase in accuracy.
  arXiv  Detail & Related papers  (2024-02-19T19:11:22Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Vector Quantized Time Series Generation with a Bidirectional Prior Model [0.3867363075280544]
  Time series generation (TSG) studies have mainly focused on the use of Generative Adversarial Networks (GANs) combined with recurrent neural network (RNN) variants. We propose TimeVQVAE, the first work, to our knowledge, that uses vector quantization (VQ) techniques to address the TSG problem. We also propose VQ modeling in a time-frequency domain, separated into low-frequency (LF) and high-frequency (HF)
  arXiv  Detail & Related papers  (2023-03-08T17:27:39Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)
• Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images [79.07468367923619]
  We develop convolutional neural generative coding (Conv-NGC) We implement a flexible neurobiologically-motivated algorithm that progressively refines latent state maps. We study the effectiveness of our brain-inspired neural system on the tasks of reconstruction and image denoising.
  arXiv  Detail & Related papers  (2022-11-22T06:42:41Z)
• SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization [13.075574481614478]
  One noted issue of vector-quantized variational autoencoder (VQ-VAE) is that the learned discrete representation uses only a fraction of the full capacity of the codebook. We propose a new training scheme that extends the standard VAE via novel dequantization and quantization. Our experiments show that SQ-VAE improves codebook utilization without using commons.
  arXiv  Detail & Related papers  (2022-05-16T09:49:37Z)
• Emergence of Lie symmetries in functional architectures learned by CNNs [63.69764116066748]
  We study the spontaneous development of symmetries in the early layers of a Convolutional Neural Network (CNN) during learning on natural images. Our architecture is built in such a way to mimic the early stages of biological visual systems.
  arXiv  Detail & Related papers  (2021-04-17T13:23:26Z)
• Margin-Based Regularization and Selective Sampling in Deep Neural Networks [7.219077740523683]
  We derive a new margin-based regularization formulation, termed multi-margin regularization (MMR) for deep neural networks (DNNs) We show improved empirical results on CIFAR10, CIFAR100 and ImageNet using state-of-the-art convolutional neural networks (CNNs) and BERT-BASE architecture for the MNLI, QQP, QNLI, MRPC, SST-2 and RTE benchmarks.
  arXiv  Detail & Related papers  (2020-09-13T15:06:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.