Latent space configuration for improved generalization in supervised autoencoder neural networks

• URL: http://arxiv.org/abs/2402.08441v2
• Date: Thu, 22 Feb 2024 07:38:30 GMT
• Title: Latent space configuration for improved generalization in supervised autoencoder neural networks
• Authors: Nikita Gabdullin
• Abstract summary: We propose two methods for obtaining LS with desired topology, called LS configuration. Knowing LS configuration allows to define similarity measure in LS to predict labels or estimate similarity for multiple inputs. We show that SAE trained for clothes texture classification using the proposed method generalizes well to unseen data from LIP, Market1501, and WildTrack datasets without fine-tuning.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoencoders (AE) are simple yet powerful class of neural networks that compress data by projecting input into low-dimensional latent space (LS). Whereas LS is formed according to the loss function minimization during training, its properties and topology are not controlled directly. In this paper we focus on AE LS properties and propose two methods for obtaining LS with desired topology, called LS configuration. The proposed methods include loss configuration using a geometric loss term that acts directly in LS, and encoder configuration. We show that the former allows to reliably obtain LS with desired configuration by defining the positions and shapes of LS clusters for supervised AE (SAE). Knowing LS configuration allows to define similarity measure in LS to predict labels or estimate similarity for multiple inputs without using decoders or classifiers. We also show that this leads to more stable and interpretable training. We show that SAE trained for clothes texture classification using the proposed method generalizes well to unseen data from LIP, Market1501, and WildTrack datasets without fine-tuning, and even allows to evaluate similarity for unseen classes. We further illustrate the advantages of pre-configured LS similarity estimation with cross-dataset searches and text-based search using a text query without language models.

Related papers

• HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms [14.10876324116018]
  We propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. We show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.
  arXiv  Detail & Related papers  (2024-10-30T16:49:59Z)
• A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
  Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
  arXiv  Detail & Related papers  (2024-07-16T23:17:36Z)
• Linguistic Steganalysis via LLMs: Two Modes for Efficient Detection of Strongly Concealed Stego [6.99735992267331]
  We design a novel LS with two modes called LSGC. In the generation mode, we created an LS-task "description" In the classification mode, LSGC deleted the LS-task "description" and used the "causalLM" LLMs to extract steganographic features.
  arXiv  Detail & Related papers  (2024-06-06T16:18:02Z)
• Automated Contrastive Learning Strategy Search for Time Series [48.68664732145665]
  We present an Automated Machine Learning (AutoML) practice at Microsoft, which automatically learns Contrastive Learning (AutoCL) for time series datasets and tasks. We first construct a principled search space of size over $3times1012$, covering data augmentation, embedding transformation, contrastive pair construction, and contrastive losses. Further, we introduce an efficient reinforcement learning algorithm, which optimize CLS from the performance on the validation tasks, to obtain effective CLS within the space.
  arXiv  Detail & Related papers  (2024-03-19T11:24:14Z)
• Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models [94.30953696090758]
  We build compositional end-to-end spoken language understanding systems. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to token-level representations. Our models outperform both cascaded and direct end-to-end models on a labeling task of named entity recognition.
  arXiv  Detail & Related papers  (2022-10-27T19:33:18Z)
• Weakly Supervised Label Smoothing [15.05158252504978]
  We study Label Smoothing (LS), a widely used regularization technique, in the context of neural learning to rank (L2R) models. Inspired by our investigation of LS in the context of neural L2R models, we propose a novel technique called Weakly Supervised Label Smoothing (WSLS)
  arXiv  Detail & Related papers  (2020-12-15T19:36:52Z)
• NSL: Hybrid Interpretable Learning From Noisy Raw Data [66.15862011405882]
  This paper introduces a hybrid neural-symbolic learning framework, called NSL, that learns interpretable rules from labelled unstructured data. NSL combines pre-trained neural networks for feature extraction with FastLAS, a state-of-the-art ILP system for rule learning under the answer set semantics. We demonstrate that NSL is able to learn robust rules from MNIST data and achieve comparable or superior accuracy when compared to neural network and random forest baselines.
  arXiv  Detail & Related papers  (2020-12-09T13:02:44Z)
• Adaptive Linear Span Network for Object Skeleton Detection [56.78705071830965]
  We propose adaptive linear span network (AdaLSN) to automatically configure and integrate scale-aware features for object skeleton detection. AdaLSN substantiates its versatility by achieving significantly higher accuracy and latency trade-off. It also demonstrates general applicability to image-to-mask tasks such as edge detection and road extraction.
  arXiv  Detail & Related papers  (2020-11-08T12:51:14Z)
• Semi-supervised source localization with deep generative modeling [27.344649091365067]
  We propose a semi-supervised localization approach based on deep generative modeling with variational autoencoders (VAEs) VAE-SSL can outperform both SRP-PHAT and CNN in label-limited scenarios.
  arXiv  Detail & Related papers  (2020-05-27T04:59:52Z)
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer [77.90012156266324]
  This paper aims to find a subspace of neural networks that can facilitate a large decision margin. We propose the Orthogonal Softmax Layer (OSL), which makes the weight vectors in the classification layer remain during both the training and test processes. Experimental results demonstrate that the proposed OSL has better performance than the methods used for comparison on four small-sample benchmark datasets.
  arXiv  Detail & Related papers  (2020-04-20T02:41:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.