Explicit and data-Efficient Encoding via Gradient Flow

• URL: http://arxiv.org/abs/2412.00864v2
• Date: Sat, 04 Jan 2025 05:09:44 GMT
• Title: Explicit and data-Efficient Encoding via Gradient Flow
• Authors: Kyriakos Flouris, Anna Volokitin, Gustav Bredell, Ender Konukoglu,
• Abstract summary: We introduce a decoder-only method using gradient flow to directly encode data into the latent space. We train the decoder via the adjoint method and show that costly integrals can be avoided with minimal accuracy loss. This work paves the way for integrating machine learning into scientific, where precise and efficient encoding is critical.
• Score: 13.424502866278822
• License:
• Abstract: The autoencoder model typically uses an encoder to map data to a lower dimensional latent space and a decoder to reconstruct it. However, relying on an encoder for inversion can lead to suboptimal representations, particularly limiting in physical sciences where precision is key. We introduce a decoder-only method using gradient flow to directly encode data into the latent space, defined by ordinary differential equations (ODEs). This approach eliminates the need for approximate encoder inversion. We train the decoder via the adjoint method and show that costly integrals can be avoided with minimal accuracy loss. Additionally, we propose a $2^{nd}$ order ODE variant, approximating Nesterov's accelerated gradient descent for faster convergence. To handle stiff ODEs, we use an adaptive solver that prioritizes loss minimization, improving robustness. Compared to traditional autoencoders, our method demonstrates explicit encoding and superior data efficiency, which is crucial for data-scarce scenarios in the physical sciences. Furthermore, this work paves the way for integrating machine learning into scientific workflows, where precise and efficient encoding is critical. \footnote{The code for this work is available at \url{https://github.com/k-flouris/gfe}.}

Related papers

• Gradient-free Decoder Inversion in Latent Diffusion Models [18.493960162113712]
  In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. We propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models.
  arXiv  Detail & Related papers  (2024-09-27T04:38:14Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Data-driven decoding of quantum error correcting codes using graph neural networks [0.0]
  We explore a model-free, data-driven, approach to decoding, using a graph neural network (GNN) We show that the GNN-based decoder can outperform a matching decoder for circuit level noise on the surface code given only simulated data. The results show that a purely data-driven approach to decoding may be a viable future option for practical quantum error correction.
  arXiv  Detail & Related papers  (2023-07-03T17:25:45Z)
• The END: An Equivariant Neural Decoder for Quantum Error Correction [73.4384623973809]
  We introduce a data efficient neural decoder that exploits the symmetries of the problem. We propose a novel equivariant architecture that achieves state of the art accuracy compared to previous neural decoders.
  arXiv  Detail & Related papers  (2023-04-14T19:46:39Z)
• Improving Dual-Encoder Training through Dynamic Indexes for Negative Mining [61.09807522366773]
  We introduce an algorithm that approximates the softmax with provable bounds and that dynamically maintains the tree. In our study on datasets with over twenty million targets, our approach cuts error by half in relation to oracle brute-force negative mining.
  arXiv  Detail & Related papers  (2023-03-27T15:18:32Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• A manifold learning perspective on representation learning: Learning decoder and representations without an encoder [0.0]
  Autoencoders are commonly used in representation learning. Inspired by manifold learning, we show that the decoder can be trained on its own by learning the representations of the training samples. Our approach of training the decoder alone facilitates representation learning even on small data sets.
  arXiv  Detail & Related papers  (2021-08-31T15:08:50Z)
• Neural Distributed Source Coding [59.630059301226474]
  We present a framework for lossy DSC that is agnostic to the correlation structure and can scale to high dimensions. We evaluate our method on multiple datasets and show that our method can handle complex correlations and state-of-the-art PSNR.
  arXiv  Detail & Related papers  (2021-06-05T04:50:43Z)
• Gradient flow encoding with distance optimization adaptive step size [10.973034520723957]
  We investigate a decoder-only method that uses gradient flow to encode data samples in the latent space. In our experiments, GFE showed a much higher data-efficiency than the autoencoding model, which can be crucial for data scarce applications.
  arXiv  Detail & Related papers  (2021-05-11T13:38:23Z)
• Auto-Encoding Twin-Bottleneck Hashing [141.5378966676885]
  This paper proposes an efficient and adaptive code-driven graph. It is updated by decoding in the context of an auto-encoder. Experiments on benchmarked datasets clearly show the superiority of our framework over the state-of-the-art hashing methods.
  arXiv  Detail & Related papers  (2020-02-27T05:58:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.