Principal Bit Analysis: Autoencoding with Schur-Concave Loss

• URL: http://arxiv.org/abs/2106.02796v2
• Date: Tue, 8 Jun 2021 16:16:13 GMT
• Title: Principal Bit Analysis: Autoencoding with Schur-Concave Loss
• Abstract summary: We consider a linear autocoder in which the variables are latent or corrupted by noise. As a second application, we consider a prototypical prototypical fixed-rate compressor.
• Score: 45.420030784128166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider a linear autoencoder in which the latent variables are quantized, or corrupted by noise, and the constraint is Schur-concave in the set of latent variances. Although finding the optimal encoder/decoder pair for this setup is a nonconvex optimization problem, we show that decomposing the source into its principal components is optimal. If the constraint is strictly Schur-concave and the empirical covariance matrix has only simple eigenvalues, then any optimal encoder/decoder must decompose the source in this way. As one application, we consider a strictly Schur-concave constraint that estimates the number of bits needed to represent the latent variables under fixed-rate encoding, a setup that we call \emph{Principal Bit Analysis (PBA)}. This yields a practical, general-purpose, fixed-rate compressor that outperforms existing algorithms. As a second application, we show that a prototypical autoencoder-based variable-rate compressor is guaranteed to decompose the source into its principal components.

Related papers

• Coset Ensemble Decoder for Quantum Error Correction with Algorithm-Hardware Co-Design [16.143601808851162]
  quantum error correction (QEC) continuously extracts and decodes error syndromes in real time.<n>A critical component in QEC is the decoder, a classical subsystem that must simultaneously deliver high logical accuracy and ultra-low latency.<n>This paper presents a novel algorithm-hardware co-design that improves the accuracy-latency trade-off over existing approaches.
  arXiv  Detail & Related papers  (2026-06-09T16:37:13Z)
• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs [50.3757247538081]
  We retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline.<n>We decompose each encoder's contribution into two axes, Capacity and Necessity.<n>Per-encoder pre-projector effective rank explains the residual score variation.
  arXiv  Detail & Related papers  (2026-06-02T16:46:42Z)
• Synthesis and Optimization of Encoding Circuits for Fault-Tolerant Quantum Computation [4.561664406615985]
  We develop methods for synthesizing efficient encoders for arbitrary stabilizer codes.<n>For families with a modular structure, such as tableaud and holographic codes, we show how large encoders can be assembled from optimized local constituent encoders.<n>We obtain improvements of up to 43% in two-qubit gate count and up to 70% in depth.
  arXiv  Detail & Related papers  (2026-05-14T18:00:01Z)
• Generative Decompression: Optimal Lossy Decoding Against Distribution Mismatch [20.28600385545221]
  This paper addresses optimal decoding strategies in lossy compression where the assumed distribution for compressor design mismatches the actual (true) distribution of the source.<n>We formally define the mismatched quantization problem, demonstrating that the optimal reconstruction rule, termed generative decompression, aligns with classical Bayesian estimation.<n>We extend this framework to transmission over noisy channels, deriving a robust soft-decoding rule that quantifies the inefficiency of standard modular source--channel separation architectures under mismatch.
  arXiv  Detail & Related papers  (2026-02-03T13:25:23Z)
• Generalization Bounds for Transformer Channel Decoders [61.55280736553095]
  This paper studies the generalization performance of ECCT from a learning-theoretic perspective.<n>To the best of our knowledge, this work provides the first theoretical generalization guarantees for this class of decoders.
  arXiv  Detail & Related papers  (2026-01-11T15:56:37Z)
• The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I) [6.453417258264177]
  This paper introduces Error-Bounded Predictive Coding ( EPC), a lossy text that leverages a Masked Language Model (MLM) as a decompressor.<n>Instead of storing a subset of original tokens, EPC allows the model to predict masked content and stores minimal, rank-based corrections only when the model's top prediction is incorrect.<n>We demonstrate that EPC consistently dominates Predictive Masking, offering superior fidelity at a significantly lower bit rate by more efficiently utilising the model's intrinsic knowledge.
  arXiv  Detail & Related papers  (2025-10-25T08:18:31Z)
• Decomposer Networks: Deep Component Analysis and Synthesis [0.14504054468850663]
  We propose a semantic autoencoder that factorizes an input into multiple interpretable components.<n>By unrolling a Gauss--Seidel style block-coordinate descent into a differentiable network, DecompNet enforces explicit competition among components.<n>We highlight its novelty as the first semantic autoencoder to implement an all-but-one residual update rule.
  arXiv  Detail & Related papers  (2025-10-10T19:55:13Z)
• Explicit and data-Efficient Encoding via Gradient Flow [13.424502866278822]
  We introduce a decoder-only method using gradient flow to directly encode data into the latent space. We train the decoder via the adjoint method and show that costly integrals can be avoided with minimal accuracy loss. This work paves the way for integrating machine learning into scientific, where precise and efficient encoding is critical.
  arXiv  Detail & Related papers  (2024-12-01T15:54:50Z)
• Rank Reduction Autoencoders -- Enhancing interpolation on nonlinear manifolds [3.180674374101366]
  Rank Reduction Autoencoder (RRAE) is an autoencoder with an enlarged latent space. Two formulations are presented, a strong and a weak one, that build a reduced basis accurately representing the latent space. We show the efficiency of our formulations by using them for tasks and comparing the results to other autoencoders.
  arXiv  Detail & Related papers  (2024-05-22T20:33:09Z)
• Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding [33.377272636443344]
  neural compression design involves transforming the source to a latent vector, which is then rounded to integers and entropy coded. We show that it is highly sub-optimal on i.i.d. sequences, and in fact always recovers scalar quantization of the original source sequence. By employing lattice quantization instead of scalar quantization in the latent space, we demonstrate that Lattice Transform Coding (LTC) is able to recover optimal vector quantization at various dimensions.
  arXiv  Detail & Related papers  (2024-03-12T05:09:25Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Bit-flipping Decoder Failure Rate Estimation for (v,w)-regular Codes [84.0257274213152]
  We propose a new technique to provide accurate estimates of the DFR of a two-iterations (parallel) bit flipping decoder. We validate our results, providing comparisons of the modeled and simulated weight of the syndrome, incorrectly-guessed error bit distribution at the end of the first iteration, and two-itcrypteration Decoding Failure Rates (DFR)
  arXiv  Detail & Related papers  (2024-01-30T11:40:24Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Error-rate-agnostic decoding of topological stabilizer codes [0.0]
  We develop a decoder that depends on the bias, i.e., the relative probability of phase-flip to bit-flip errors, but is agnostic to error rate. Our decoder is based on counting the number and effective weight of the most likely error chains in each equivalence class of a given syndrome.
  arXiv  Detail & Related papers  (2021-12-03T15:45:12Z)
• End-to-end optimized image compression with competition of prior distributions [29.585370305561582]
  We propose a compression scheme that uses a single convolutional autoencoder and multiple learned prior distributions. Our method offers rate-distortion performance comparable to that obtained with a predicted parametrized prior.
  arXiv  Detail & Related papers  (2021-11-17T15:04:01Z)
• Neural Distributed Source Coding [59.630059301226474]
  We present a framework for lossy DSC that is agnostic to the correlation structure and can scale to high dimensions. We evaluate our method on multiple datasets and show that our method can handle complex correlations and state-of-the-art PSNR.
  arXiv  Detail & Related papers  (2021-06-05T04:50:43Z)
• Integer Programming-based Error-Correcting Output Code Design for Robust Classification [4.7464518249313805]
  Error-Correcting Output Codes (ECOCs) offer a principled approach for combining simple binary classifiers into multiclass classifiers. In this paper, we investigate the problem of designing optimal ECOCs to achieve both nominal and adversarial accuracy. Our work leverages the advances in IP solvers to generate codebooks with optimality guarantees.
  arXiv  Detail & Related papers  (2020-10-30T23:35:18Z)
• Simple and Effective VAE Training with Calibrated Decoders [123.08908889310258]
  Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution. We propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically.
  arXiv  Detail & Related papers  (2020-06-23T17:57:47Z)
• Variance Constrained Autoencoding [0.0]
  We show that for encoders, simultaneously attempting to enforce a distribution constraint and minimising an output distortion leads to a reduction in generative and reconstruction quality. We propose the variance-constrained autoencoder (VCAE), which only enforces a variance constraint on the latent distribution. Our experiments show that VCAE improves upon Wasserstein Autoencoder and the Variational Autoencoder in both reconstruction and generative quality on MNIST and CelebA.
  arXiv  Detail & Related papers  (2020-05-08T00:50:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.