Investigating Permutation-Invariant Discrete Representation Learning for Spatially Aligned Images

• URL: http://arxiv.org/abs/2604.01843v1
• Date: Thu, 02 Apr 2026 09:56:59 GMT
• Title: Investigating Permutation-Invariant Discrete Representation Learning for Spatially Aligned Images
• Abstract summary: We propose the permutation-invariant vector-quantized autoencoder (PI-VQ), in which latent codes are constrained to carry no positional information.<n>We find that this constraint encourages codes to capture global, semantic features, and enables direct between images without a learned prior.<n>We evaluate PI-VQ on CelebA, CelebA-HQ and FFHQ, obtaining competitive precision, density and coverage metrics for images synthesised with our approach.
• Score: 20.603403172654726
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vector quantization approaches (VQ-VAE, VQ-GAN) learn discrete neural representations of images, but these representations are inherently position-dependent: codes are spatially arranged and contextually entangled, requiring autoregressive or diffusion-based priors to model their dependencies at sample time. In this work, we ask whether positional information is necessary for discrete representations of spatially aligned data. We propose the permutation-invariant vector-quantized autoencoder (PI-VQ), in which latent codes are constrained to carry no positional information. We find that this constraint encourages codes to capture global, semantic features, and enables direct interpolation between images without a learned prior. To address the reduced information capacity of permutation-invariant representations, we introduce matching quantization, a vector quantization algorithm based on optimal bipartite matching that increases effective bottleneck capacity by $3.5\times$ relative to naive nearest-neighbour quantization. The compositional structure of the learned codes further enables interpolation-based sampling, allowing synthesis of novel images in a single forward pass. We evaluate PI-VQ on CelebA, CelebA-HQ and FFHQ, obtaining competitive precision, density and coverage metrics for images synthesised with our approach. We discuss the trade-offs inherent to position-free representations, including separability and interpretability of the latent codes, pointing to numerous directions for future work.

Related papers

• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• Single-pass Adaptive Image Tokenization for Minimum Program Search [75.59409288259151]
  We propose a single-pass adaptive tokenizer, KARL, which predicts the appropriate number of tokens for an image in a single forward pass.<n> KARL matches the performance of recent adaptive tokenizers while operating in a single pass.
  arXiv  Detail & Related papers  (2025-07-10T17:59:53Z)
• Cross-Layer Discrete Concept Discovery for Interpreting Language Models [13.842670153893977]
  Cross-layer VQ-VAE is a framework that uses vector quantization to map representations across layers.<n>Our approach uniquely combines top-k temperature-based sampling during quantization with EMA codebook updates.
  arXiv  Detail & Related papers  (2025-06-24T22:43:36Z)
• Style Quantization for Data-Efficient GAN Training [18.40243591024141]
  Under limited data setting, GANs often struggle to navigate and effectively exploit the input latent space.<n>We propose textitSQ-GAN, a novel approach that enhances consistency regularization.<n>Experiments demonstrate significant improvements in both discriminator robustness and generation quality.
  arXiv  Detail & Related papers  (2025-03-31T16:28:44Z)
• SQ-GAN: Semantic Image Communications Using Masked Vector Quantization [54.35918290143049]
  This work introduces Semantically Masked Vector Quantized Generative Adversarial Network (SQ-GAN)<n>It is a novel approach integrating semantically driven image coding and vector quantization to optimize image compression for semantic/task-oriented communications.<n>SQ-GAN outperforms state-of-the-art image compression schemes such as JPEG2000, BPG, and deep-learning based methods across multiple metrics.
  arXiv  Detail & Related papers  (2025-02-13T17:35:57Z)
• Locality-Aware Generalizable Implicit Neural Representation [54.93702310461174]
  Generalizable implicit neural representation (INR) enables a single continuous function to represent multiple data instances. We propose a novel framework for generalizable INR that combines a transformer encoder with a locality-aware INR decoder. Our framework significantly outperforms previous generalizable INRs and validates the usefulness of the locality-aware latents for downstream tasks.
  arXiv  Detail & Related papers  (2023-10-09T11:26:58Z)
• Coordinate Quantized Neural Implicit Representations for Multi-view Reconstruction [28.910183274743872]
  We introduce neural implicit representations with quantized coordinates, which reduces the uncertainty and ambiguity in the field during optimization. We use discrete coordinates and their positional encodings to learn implicit functions through volume rendering. Our evaluations under the widely used benchmarks show our superiority over the state-of-the-art.
  arXiv  Detail & Related papers  (2023-08-21T20:27:33Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)
• Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization [76.68866368409216]
  We propose learning to dynamically select discretization tightness conditioned on inputs. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
  arXiv  Detail & Related papers  (2022-02-02T23:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.