DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders

• URL: http://arxiv.org/abs/2605.22777v1
• Date: Thu, 21 May 2026 17:34:04 GMT
• Title: DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders
• Abstract summary: Representation Autoencoders (RAEs) leverage frozen vision foundation models (VFMs) as tokenizer encoders.<n>In contrast, incorporating reconstruction-oriented signals via fine-tuning disrupts the pretrained semantic space and degrades generative fidelity.<n>We propose DecQ, a simple yet effective framework for RAEs. Specifically, DecQ introduces lightweight detail-condensing queries that extract fine-grained information from intermediate VFM features.
• Score: 50.02172864947277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representation Autoencoders (RAEs) leverage frozen vision foundation models (VFMs) as tokenizer encoders, providing robust high-level representations that facilitate fast convergence and high-quality generation in latent diffusion models. However, freezing the VFM inherently constrains its spatial reconstruction capacity, limiting fine-grained generation and image editing; in contrast, incorporating reconstruction-oriented signals via fine-tuning disrupts the pretrained semantic space and degrades generative fidelity. To address this trade-off, we propose DecQ, a simple yet effective framework for RAEs. Specifically, DecQ introduces lightweight detail-condensing queries that extract fine-grained information from intermediate VFM features through condenser modules. These queries are incorporated into the decoder to support reconstruction and are jointly generated with patch tokens during generative modeling. By aggregating information from both shallow and deep layers, DecQ effectively mitigates the reconstruction--generation trade-off, improving both reconstruction quality and generative performance. Our experiments demonstrate that: (1) with only 8 additional queries and 3.9% extra computation, DecQ improves reconstruction over the frozen DINOv2-based RAE, increasing PSNR from 19.13 dB to 22.76 dB; and (2) for generative modeling, DecQ achieves 3.3$\times$ faster convergence than RAE, attaining an FID of 1.41 without guidance and 1.05 with guidance.

Related papers

• HDRFace: Rethinking Face Restoration with High-Dimensional Representation [47.1564555410251]
  We propose a High-Dimensional Representation conditioned Face restoration framework that injects semantically rich priors into the conditional flow without modifying the generative backbone.<n>Our pipeline first obtains a structurally reliable intermediate restoration with an off-the-shelf restorer, then uses a pretrained high-dimensional feature encoder to extract fine-grained facial representations from both the low-quality input and the intermediate result.<n>We further introduce SDFM, a Structure-Detail aware adaptive Fusion Mechanism that emphasizes global constraints during structure modeling and strengthens representation guidance during detail synthesis.
  arXiv  Detail & Related papers  (2026-05-14T13:31:24Z)
• RAE-AR: Taming Autoregressive Models with Representation Autoencoders [61.73674018219353]
  We introduce token simplification via distribution normalization to ease modeling difficulty and improve convergence.<n>We enhance prediction by incorporating Gaussian noise injection during training to mitigate exposure bias.<n>This work paves the way for a more unified architecture across visual understanding and generative modeling.
  arXiv  Detail & Related papers  (2026-04-02T02:39:28Z)
• Improving Reconstruction of Representation Autoencoder [52.817427902597416]
  We propose LV-RAE, a representation autoencoder that augments semantic features with missing low-level information.<n>Our experiments demonstrate that LV-RAE significantly improves reconstruction fidelity while preserving the semantic abstraction.
  arXiv  Detail & Related papers  (2026-02-09T13:12:35Z)
• DINO-SAE: DINO Spherical Autoencoder for High-Fidelity Image Reconstruction and Generation [47.409626500688866]
  We present the DINO Spherical Autoencoder (DINO-SAE), a framework that bridges semantic representation and pixel-level reconstruction.<n>Our approach achieves state-of-the-art reconstruction quality, reaching 0.37 rFID and 26.2 dB PSNR, while maintaining strong semantic alignment to the pretrained VFM.
  arXiv  Detail & Related papers  (2026-01-30T12:25:34Z)
• iFSQ: Improving FSQ for Image Generation with 1 Line of Code [40.61338660155903]
  We show how to replace the activation function in FSQ with a distribution-matching mapping to enforce a uniform prior.<n>This simple strategy requires just one line of code yet mathematically guarantees both optimal bin utilization and reconstruction precision.<n>We extend our analysis by adapting Representation Alignment (REPA) to AR models, yielding LlamaGen-REPA.
  arXiv  Detail & Related papers  (2026-01-23T19:00:35Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• Epsilon-VAE: Denoising as Visual Decoding [61.29255979767292]
  We propose denoising as decoding, shifting from single-step reconstruction to iterative refinement.<n>Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image.<n>By adopting iterative reconstruction through diffusion, our autoencoder, namely Epsilon-VAE, achieves high reconstruction quality.
  arXiv  Detail & Related papers  (2024-10-05T08:27:53Z)
• VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations [25.88881764546414]
  VQ-NeRF is an efficient pipeline for enhancing implicit neural representations via vector quantization. We present an innovative multi-scale NeRF sampling scheme that concurrently optimize the NeRF model at both compressed and original scales. We incorporate a semantic loss function to improve the geometric fidelity and semantic coherence of our 3D reconstructions.
  arXiv  Detail & Related papers  (2023-10-23T01:41:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.