High Fidelity Image Synthesis With Deep VAEs In Latent Space

• URL: http://arxiv.org/abs/2303.13714v1
• Date: Thu, 23 Mar 2023 23:45:19 GMT
• Title: High Fidelity Image Synthesis With Deep VAEs In Latent Space
• Abstract summary: We present fast, realistic image generation on high-resolution, multimodal datasets using hierarchical variational autoencoders (VAEs) In this two-stage setup, the autoencoder compresses the image into its semantic features, which are then modeled with a deep VAE. We demonstrate the effectiveness of our two-stage approach, achieving a FID of 9.34 on the ImageNet-256 dataset which is comparable to BigGAN.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present fast, realistic image generation on high-resolution, multimodal datasets using hierarchical variational autoencoders (VAEs) trained on a deterministic autoencoder's latent space. In this two-stage setup, the autoencoder compresses the image into its semantic features, which are then modeled with a deep VAE. With this method, the VAE avoids modeling the fine-grained details that constitute the majority of the image's code length, allowing it to focus on learning its structural components. We demonstrate the effectiveness of our two-stage approach, achieving a FID of 9.34 on the ImageNet-256 dataset which is comparable to BigGAN. We make our implementation available online.

Related papers

• Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation [66.53544128707817]
  Cheers is a unified multimodal model that decouples patch-level details from semantic representations.<n> Cheers matches or surpasses advanced UMMs in both visual understanding and generation.
  arXiv  Detail & Related papers  (2026-03-13T08:55:27Z)
• Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders [74.72147962028265]
  Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet.<n>We investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation.
  arXiv  Detail & Related papers  (2026-01-22T18:58:16Z)
• Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving [54.85072592658933]
  We present Flex, an efficient and effective scene encoder that addresses the computational bottleneck of processing high-volume multi-camera data in autonomous driving.<n>By design, our approach is geometry-agnostic, learning a compact scene representation directly from data without relying on the explicit 3D inductive biases.<n>Our findings challenge the prevailing assumption that 3D priors are necessary, demonstrating that a data-driven, joint encoding strategy offers a more scalable, efficient and effective path for future autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• VUGEN: Visual Understanding priors for GENeration [18.840804846528865]
  VUGEN is a novel framework that explicitly leverages VLM's pretrained visual understanding priors for efficient and high-quality image generation.<n>Our approach first transforms the high-dimensional latent space of the VLM's native vision encoder into a lower-dimensional, tractable distribution.<n>A dedicated pixel decoder maps these generated latents back to the image space.
  arXiv  Detail & Related papers  (2025-10-08T00:04:47Z)
• Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models [37.59115132356727]
  We propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation.<n>On ImageNet 256$times$256, our tokenizer accelerates the convergence of diffusion models, reaching a gFID of 1.90 within just 64 epochs.<n>Our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.
  arXiv  Detail & Related papers  (2025-09-29T17:57:39Z)
• One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models [65.96186414865747]
  Text-to-Image (T2I) diffusion models face a trade-off between inference speed and image quality.<n>We introduce the first Time-independent Unified TiUE for the student model UNet architecture.<n>Using a one-pass scheme, TiUE shares encoder features across multiple decoder time steps, enabling parallel sampling.
  arXiv  Detail & Related papers  (2025-05-28T04:23:22Z)
• Multimodal Autoregressive Pre-training of Large Vision Encoders [85.39154488397931]
  We present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and classification.
  arXiv  Detail & Related papers  (2024-11-21T18:31:25Z)
• Pixel-Aligned Multi-View Generation with Depth Guided Decoder [86.1813201212539]
  We propose a novel method for pixel-level image-to-multi-view generation. Unlike prior work, we incorporate attention layers across multi-view images in the VAE decoder of a latent video diffusion model. Our model enables better pixel alignment across multi-view images.
  arXiv  Detail & Related papers  (2024-08-26T04:56:41Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [52.598772767324036]
  We introduce Image-GS, a content-adaptive image representation based on 2D Gaussians radiance.<n>It supports hardware-friendly rapid access for real-time usage, requiring only 0.3K MACs to decode a pixel.<n>We demonstrate its versatility with several applications, including texture compression, semantics-aware compression, and joint image compression and restoration.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• CE-VAE: Capsule Enhanced Variational AutoEncoder for Underwater Image Enhancement [8.16306466526838]
  Unmanned underwater image analysis for marine monitoring faces two key challenges: degraded image quality and hardware storage constraints. We introduce the Capsule Enhanced Variational AutoEncoder (CE-VAE), a novel architecture designed to efficiently compress and enhance degraded underwater images. CE-VAE achieves state-of-the-art performance in underwater image enhancement on six benchmark datasets.
  arXiv  Detail & Related papers  (2024-06-03T13:04:42Z)
• Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection [13.840950434728533]
  State-of-the-art Synthetic Image Detection (SID) research has led to strong evidence on the advantages of feature extraction from foundation models. We leverage the image representations extracted by intermediate Transformer blocks of CLIP's image-encoder via a lightweight network. Our method is compared against the state-of-the-art by evaluating it on 20 test datasets and exhibits an average +10.6% absolute performance improvement.
  arXiv  Detail & Related papers  (2024-02-29T12:18:43Z)
• I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models [54.99771394322512]
  Video synthesis has recently made remarkable strides benefiting from the rapid development of diffusion models. It still challenges encounters in terms of semantic accuracy, clarity, and continuity-temporal continuity. We propose a cascaded I2VGen-XL approach that enhances model performance by decoupling these two factors. I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos.
  arXiv  Detail & Related papers  (2023-11-07T17:16:06Z)
• Soft-IntroVAE for Continuous Latent space Image Super-Resolution [12.344557879284219]
  We propose a Soft-introVAE for continuous latent space image super-resolution (SVAE-SR) Inspired by Variational AutoEncoder, we propose a Soft-introVAE for continuous latent space image super-resolution (SVAE-SR)
  arXiv  Detail & Related papers  (2023-07-18T06:54:42Z)
• A Model-data-driven Network Embedding Multidimensional Features for Tomographic SAR Imaging [5.489791364472879]
  We propose a new model-data-driven network to achieve tomoSAR imaging based on multi-dimensional features. We add two 2D processing modules, both convolutional encoder-decoder structures, to enhance multi-dimensional features of the imaging scene effectively. Compared with the conventional CS-based FISTA method and DL-based gamma-Net method, the result of our proposed method has better performance on completeness while having decent imaging accuracy.
  arXiv  Detail & Related papers  (2022-11-28T02:01:43Z)
• Wider and Higher: Intensive Integration and Global Foreground Perception for Image Matting [44.51635913732913]
  This paper reviews recent deep-learning-based matting research and conceives our wider and higher motivation for image matting. Image matting is essentially a pixel-wise regression, and the ideal situation is to perceive the maximum opacity from the input image. We propose an Intensive Integration and Global Foreground Perception network (I2GFP) to integrate wider and higher feature streams.
  arXiv  Detail & Related papers  (2022-10-13T11:34:46Z)
• Learning Enriched Features for Fast Image Restoration and Enhancement [166.17296369600774]
  This paper presents a holistic goal of maintaining spatially-precise high-resolution representations through the entire network. We learn an enriched set of features that combines contextual information from multiple scales, while simultaneously preserving the high-resolution spatial details. Our approach achieves state-of-the-art results for a variety of image processing tasks, including defocus deblurring, image denoising, super-resolution, and image enhancement.
  arXiv  Detail & Related papers  (2022-04-19T17:59:45Z)
• Neural Data-Dependent Transform for Learned Image Compression [72.86505042102155]
  We build a neural data-dependent transform and introduce a continuous online mode decision mechanism to jointly optimize the coding efficiency for each individual image. The experimental results show the effectiveness of the proposed neural-syntax design and the continuous online mode decision mechanism.
  arXiv  Detail & Related papers  (2022-03-09T14:56:48Z)
• Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling [79.15521784128102]
  We introduce a novel neural network for building image generators (decoders) and apply it to variational autoencoders (VAEs) In our spatial dependency networks (SDNs), feature maps at each level of a deep neural net are computed in a spatially coherent way. We show that augmenting the decoder of a hierarchical VAE by spatial dependency layers considerably improves density estimation.
  arXiv  Detail & Related papers  (2021-03-16T07:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.