VIVAT: Virtuous Improving VAE Training through Artifact Mitigation

• URL: http://arxiv.org/abs/2506.07863v1
• Date: Mon, 09 Jun 2025 15:27:03 GMT
• Title: VIVAT: Virtuous Improving VAE Training through Artifact Mitigation
• Authors: Lev Novitskiy, Viacheslav Vasilev, Maria Kovaleva, Vladimir Arkhipkin, Denis Dimitrov,
• Abstract summary: This paper introduces VIVAT, a systematic approach to mitigating common artifacts in KL-VAE training without requiring radical architectural changes.<n>We present a detailed taxonomy of five prevalent artifacts - color shift, grid patterns, blur, corner and droplet artifacts - and analyze their root causes.
• Score: 4.295130967329365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Variational Autoencoders (VAEs) remain a cornerstone of generative computer vision, yet their training is often plagued by artifacts that degrade reconstruction and generation quality. This paper introduces VIVAT, a systematic approach to mitigating common artifacts in KL-VAE training without requiring radical architectural changes. We present a detailed taxonomy of five prevalent artifacts - color shift, grid patterns, blur, corner and droplet artifacts - and analyze their root causes. Through straightforward modifications, including adjustments to loss weights, padding strategies, and the integration of Spatially Conditional Normalization, we demonstrate significant improvements in VAE performance. Our method achieves state-of-the-art results in image reconstruction metrics (PSNR and SSIM) across multiple benchmarks and enhances text-to-image generation quality, as evidenced by superior CLIP scores. By preserving the simplicity of the KL-VAE framework while addressing its practical challenges, VIVAT offers actionable insights for researchers and practitioners aiming to optimize VAE training.

Related papers

• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• VACoT: Rethinking Visual Data Augmentation with VLMs [47.68285534481867]
  Visual Augmentation Chain-of-Thought (VACoT) is a framework that dynamically invokes image augmentations during model inference.<n>VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios.<n>We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses.
  arXiv  Detail & Related papers  (2025-12-02T03:11:32Z)
• EyeSim-VQA: A Free-Energy-Guided Eye Simulation Framework for Video Quality Assessment [68.77813885751308]
  EyeSimVQA is a novel VQA framework that incorporates free-energy-based self-repair.<n>We show EyeSimVQA achieves competitive or superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-13T08:00:54Z)
• RestoreVAR: Visual Autoregressive Generation for All-in-One Image Restoration [51.77917733024544]
  latent diffusion models (LDMs) have improved the perceptual quality of All-in-One image Restoration (AiOR) methods.<n>LDMs suffer from slow inference due to their iterative denoising process, rendering them impractical for time-sensitive applications.<n>Visual autoregressive modeling ( VAR) performs scale-space autoregression and achieves comparable performance to that of state-of-the-art diffusion transformers.
  arXiv  Detail & Related papers  (2025-05-23T15:52:26Z)
• VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation [11.529598741483076]
  A visual tokenizer (VT) maps continuous pixel inputs to discrete token sequences.<n>Current discrete VTs fall significantly behind continuous variational autoencoders (VAEs), leading to degraded image reconstructions and poor preservation of details and text.<n>Existing benchmarks focus on end-to-end generation quality, without isolating VT performance.<n>We introduce VTBench, a comprehensive benchmark that systematically evaluates VTs across three core tasks: Image Reconstruction, Detail Preservation, and Text Preservation.
  arXiv  Detail & Related papers  (2025-05-19T17:59:01Z)
• Enhancing Variational Autoencoders with Smooth Robust Latent Encoding [54.74721202894622]
  Variational Autoencoders (VAEs) have played a key role in scaling up diffusion-based generative models.<n>We introduce Smooth Robust Latent VAE, a novel adversarial training framework that boosts both generation quality and robustness.<n>Experiments show that SRL-VAE improves both generation quality, in image reconstruction and text-guided image editing, and robustness, against Nightshade attacks and image editing attacks.
  arXiv  Detail & Related papers  (2025-04-24T03:17:57Z)
• Progressive Fine-to-Coarse Reconstruction for Accurate Low-Bit Post-Training Quantization in Vision Transformers [13.316135182889296]
  Post-Training Quantization (PTQ) has been widely adopted for compressing Vision Transformers (ViTs)<n>When quantized into low-bit representations, there is often a significant performance drop compared to their full-precision counterparts.<n>We propose a Progressive Fine-to-Coarse Reconstruction (PFCR) method for accurate PTQ, which significantly improves the performance of low-bit quantized vision transformers.
  arXiv  Detail & Related papers  (2024-12-19T08:38:59Z)
• HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models [96.76995840807615]
  HiRes-LLaVA is a novel framework designed to process any size of high-resolution input without altering the original contextual and geometric information. HiRes-LLaVA comprises two innovative components: (i) a SliceRestore adapter that reconstructs sliced patches into their original form, efficiently extracting both global and local features via down-up-sampling and convolution layers, and (ii) a Self-Mining Sampler to compress the vision tokens based on themselves.
  arXiv  Detail & Related papers  (2024-07-11T17:42:17Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Boosting Image Restoration via Priors from Pre-trained Models [54.83907596825985]
  We learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.
  arXiv  Detail & Related papers  (2024-03-11T15:11:57Z)
• Attention-Guided Masked Autoencoders For Learning Image Representations [16.257915216763692]
  Masked autoencoders (MAEs) have established themselves as a powerful method for unsupervised pre-training for computer vision tasks. We propose to inform the reconstruction process through an attention-guided loss function. Our evaluations show that our pre-trained models learn better latent representations than the vanilla MAE.
  arXiv  Detail & Related papers  (2024-02-23T08:11:25Z)
• Image Reconstruction using Enhanced Vision Transformer [0.08594140167290097]
  We propose a novel image reconstruction framework which can be used for tasks such as image denoising, deblurring or inpainting. The model proposed in this project is based on Vision Transformer (ViT) that takes 2D images as input and outputs embeddings. We incorporate four additional optimization techniques in the framework to improve the model reconstruction capability.
  arXiv  Detail & Related papers  (2023-07-11T02:14:18Z)
• Defending Variational Autoencoders from Adversarial Attacks with MCMC [74.36233246536459]
  Variational autoencoders (VAEs) are deep generative models used in various domains. As previous work has shown, one can easily fool VAEs to produce unexpected latent representations and reconstructions for a visually slightly modified input. Here, we examine several objective functions for adversarial attacks construction, suggest metrics assess the model robustness, and propose a solution.
  arXiv  Detail & Related papers  (2022-03-18T13:25:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.