Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders

• URL: http://arxiv.org/abs/2603.14536v1
• Date: Sun, 15 Mar 2026 18:20:50 GMT
• Title: Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders
• Abstract summary: Variational Autoencoder (VAE) encoders play a critical role in modern generative models.<n>We report a counter-intuitive phenomenon in VAE encoder distillation: a compact encoder distilled only at low resolutions exhibits poor reconstruction performance at its native resolution.<n>Despite never being trained beyond $2562$ resolution, the distilled encoder generalizes effectively to $5122$ resolution inputs.
• Score: 6.6001209552146305
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Variational Autoencoder (VAE) encoders play a critical role in modern generative models, yet their computational cost often motivates the use of knowledge distillation or quantification to obtain compact alternatives. Existing studies typically believe that the model work better on the samples closed to their training data distribution than unseen data distribution. In this work, we report a counter-intuitive phenomenon in VAE encoder distillation: a compact encoder distilled only at low resolutions exhibits poor reconstruction performance at its native resolution, but achieves dramatically improved results when evaluated at higher, unseen input resolutions. Despite never being trained beyond $256^2$ resolution, the distilled encoder generalizes effectively to $512^2$ resolution inputs, partially inheriting the teacher model's resolution preference.We further analyze latent distributions across resolutions and find that higher-resolution inputs produce latent representations more closely aligned with the teacher's manifold. Through extensive experiments on ImageNet-256, we show that simple resolution remapping-upsampling inputs before encoding and downsampling reconstructions for evaluation-leads to substantial gains across PSNR, MSE, SSIM, LPIPS, and rFID metrics. These findings suggest that VAE encoder distillation learns resolution-consistent latent manifolds rather than resolution-specific pixel mappings. This also means that the high training cost on memory, time and high-resolution datasets are not necessary conditions for distilling a VAE with high-resolution image reconstruction capabilities. On low resolution datasets, the distillation model still could learn the detailed knowledge of the teacher model in high-resolution image reconstruction.

Related papers

• Quantum Down Sampling Filter for Variational Auto-encoder [0.504868948270058]
  Variational autoencoders (VAEs) are fundamental for generative modeling and image reconstruction.<n>This study introduces a hybrid model, quantum variational autoencoder (Q-VAE)<n>Q-VAE integrates quantum encoding within the encoder while utilizing fully connected layers to extract meaningful representations.
  arXiv  Detail & Related papers  (2025-01-09T11:08:55Z)
• Boosting Latent Diffusion with Perceptual Objectives [29.107038084215514]
  Latent diffusion models (LDMs) power state-of-the-art high-resolution generative image models.<n>We propose to leverage the internal features of the decoder to define a latent perceptual loss (LPL)<n>This loss encourages the models to create sharper and more realistic images.
  arXiv  Detail & Related papers  (2024-11-06T16:28:21Z)
• One Step Diffusion-based Super-Resolution with Time-Aware Distillation [60.262651082672235]
  Diffusion-based image super-resolution (SR) methods have shown promise in reconstructing high-resolution images with fine details from low-resolution counterparts. Recent techniques have been devised to enhance the sampling efficiency of diffusion-based SR models via knowledge distillation. We propose a time-aware diffusion distillation method, named TAD-SR, to accomplish effective and efficient image super-resolution.
  arXiv  Detail & Related papers  (2024-08-14T11:47:22Z)
• Pairwise Distance Distillation for Unsupervised Real-World Image Super-Resolution [38.79439380482431]
  Real-world super-resolution (RWSR) faces unknown degradations in the low-resolution inputs, all the while lacking paired training data. Existing methods approach this problem by learning blind general models through complex synthetic augmentations on training inputs. We introduce a novel pairwise distance distillation framework to address the unsupervised RWSR for a targeted real-world degradation.
  arXiv  Detail & Related papers  (2024-07-10T01:46:40Z)
• Learning to Be a Transformer to Pinpoint Anomalies [12.442574943138794]
  Recent Industrial Anomaly Detection and (IADS) methods process low-resolution images, e.g., 224x224 pixels, obtained by downsampling the original input images.<n>We propose a novel Teacher--Student paradigm to leverage strong pre-trained features while processing high-resolution input images very efficiently.<n>Our method can spot anomalies from high-resolution images and runs way faster than competitors.
  arXiv  Detail & Related papers  (2024-07-04T17:59:26Z)
• ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models [126.35334860896373]
  We investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. Existing works for higher-resolution generation, such as attention-based and joint-diffusion approaches, cannot well address these issues. We propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference.
  arXiv  Detail & Related papers  (2023-10-11T17:52:39Z)
• ResFormer: Scaling ViTs with Multi-Resolution Training [100.01406895070693]
  We introduce ResFormer, a framework for improved performance on a wide spectrum of, mostly unseen, testing resolutions. In particular, ResFormer operates on replicated images of different resolutions and enforces a scale consistency loss to engage interactive information across different scales. We demonstrate, moreover, ResFormer is flexible and can be easily extended to semantic segmentation, object detection and video action recognition.
  arXiv  Detail & Related papers  (2022-12-01T18:57:20Z)
• Any-resolution Training for High-resolution Image Synthesis [55.19874755679901]
  Generative models operate at fixed resolution, even though natural images come in a variety of sizes. We argue that every pixel matters and create datasets with variable-size images, collected at their native resolutions. We introduce continuous-scale training, a process that samples patches at random scales to train a new generator with variable output resolutions.
  arXiv  Detail & Related papers  (2022-04-14T17:59:31Z)
• Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition [124.80263629921498]
  We propose Pixel Distillation that extends knowledge distillation into the input level while simultaneously breaking architecture constraints. Such a scheme can achieve flexible cost control for deployment, as it allows the system to adjust both network architecture and image quality according to the overall requirement of resources.
  arXiv  Detail & Related papers  (2021-12-17T14:31:40Z)
• Multi-Scale Aligned Distillation for Low-Resolution Detection [68.96325141432078]
  This paper focuses on boosting the performance of low-resolution models by distilling knowledge from a high- or multi-resolution model. On several instance-level detection tasks and datasets, the low-resolution models trained via our approach perform competitively with high-resolution models trained via conventional multi-scale training.
  arXiv  Detail & Related papers  (2021-09-14T12:53:35Z)
• Learning When and Where to Zoom with Deep Reinforcement Learning [101.79271767464947]
  We propose a reinforcement learning approach to identify when and where to use/acquire high resolution data conditioned on paired, cheap, low resolution images. We conduct experiments on CIFAR10, CIFAR100, ImageNet and fMoW datasets where we use significantly less high resolution data while maintaining similar accuracy to models which use full high resolution images.
  arXiv  Detail & Related papers  (2020-03-01T07:16:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.