Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration
- URL: http://arxiv.org/abs/2306.06513v2
- Date: Tue, 13 Jun 2023 05:06:32 GMT
- Title: Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration
- Authors: Kechun Liu, Yitong Jiang, Inchang Choi, Jinwei Gu
- Abstract summary: We propose AdaCode for learning image-adaptive codebooks for class-agnostic image restoration.
AdaCode is a more flexible and expressive discrete generative prior than previous work.
- Score: 13.718779033187786
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent work on discrete generative priors, in the form of codebooks, has
shown exciting performance for image reconstruction and restoration, as the
discrete prior space spanned by the codebooks increases the robustness against
diverse image degradations. Nevertheless, these methods require separate
training of codebooks for different image categories, which limits their use to
specific image categories only (e.g. face, architecture, etc.), and fail to
handle arbitrary natural images. In this paper, we propose AdaCode for learning
image-adaptive codebooks for class-agnostic image restoration. Instead of
learning a single codebook for each image category, we learn a set of basis
codebooks. For a given input image, AdaCode learns a weight map with which we
compute a weighted combination of these basis codebooks for adaptive image
restoration. Intuitively, AdaCode is a more flexible and expressive discrete
generative prior than previous work. Experimental results demonstrate that
AdaCode achieves state-of-the-art performance on image reconstruction and
restoration tasks, including image super-resolution and inpainting.
Related papers
- Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling [15.132926378740882]
We propose a novel codebook transfer framework with part-of-speech, called VQCT, which aims to transfer a well-trained codebook from pretrained language models to VQIM.
Experimental results on four datasets show that our VQCT method achieves superior VQIM performance over previous state-of-the-art methods.
arXiv Detail & Related papers (2024-03-15T07:24:13Z) - Masked Autoencoders are Efficient Class Incremental Learners [64.90846899051164]
Class Incremental Learning (CIL) aims to sequentially learn new classes while avoiding catastrophic forgetting of previous knowledge.
We propose to use Masked Autoencoders (MAEs) as efficient learners for CIL.
arXiv Detail & Related papers (2023-08-24T02:49:30Z) - DiffusePast: Diffusion-based Generative Replay for Class Incremental
Semantic Segmentation [73.54038780856554]
Class Incremental Semantic (CISS) extends the traditional segmentation task by incrementally learning newly added classes.
Previous work has introduced generative replay, which involves replaying old class samples generated from a pre-trained GAN.
We propose DiffusePast, a novel framework featuring a diffusion-based generative replay module that generates semantically accurate images with more reliable masks guided by different instructions.
arXiv Detail & Related papers (2023-08-02T13:13:18Z) - Not All Image Regions Matter: Masked Vector Quantization for
Autoregressive Image Generation [78.13793505707952]
Existing autoregressive models follow the two-stage generation paradigm that first learns a codebook in the latent space for image reconstruction and then completes the image generation autoregressively based on the learned codebook.
We propose a novel two-stage framework, which consists of Masked Quantization VAE (MQ-VAE) Stack model from modeling redundancy.
arXiv Detail & Related papers (2023-05-23T02:15:53Z) - Towards Accurate Image Coding: Improved Autoregressive Image Generation
with Dynamic Vector Quantization [73.52943587514386]
Existing vector quantization (VQ) based autoregressive models follow a two-stage generation paradigm.
We propose a novel two-stage framework: (1) Dynamic-Quantization VAE (DQ-VAE) which encodes image regions into variable-length codes based their information densities for accurate representation.
arXiv Detail & Related papers (2023-05-19T14:56:05Z) - Reinforcing Generated Images via Meta-learning for One-Shot Fine-Grained
Visual Recognition [36.02360322125622]
We propose a meta-learning framework to combine generated images with original images, so that the resulting "hybrid" training images improve one-shot learning.
Our experiments demonstrate consistent improvement over baselines on one-shot fine-grained image classification benchmarks.
arXiv Detail & Related papers (2022-04-22T13:11:05Z) - EEC: Learning to Encode and Regenerate Images for Continual Learning [9.89901717499058]
We train autoencoders with Neural Style Transfer to encode and store images.
reconstructed images from encoded episodes are replayed in order to avoid catastrophic forgetting.
Our approach increases classification accuracy by 13-17% over state-of-the-art methods on benchmark datasets.
arXiv Detail & Related papers (2021-01-13T06:43:10Z) - Storing Encoded Episodes as Concepts for Continual Learning [22.387008072671005]
Two main challenges faced by continual learning approaches are catastrophic forgetting and memory limitations on the storage of data.
We propose a cognitively-inspired approach which trains autoencoders with Neural Style Transfer to encode and store images.
Our approach increases classification accuracy by 13-17% over state-of-the-art methods on benchmark datasets, while requiring 78% less storage space.
arXiv Detail & Related papers (2020-06-26T04:15:56Z) - Memory-Efficient Incremental Learning Through Feature Adaptation [71.1449769528535]
We introduce an approach for incremental learning that preserves feature descriptors of training images from previously learned classes.
Keeping the much lower-dimensional feature embeddings of images reduces the memory footprint significantly.
Experimental results show that our method achieves state-of-the-art classification accuracy in incremental learning benchmarks.
arXiv Detail & Related papers (2020-04-01T21:16:05Z) - Exploiting Deep Generative Prior for Versatile Image Restoration and
Manipulation [181.08127307338654]
This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images.
The deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images.
arXiv Detail & Related papers (2020-03-30T17:45:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.