Prompt-Based Exemplar Super-Compression and Regeneration for Class-Incremental Learning

• URL: http://arxiv.org/abs/2311.18266v2
• Date: Sat, 23 Nov 2024 07:14:16 GMT
• Title: Prompt-Based Exemplar Super-Compression and Regeneration for Class-Incremental Learning
• Authors: Ruxiao Duan, Yaoyao Liu, Jieneng Chen, Adam Kortylewski, Alan Yuille,
• Abstract summary: We introduce PESCR, a novel approach that substantially increases the quantity and enhances the diversity of exemplars. Images are compressed into visual and textual prompts, which are saved instead of the original images. In subsequent phases, diverse exemplars are regenerated by the diffusion model.
• Score: 21.136513495039242
• License:
• Abstract: Replay-based methods in class-incremental learning~(CIL) have attained remarkable success. Despite their effectiveness, the inherent memory restriction results in saving a limited number of exemplars with poor diversity. In this paper, we introduce PESCR, a novel approach that substantially increases the quantity and enhances the diversity of exemplars based on a pre-trained general-purpose diffusion model, without fine-tuning it on target datasets or storing it in the memory buffer. Images are compressed into visual and textual prompts, which are saved instead of the original images, decreasing memory consumption by a factor of 24. In subsequent phases, diverse exemplars are regenerated by the diffusion model. We further propose partial compression and diffusion-based data augmentation to minimize the domain gap between generated exemplars and real images. Comprehensive experiments demonstrate that PESCR significantly improves CIL performance across multiple benchmarks, e.g., 3.2% above the previous state-of-the-art on ImageNet-100.

Related papers

• Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models [51.3915762595891]
  This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net.
  arXiv  Detail & Related papers  (2024-11-02T08:42:48Z)
• Effective Diffusion Transformer Architecture for Image Super-Resolution [63.254644431016345]
  We design an effective diffusion transformer for image super-resolution (DiT-SR) In practice, DiT-SR leverages an overall U-shaped architecture, and adopts a uniform isotropic design for all the transformer blocks. We analyze the limitation of the widely used AdaLN, and present a frequency-adaptive time-step conditioning module.
  arXiv  Detail & Related papers  (2024-09-29T07:14:16Z)
• Probing Image Compression For Class-Incremental Learning [8.711266563753846]
  Continual machine learning (ML) systems rely on storing representative samples, also known as exemplars, within a limited memory constraint to maintain the performance on previously learned data. In this paper, we explore the use of image compression as a strategy to enhance the buffer's capacity, thereby increasing exemplar diversity. We introduce a new framework to incorporate image compression for continual ML including a pre-processing data compression step and an efficient compression rate/algorithm selection method.
  arXiv  Detail & Related papers  (2024-03-10T18:58:14Z)
• Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement [7.9911486976035215]
  We introduce a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. We leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models.
  arXiv  Detail & Related papers  (2023-09-13T18:43:13Z)
• MOFA: A Model Simplification Roadmap for Image Restoration on Mobile Devices [17.54747506334433]
  We propose a roadmap that can be applied to further accelerate image restoration models prior to deployment. Our approach decreases runtime by up to 13% and reduces the number of parameters by up to 23%, while increasing PSNR and SSIM.
  arXiv  Detail & Related papers  (2023-08-24T01:29:15Z)
• LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement [118.83316133601319]
  Current deep learning methods for low-light image enhancement (LLIE) typically rely on pixel-wise mapping learned from paired data. We propose a degradation-aware learning scheme for LLIE using diffusion models, which effectively integrates degradation and image priors into the diffusion process.
  arXiv  Detail & Related papers  (2023-07-27T07:22:51Z)
• Beyond Learned Metadata-based Raw Image Reconstruction [86.1667769209103]
  Raw images have distinct advantages over sRGB images, e.g., linearity and fine-grained quantization levels. They are not widely adopted by general users due to their substantial storage requirements. We propose a novel framework that learns a compact representation in the latent space, serving as metadata.
  arXiv  Detail & Related papers  (2023-06-21T06:59:07Z)
• Multimodal Data Augmentation for Image Captioning using Diffusion Models [12.221685807426264]
  We propose a data augmentation method, leveraging a text-to-image model called Stable Diffusion, to expand the training set. Experiments on the MS COCO dataset demonstrate the advantages of our approach over several benchmark methods. Further improvement regarding the training efficiency and effectiveness can be obtained after intentionally filtering the generated data.
  arXiv  Detail & Related papers  (2023-05-03T01:57:33Z)
• Effective Data Augmentation With Diffusion Models [65.09758931804478]
  We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models. Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.
  arXiv  Detail & Related papers  (2023-02-07T20:42:28Z)
• Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning [73.24988226158497]
  We consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL) We propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation. Our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks.
  arXiv  Detail & Related papers  (2021-06-17T17:56:08Z)
• Improving Inference for Neural Image Compression [31.999462074510305]
  State-of-the-art methods build on hierarchical variational autoencoders to predict a compressible latent representation of each data point. We identify three approximation gaps which limit performance in the conventional approach. We propose remedies for each of these three limitations based on ideas related to iterative inference.
  arXiv  Detail & Related papers  (2020-06-07T19:26:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.