Continual Learning of Personalized Generative Face Models with Experience Replay

• URL: http://arxiv.org/abs/2412.02627v1
• Date: Tue, 03 Dec 2024 17:56:23 GMT
• Title: Continual Learning of Personalized Generative Face Models with Experience Replay
• Authors: Annie N. Wang, Luchao Qi, Roni Sengupta,
• Abstract summary: We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model.<n>We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face.<n>We propose a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull.
• Score: 1.024113475677323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances, styles, poses, and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However, for long-term deployment of these models with relatively smaller storage, this simple random sampling-based replay technique also forgets past representations. Thus, we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound.

Related papers

• Lifelong Person Re-identification via Privacy-Preserving Data Replay [14.764580534110666]
  Lifelong person re-identification (LReID) aims to incrementally accumulate knowledge across a sequence of tasks under domain shifts.<n>Recent replay-based methods have demonstrated strong effectiveness in LReID by rehearsing past samples stored in an auxiliary memory.<n>We propose to condense information from sequential data into the pixel space in the replay memory, enabling Privacy-Preserving Replay (Pr2R)
  arXiv  Detail & Related papers  (2025-08-03T05:00:19Z)
• LD-RPS: Zero-Shot Unified Image Restoration via Latent Diffusion Recurrent Posterior Sampling [23.886038479359918]
  We propose a dataset-free, and unified approach through recurrent posterior sampling utilizing a pretrained latent diffusion model.<n>Our method incorporates the multimodal understanding model to provide sematic priors for the generative model under a task-blind condition.
  arXiv  Detail & Related papers  (2025-07-01T14:25:09Z)
• Replay Can Provably Increase Forgetting [24.538643224479515]
  A critical challenge for continual learning is forgetting, where the performance on previously learned tasks decreases as new tasks are introduced.<n>One of the commonly used techniques to mitigate forgetting, sample replay, has been shown empirically to reduce forgetting.<n>We show that even in a noiseless setting, forgetting can be non-monotonic with respect to the number of replay samples.
  arXiv  Detail & Related papers  (2025-06-04T18:46:23Z)
• Non-Uniform Memory Sampling in Experience Replay [1.9580473532948401]
  A popular strategy to alleviate catastrophic forgetting is experience replay. Most approaches assume that sampling from this buffer is uniform by default. We generate 50 different non-uniform sampling probability weights for each trial and compare their final accuracy to the uniform sampling baseline.
  arXiv  Detail & Related papers  (2025-02-16T23:04:16Z)
• Redistribute Ensemble Training for Mitigating Memorization in Diffusion Models [31.92526915009259]
  Diffusion models are known for their tremendous ability to generate high-quality samples. Recent methods for memory mitigation have primarily addressed the issue within the context of the text modality. We propose a novel method for diffusion models from the perspective of visual modality, which is more generic and fundamental for mitigating memorization.
  arXiv  Detail & Related papers  (2025-02-13T15:56:44Z)
• Posterior sampling via Langevin dynamics based on generative priors [31.84543941736757]
  Posterior sampling in high-dimensional spaces using generative models holds significant promise for various applications. Existing methods require restarting the entire generative process for each new sample, making the procedure computationally expensive. We propose efficient posterior sampling by simulating Langevin dynamics in the noise space of a pre-trained generative model.
  arXiv  Detail & Related papers  (2024-10-02T22:57:47Z)
• Self-Guided Generation of Minority Samples Using Diffusion Models [57.319845580050924]
  We present a novel approach for generating minority samples that live on low-density regions of a data manifold. Our framework is built upon diffusion models, leveraging the principle of guided sampling. Experiments on benchmark real datasets demonstrate that our approach can greatly improve the capability of creating realistic low-likelihood minority instances.
  arXiv  Detail & Related papers  (2024-07-16T10:03:29Z)
• GUIDE: Guidance-based Incremental Learning with Diffusion Models [3.046689922445082]
  We introduce GUIDE, a novel continual learning approach that directs diffusion models to rehearse samples at risk of being forgotten. Our experimental results show that GUIDE significantly reduces catastrophic forgetting, outperforming conventional random sampling approaches and surpassing recent state-of-the-art methods in continual learning with generative replay.
  arXiv  Detail & Related papers  (2024-03-06T18:47:32Z)
• RANRAC: Robust Neural Scene Representations via Random Ray Consensus [12.161889666145127]
  RANdom RAy Consensus (RANRAC) is an efficient approach to eliminate the effect of inconsistent data. We formulate a fuzzy adaption of the RANSAC paradigm, enabling its application to large scale models. Results indicate significant improvements compared to state-of-the-art robust methods for novel-view synthesis.
  arXiv  Detail & Related papers  (2023-12-15T13:33:09Z)
• BAA-NGP: Bundle-Adjusting Accelerated Neural Graphics Primitives [6.431806897364565]
  Implicit neural representations have become pivotal in robotic perception, enabling robots to comprehend 3D environments from 2D images. We propose a framework called bundle-adjusting accelerated neural graphics primitives (BAA-NGP) Results demonstrate 10 to 20 x speed improvement compared to other bundle-adjusting neural radiance field methods.
  arXiv  Detail & Related papers  (2023-06-07T05:36:45Z)
• Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction [60.52716381465063]
  We introduce Deceptive-NeRF/3DGS to enhance sparse-view reconstruction with only a limited set of input images. Specifically, we propose a deceptive diffusion model turning noisy images rendered from few-view reconstructions into high-quality pseudo-observations. Our system progressively incorporates diffusion-generated pseudo-observations into the training image sets, ultimately densifying the sparse input observations by 5 to 10 times.
  arXiv  Detail & Related papers  (2023-05-24T14:00:32Z)
• ReDi: Efficient Learning-Free Diffusion Inference via Trajectory Retrieval [68.7008281316644]
  ReDi is a learning-free Retrieval-based Diffusion sampling framework. We show that ReDi improves the model inference efficiency by 2x speedup.
  arXiv  Detail & Related papers  (2023-02-05T03:01:28Z)
• LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space [90.74976459491303]
  We introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space. A normalizing flow bridges the two representation spaces and transforms latent samples from one domain to another, allowing us to define a latent likelihood objective. We show that our approach leads to an expressive and effective prior, capturing facial dynamics and subtle expressions better.
  arXiv  Detail & Related papers  (2022-03-15T13:22:57Z)
• RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs [79.00855490550367]
  We show that NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but also conditional models that are extensively pre-trained on large multi-view datasets.
  arXiv  Detail & Related papers  (2021-12-01T18:59:46Z)
• Anytime Sampling for Autoregressive Models via Ordered Autoencoding [88.01906682843618]
  Autoregressive models are widely used for tasks such as image and audio generation. The sampling process of these models does not allow interruptions and cannot adapt to real-time computational resources. We propose a new family of autoregressive models that enables anytime sampling.
  arXiv  Detail & Related papers  (2021-02-23T05:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.