Incremental Neural Implicit Representation with Uncertainty-Filtered
Knowledge Distillation
- URL: http://arxiv.org/abs/2212.10950v2
- Date: Sun, 4 Jun 2023 04:16:42 GMT
- Title: Incremental Neural Implicit Representation with Uncertainty-Filtered
Knowledge Distillation
- Authors: Mengqi Guo, Chen Li, Hanlin Chen, Gim Hee Lee
- Abstract summary: Recent neural implicit representations (NIRs) have achieved great success in the tasks of 3D reconstruction and novel view synthesis.
They suffer from the catastrophic forgetting problem when continuously learning from streaming data without revisiting the previously seen data.
We design a student-teacher framework to mitigate the catastrophic forgetting problem.
- Score: 59.95692054302568
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent neural implicit representations (NIRs) have achieved great success in
the tasks of 3D reconstruction and novel view synthesis. However, they suffer
from the catastrophic forgetting problem when continuously learning from
streaming data without revisiting the previously seen data. This limitation
prohibits the application of existing NIRs to scenarios where images come in
sequentially. In view of this, we explore the task of incremental learning for
NIRs in this work. We design a student-teacher framework to mitigate the
catastrophic forgetting problem. Specifically, we iterate the process of using
the student as the teacher at the end of each time step and let the teacher
guide the training of the student in the next step. As a result, the student
network is able to learn new information from the streaming data and retain old
knowledge from the teacher network simultaneously. Although intuitive, naively
applying the student-teacher pipeline does not work well in our task. Not all
information from the teacher network is helpful since it is only trained with
the old data. To alleviate this problem, we further introduce a random inquirer
and an uncertainty-based filter to filter useful information. Our proposed
method is general and thus can be adapted to different implicit representations
such as neural radiance field (NeRF) and neural SDF. Extensive experimental
results for both 3D reconstruction and novel view synthesis demonstrate the
effectiveness of our approach compared to different baselines.
Related papers
- Negotiated Representations to Prevent Forgetting in Machine Learning
Applications [0.0]
Catastrophic forgetting is a significant challenge in the field of machine learning.
We propose a novel method for preventing catastrophic forgetting in machine learning applications.
arXiv Detail & Related papers (2023-11-30T22:43:50Z) - Hierarchical Supervision and Shuffle Data Augmentation for 3D
Semi-Supervised Object Detection [90.32180043449263]
State-of-the-art 3D object detectors are usually trained on large-scale datasets with high-quality 3D annotations.
A natural remedy is to adopt semi-supervised learning (SSL) by leveraging a limited amount of labeled samples and abundant unlabeled samples.
This paper introduces a novel approach of Hierarchical Supervision and Shuffle Data Augmentation (HSSDA), which is a simple yet effective teacher-student framework.
arXiv Detail & Related papers (2023-04-04T02:09:32Z) - Improved knowledge distillation by utilizing backward pass knowledge in
neural networks [17.437510399431606]
Knowledge distillation (KD) is one of the prominent techniques for model compression.
In this work, we generate new auxiliary training samples based on extracting knowledge from the backward pass of the teacher.
We show how this technique can be used successfully in applications of natural language processing (NLP) and language understanding.
arXiv Detail & Related papers (2023-01-27T22:07:38Z) - Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student
Settings and its Superiority to Kernel Methods [58.44819696433327]
We investigate the risk of two-layer ReLU neural networks in a teacher regression model.
We find that the student network provably outperforms any solution methods.
arXiv Detail & Related papers (2022-05-30T02:51:36Z) - Increasing Depth of Neural Networks for Life-long Learning [2.0305676256390934]
We propose a novel method for continual learning based on the increasing depth of neural networks.
This work explores whether extending neural network depth may be beneficial in a life-long learning setting.
arXiv Detail & Related papers (2022-02-22T11:21:41Z) - Learning Bayesian Sparse Networks with Full Experience Replay for
Continual Learning [54.7584721943286]
Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered.
Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal.
We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
arXiv Detail & Related papers (2022-02-21T13:25:03Z) - Neural Networks Are More Productive Teachers Than Human Raters: Active
Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model [57.41841346459995]
We study how to train a student deep neural network for visual recognition by distilling knowledge from a blackbox teacher model in a data-efficient manner.
We propose an approach that blends mixup and active learning.
arXiv Detail & Related papers (2020-03-31T05:44:55Z) - Overcoming Catastrophic Forgetting in Graph Neural Networks with
Experience Replay [16.913443823792022]
Graph Neural Networks (GNNs) have recently received significant research attention due to their superior performance on a variety of graph-related learning tasks.
In this work, we investigate the question can GNNs be applied to continuously learning tasks?
arXiv Detail & Related papers (2020-03-22T14:29:53Z) - Curriculum By Smoothing [52.08553521577014]
Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation.
We propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters.
As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data.
arXiv Detail & Related papers (2020-03-03T07:27:44Z) - Adversarial Incremental Learning [0.0]
Deep learning can forget previously learned information upon learning new tasks where previous data is not available.
We propose an adversarial discriminator based method that does not make use of old data at all while training on new tasks.
We are able to outperform other state-of-the-art methods on CIFAR-100, SVHN, and MNIST datasets.
arXiv Detail & Related papers (2020-01-30T02:25:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.