UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit Representation

• URL: http://arxiv.org/abs/2212.10950v3
• Date: Mon, 29 Jul 2024 07:25:37 GMT
• Title: UNIKD: UNcertainty-filtered Incremental Knowledge Distillation for Neural Implicit Representation
• Authors: Mengqi Guo, Chen Li, Hanlin Chen, Gim Hee Lee,
• Abstract summary: Recent neural implicit representations (NIRs) have achieved great success in the tasks of 3D reconstruction and novel view synthesis. They require the images of a scene from different camera views to be available for one-time training. This is expensive especially for scenarios with large-scale scenes and limited data storage. We design a student-teacher framework to mitigate the catastrophic problem.
• Score: 48.49860868061573
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent neural implicit representations (NIRs) have achieved great success in the tasks of 3D reconstruction and novel view synthesis. However, they require the images of a scene from different camera views to be available for one-time training. This is expensive especially for scenarios with large-scale scenes and limited data storage. In view of this, we explore the task of incremental learning for NIRs in this work. We design a student-teacher framework to mitigate the catastrophic forgetting problem. Specifically, we iterate the process of using the student as the teacher at the end of each time step and let the teacher guide the training of the student in the next step. As a result, the student network is able to learn new information from the streaming data and retain old knowledge from the teacher network simultaneously. Although intuitive, naively applying the student-teacher pipeline does not work well in our task. Not all information from the teacher network is helpful since it is only trained with the old data. To alleviate this problem, we further introduce a random inquirer and an uncertainty-based filter to filter useful information. Our proposed method is general and thus can be adapted to different implicit representations such as neural radiance field (NeRF) and neural surface field. Extensive experimental results for both 3D reconstruction and novel view synthesis demonstrate the effectiveness of our approach compared to different baselines.

Related papers

• Negotiated Representations to Prevent Forgetting in Machine Learning Applications [0.0]
  Catastrophic forgetting is a significant challenge in the field of machine learning. We propose a novel method for preventing catastrophic forgetting in machine learning applications.
  arXiv  Detail & Related papers  (2023-11-30T22:43:50Z)
• Hierarchical Supervision and Shuffle Data Augmentation for 3D Semi-Supervised Object Detection [90.32180043449263]
  State-of-the-art 3D object detectors are usually trained on large-scale datasets with high-quality 3D annotations. A natural remedy is to adopt semi-supervised learning (SSL) by leveraging a limited amount of labeled samples and abundant unlabeled samples. This paper introduces a novel approach of Hierarchical Supervision and Shuffle Data Augmentation (HSSDA), which is a simple yet effective teacher-student framework.
  arXiv  Detail & Related papers  (2023-04-04T02:09:32Z)
• Improved knowledge distillation by utilizing backward pass knowledge in neural networks [17.437510399431606]
  Knowledge distillation (KD) is one of the prominent techniques for model compression. In this work, we generate new auxiliary training samples based on extracting knowledge from the backward pass of the teacher. We show how this technique can be used successfully in applications of natural language processing (NLP) and language understanding.
  arXiv  Detail & Related papers  (2023-01-27T22:07:38Z)
• Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods [58.44819696433327]
  We investigate the risk of two-layer ReLU neural networks in a teacher regression model. We find that the student network provably outperforms any solution methods.
  arXiv  Detail & Related papers  (2022-05-30T02:51:36Z)
• Increasing Depth of Neural Networks for Life-long Learning [2.0305676256390934]
  We propose a novel method for continual learning based on the increasing depth of neural networks. This work explores whether extending neural network depth may be beneficial in a life-long learning setting.
  arXiv  Detail & Related papers  (2022-02-22T11:21:41Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Exploring Knowledge Distillation of a Deep Neural Network for Multi-Script identification [8.72467690936929]
  Multi-lingual script identification is a difficult task consisting of different language with complex backgrounds in scene text images. Deep neural networks are employed as teacher models to train a smaller student network by utilizing the teacher model's predictions.
  arXiv  Detail & Related papers  (2021-02-20T12:54:07Z)
• Teacher-Class Network: A Neural Network Compression Mechanism [2.257416403770908]
  Instead of transferring knowledge to one student only, the proposed method transfers a chunk of knowledge to each student. Our students are not trained for problem-specific logits, they are trained to mimic knowledge (dense representation) learned by the teacher network. The proposed teacher-class architecture is evaluated on several benchmark datasets such as MNIST, Fashion MNIST, IMDB Movie Reviews, CAMVid, CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2020-04-07T11:31:20Z)
• Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model [57.41841346459995]
  We study how to train a student deep neural network for visual recognition by distilling knowledge from a blackbox teacher model in a data-efficient manner. We propose an approach that blends mixup and active learning.
  arXiv  Detail & Related papers  (2020-03-31T05:44:55Z)
• Curriculum By Smoothing [52.08553521577014]
  Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation. We propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters. As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data.
  arXiv  Detail & Related papers  (2020-03-03T07:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.