A Comparative Study of Knowledge Transfer Methods for Misaligned Urban
Building Labels
- URL: http://arxiv.org/abs/2311.03867v1
- Date: Tue, 7 Nov 2023 10:31:41 GMT
- Title: A Comparative Study of Knowledge Transfer Methods for Misaligned Urban
Building Labels
- Authors: Bipul Neupane, Jagannath Aryal, Abbas Rajabifard
- Abstract summary: Misalignment in Earth observation (EO) images and building labels impact the training of accurate convolutional neural networks (CNNs) for semantic segmentation of building footprints.
Recently, three Teacher-Student knowledge transfer methods have been introduced to address this issue.
We present a workflow for the systematic comparative study of the three methods.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Misalignment in Earth observation (EO) images and building labels impact the
training of accurate convolutional neural networks (CNNs) for semantic
segmentation of building footprints. Recently, three Teacher-Student knowledge
transfer methods have been introduced to address this issue: supervised domain
adaptation (SDA), knowledge distillation (KD), and deep mutual learning (DML).
However, these methods are merely studied for different urban buildings
(low-rise, mid-rise, high-rise, and skyscrapers), where misalignment increases
with building height and spatial resolution. In this study, we present a
workflow for the systematic comparative study of the three methods. The
workflow first identifies the best (with the highest evaluation scores)
hyperparameters, lightweight CNNs for the Student (among 43 CNNs from Computer
Vision), and encoder-decoder networks (EDNs) for both Teachers and Students.
Secondly, three building footprint datasets are developed to train and evaluate
the identified Teachers and Students in the three transfer methods. The results
show that U-Net with VGG19 (U-VGG19) is the best Teacher, and
U-EfficientNetv2B3 and U-EfficientNet-lite0 are among the best Students. With
these Teacher-Student pairs, SDA could yield upto 0.943, 0.868, 0.912, and
0.697 F1 scores in the low-rise, mid-rise, high-rise, and skyscrapers
respectively. KD and DML provide model compression of upto 82%, despite
marginal loss in performance. This new comparison concludes that SDA is the
most effective method to address the misalignment problem, while KD and DML can
efficiently compress network size without significant loss in performance. The
158 experiments and datasets developed in this study will be valuable to
minimise the misaligned labels.
Related papers
- Relative Difficulty Distillation for Semantic Segmentation [54.76143187709987]
We propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD)
RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals.
Our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.
arXiv Detail & Related papers (2024-07-04T08:08:25Z) - Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient
Semantic Segmentation [16.957139277317005]
Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD) is a new contrastive distillation learning paradigm.
Af-DCD trains compact and accurate deep neural networks for semantic segmentation applications.
arXiv Detail & Related papers (2023-12-07T09:37:28Z) - One-for-All: Bridge the Gap Between Heterogeneous Architectures in
Knowledge Distillation [69.65734716679925]
Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme.
Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family.
We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
arXiv Detail & Related papers (2023-10-30T11:13:02Z) - Representation Disparity-aware Distillation for 3D Object Detection [44.17712259352281]
This paper presents a novel representation disparity-aware distillation (RDD) method to address the representation disparity issue.
Our RDD increases mAP of CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance while taking up only 42% FLOPs.
arXiv Detail & Related papers (2023-08-20T16:06:42Z) - Improving Knowledge Distillation via Regularizing Feature Norm and
Direction [16.98806338782858]
Knowledge distillation (KD) exploits a large well-trained model (i.e., teacher) to train a small student model on the same dataset for the same task.
Treating teacher features as knowledge, prevailing methods of knowledge distillation train student by aligning its features with the teacher's, e.g., by minimizing the KL-divergence between their logits or L2 distance between their intermediate features.
While it is natural to believe that better alignment of student features to the teacher better distills teacher knowledge, simply forcing this alignment does not directly contribute to the student's performance, e.g.
arXiv Detail & Related papers (2023-05-26T15:05:19Z) - Hierarchical Supervision and Shuffle Data Augmentation for 3D
Semi-Supervised Object Detection [90.32180043449263]
State-of-the-art 3D object detectors are usually trained on large-scale datasets with high-quality 3D annotations.
A natural remedy is to adopt semi-supervised learning (SSL) by leveraging a limited amount of labeled samples and abundant unlabeled samples.
This paper introduces a novel approach of Hierarchical Supervision and Shuffle Data Augmentation (HSSDA), which is a simple yet effective teacher-student framework.
arXiv Detail & Related papers (2023-04-04T02:09:32Z) - Open-Set Semi-Supervised Learning for 3D Point Cloud Understanding [62.17020485045456]
It is commonly assumed in semi-supervised learning (SSL) that the unlabeled data are drawn from the same distribution as that of the labeled ones.
We propose to selectively utilize unlabeled data through sample weighting, so that only conducive unlabeled data would be prioritized.
arXiv Detail & Related papers (2022-05-02T16:09:17Z) - What Stops Learning-based 3D Registration from Working in the Real
World? [53.68326201131434]
This work identifies the sources of 3D point cloud registration failures, analyze the reasons behind them, and propose solutions.
Ultimately, this translates to a best-practice 3D registration network (BPNet), constituting the first learning-based method able to handle previously-unseen objects in real-world data.
Our model generalizes to real data without any fine-tuning, reaching an accuracy of up to 67% on point clouds of unseen objects obtained with a commercial sensor.
arXiv Detail & Related papers (2021-11-19T19:24:27Z) - Boosting Light-Weight Depth Estimation Via Knowledge Distillation [21.93879961636064]
We propose a lightweight network that can accurately estimate depth maps using minimal computing resources.
We achieve this by designing a compact model architecture that maximally reduces model complexity.
Our method achieves comparable performance to state-of-the-art methods while using only 1% of their parameters.
arXiv Detail & Related papers (2021-05-13T08:42:42Z) - Spirit Distillation: Precise Real-time Prediction with Insufficient Data [4.6247655021017655]
We propose a new training framework named Spirit Distillation(SD)
It extends the ideas of fine-tuning-based transfer learning(FTT) and feature-based knowledge distillation.
Results demonstrate the boosting performance in segmentation(mIOU) and high-precision accuracy boost by 1.4% and 8.2% respectively.
arXiv Detail & Related papers (2021-03-25T10:23:30Z) - SelfVoxeLO: Self-supervised LiDAR Odometry with Voxel-based Deep Neural
Networks [81.64530401885476]
We propose a self-supervised LiDAR odometry method, dubbed SelfVoxeLO, to tackle these two difficulties.
Specifically, we propose a 3D convolution network to process the raw LiDAR data directly, which extracts features that better encode the 3D geometric patterns.
We evaluate our method's performances on two large-scale datasets, i.e., KITTI and Apollo-SouthBay.
arXiv Detail & Related papers (2020-10-19T09:23:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.