Exploring Content Relationships for Distilling Efficient GANs

• URL: http://arxiv.org/abs/2212.11091v1
• Date: Wed, 21 Dec 2022 15:38:12 GMT
• Title: Exploring Content Relationships for Distilling Efficient GANs
• Authors: Lizhou You, Mingbao Lin, Tie Hu, Fei Chao, Rongrong Ji
• Abstract summary: This paper proposes a content relationship distillation (CRD) to tackle the over- parameterized generative adversarial networks (GANs) In contrast to traditional instance-level distillation, we design a novel GAN compression oriented knowledge by slicing the contents of teacher outputs into multiple fine-grained granularities. Built upon our proposed content-level distillation, we also deploy an online teacher discriminator, which keeps updating when co-trained with the teacher generator and keeps freezing when co-trained with the student generator for better adversarial training.
• Score: 69.86835014810714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes a content relationship distillation (CRD) to tackle the over-parameterized generative adversarial networks (GANs) for the serviceability in cutting-edge devices. In contrast to traditional instance-level distillation, we design a novel GAN compression oriented knowledge by slicing the contents of teacher outputs into multiple fine-grained granularities, such as row/column strips (global information) and image patches (local information), modeling the relationships among them, such as pairwise distance and triplet-wise angle, and encouraging the student to capture these relationships within its output contents. Built upon our proposed content-level distillation, we also deploy an online teacher discriminator, which keeps updating when co-trained with the teacher generator and keeps freezing when co-trained with the student generator for better adversarial training. We perform extensive experiments on three benchmark datasets, the results of which show that our CRD reaches the most complexity reduction on GANs while obtaining the best performance in comparison with existing methods. For example, we reduce MACs of CycleGAN by around 40x and parameters by over 80x, meanwhile, 46.61 FIDs are obtained compared with these of 51.92 for the current state-of-the-art. Code of this project is available at https://github.com/TheKernelZ/CRD.

Related papers

• CoroNetGAN: Controlled Pruning of GANs via Hypernetworks [5.765950477682605]
  We propose CoroNet-GAN for compressing GAN using the combined strength of differentiable pruning method via hypernetworks. Our approach succeeds to outperform the baselines on Zebra-to-Horse and Summer-to-Winter achieving the best FID score of 32.3 and 72.3 respectively.
  arXiv  Detail & Related papers  (2024-03-13T05:24:28Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• Discriminator-Cooperated Feature Map Distillation for GAN Compression [69.86835014810714]
  We present an inventive discriminator-cooperated distillation, abbreviated as DCD, towards refining better feature maps from the generator. Our DCD shows superior results compared with existing GAN compression methods.
  arXiv  Detail & Related papers  (2022-12-29T03:50:27Z)
• DETRs with Collaborative Hybrid Assignments Training [11.563949886871713]
  We present a novel collaborative hybrid assignments training scheme, namely $mathcalC$o-DETR. This training scheme can easily enhance the encoder's learning ability in end-to-end detectors. We conduct extensive experiments to evaluate the effectiveness of the proposed approach on DETR variants.
  arXiv  Detail & Related papers  (2022-11-22T16:19:52Z)
• Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation [74.67594286008317]
  This article addresses the problem of distilling knowledge from a large teacher model to a slim student network for LiDAR semantic segmentation. We propose the Point-to-Voxel Knowledge Distillation (PVD), which transfers the hidden knowledge from both point level and voxel level.
  arXiv  Detail & Related papers  (2022-06-05T05:28:32Z)
• Deep Structured Instance Graph for Distilling Object Detectors [82.16270736573176]
  We present a simple knowledge structure to exploit and encode information inside the detection system to facilitate detector knowledge distillation. We achieve new state-of-the-art results on the challenging COCO object detection task with diverse student-teacher pairs on both one- and two-stage detectors.
  arXiv  Detail & Related papers  (2021-09-27T08:26:00Z)
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers [52.85472936277762]
  We apply knowledge distillation to improve the recently proposed late-interaction ColBERT model. We distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT.
  arXiv  Detail & Related papers  (2020-10-22T02:26:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.