Migrating Face Swap to Mobile Devices: A lightweight Framework and A Supervised Training Solution

• URL: http://arxiv.org/abs/2204.08339v1
• Date: Wed, 13 Apr 2022 05:35:11 GMT
• Title: Migrating Face Swap to Mobile Devices: A lightweight Framework and A Supervised Training Solution
• Authors: Haiming Yu and Hao Zhu and Xiangju Lu and Junhui Liu
• Abstract summary: MobileFSGAN is a novel lightweight GAN for face swap that can run on mobile devices with much fewer parameters while achieving competitive performance. A lightweight encoder-decoder structure is designed especially for image synthesis tasks, which is only 10.2MB and can run on mobile devices at a real-time speed.
• Score: 7.572886749166295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing face swap methods rely heavily on large-scale networks for adequate capacity to generate visually plausible results, which inhibits its applications on resource-constraint platforms. In this work, we propose MobileFSGAN, a novel lightweight GAN for face swap that can run on mobile devices with much fewer parameters while achieving competitive performance. A lightweight encoder-decoder structure is designed especially for image synthesis tasks, which is only 10.2MB and can run on mobile devices at a real-time speed. To tackle the unstability of training such a small network, we construct the FSTriplets dataset utilizing facial attribute editing techniques. FSTriplets provides source-target-result training triplets, yielding pixel-level labels thus for the first time making the training process supervised. We also designed multi-scale gradient losses for efficient back-propagation, resulting in faster and better convergence. Experimental results show that our model reaches comparable performance towards state-of-the-art methods, while significantly reducing the number of network parameters. Codes and the dataset have been released.

Related papers

• Distilled Pooling Transformer Encoder for Efficient Realistic Image Dehazing [0.0]
  This paper proposes a lightweight neural network designed for realistic image dehazing, utilizing a Distilled Pooling Transformer, named DPTE-Net. Experimental results on various benchmark datasets have shown that the proposed DPTE-Net can achieve competitive dehazing performance when compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-18T14:16:23Z)
• Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks [7.439160287320074]
  Federated split learning (FedSL) implements collaborative training across the edge devices and the server through model splitting. We propose a lightweight FedSL scheme, that further alleviates the training burden on resource-constrained edge devices. We conduct theoretical analysis to quantify the convergence performance of the proposed scheme.
  arXiv  Detail & Related papers  (2024-12-09T11:43:03Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Cross-Scan Mamba with Masked Training for Robust Spectral Imaging [51.557804095896174]
  We propose the Cross-Scanning Mamba, named CS-Mamba, that employs a Spatial-Spectral SSM for global-local balanced context encoding. Experiment results show that our CS-Mamba achieves state-of-the-art performance and the masked training method can better reconstruct smooth features to improve the visual quality.
  arXiv  Detail & Related papers  (2024-08-01T15:14:10Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• LegoNet: A Fast and Exact Unlearning Architecture [59.49058450583149]
  Machine unlearning aims to erase the impact of specific training samples upon deleted requests from a trained model. We present a novel network, namely textitLegoNet, which adopts the framework of fixed encoder + multiple adapters'' We show that LegoNet accomplishes fast and exact unlearning while maintaining acceptable performance, synthetically outperforming unlearning baselines.
  arXiv  Detail & Related papers  (2022-10-28T09:53:05Z)
• DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization [27.231327287238102]
  We propose the DenseShift network, which significantly improves the accuracy of Shift networks. Our experiments on various computer vision and speech tasks demonstrate that DenseShift outperforms existing low-bit multiplication-free networks.
  arXiv  Detail & Related papers  (2022-08-20T15:17:40Z)
• Multi-Exit Semantic Segmentation Networks [78.44441236864057]
  We propose a framework for converting state-of-the-art segmentation models to MESS networks. specially trained CNNs that employ parametrised early exits along their depth to save during inference on easier samples. We co-optimise the number, placement and architecture of the attached segmentation heads, along with the exit policy, to adapt to the device capabilities and application-specific requirements.
  arXiv  Detail & Related papers  (2021-06-07T11:37:03Z)
• Learning Efficient GANs for Image Translation via Differentiable Masks and co-Attention Distillation [130.30465659190773]
  Generative Adversarial Networks (GANs) have been widely-used in image translation, but their high computation and storage costs impede the deployment on mobile devices. We introduce a novel GAN compression method, termed DMAD, by proposing a Differentiable Mask and a co-Attention Distillation. Experiments show DMAD can reduce the Multiply Accumulate Operations (MACs) of CycleGAN by 13x and that of Pix2Pix by 4x while retaining a comparable performance against the full model.
  arXiv  Detail & Related papers  (2020-11-17T02:39:19Z)
• DBQ: A Differentiable Branch Quantizer for Lightweight Deep Neural Networks [4.358626952482686]
  We present a novel non-uniform quantizer that can be seamlessly mapped onto efficient ternary-based dot product engines. The proposed quantizer (DBQ) successfully tackles the daunting task of aggressively quantizing lightweight networks such as MobileNetV1, MobileNetV2, and ShuffleNetV2. DBQ achieves state-of-the art results with minimal training overhead and provides the best (pareto-optimal) accuracy-complexity trade-off.
  arXiv  Detail & Related papers  (2020-07-19T23:50:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.