Multi-spectral Class Center Network for Face Manipulation Detection and Localization

• URL: http://arxiv.org/abs/2305.10794v3
• Date: Sat, 13 Jul 2024 14:29:30 GMT
• Title: Multi-spectral Class Center Network for Face Manipulation Detection and Localization
• Authors: Changtao Miao, Qi Chu, Zhentao Tan, Zhenchao Jin, Tao Gong, Wanyi Zhuang, Yue Wu, Bin Liu, Honggang Hu, Nenghai Yu,
• Abstract summary: We propose a novel Multi-Spectral Class Center Network (MSCCNet) for face manipulation detection and localization. Based on the features of different frequency bands, the MSCC module collects multi-spectral class centers and computes pixel-to-class relations. Applying multi-spectral class-level representations suppresses the semantic information of the visual concepts which is insensitive to manipulated regions of forgery images.
• Score: 52.569170436393165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As deepfake content proliferates online, advancing face manipulation forensics has become crucial. To combat this emerging threat, previous methods mainly focus on studying how to distinguish authentic and manipulated face images. Although impressive, image-level classification lacks explainability and is limited to specific application scenarios, spurring recent research on pixel-level prediction for face manipulation forensics. However, existing forgery localization methods suffer from exploring frequency-based forgery traces in the localization network. In this paper, we observe that multi-frequency spectrum information is effective for identifying tampered regions. To this end, a novel Multi-Spectral Class Center Network (MSCCNet) is proposed for face manipulation detection and localization. Specifically, we design a Multi-Spectral Class Center (MSCC) module to learn more generalizable and multi-frequency features. Based on the features of different frequency bands, the MSCC module collects multi-spectral class centers and computes pixel-to-class relations. Applying multi-spectral class-level representations suppresses the semantic information of the visual concepts which is insensitive to manipulated regions of forgery images. Furthermore, we propose a Multi-level Features Aggregation (MFA) module to employ more low-level forgery artifacts and structural textures. Meanwhile, we conduct a comprehensive localization benchmark based on pixel-level FF++ and Dolos datasets. Experimental results quantitatively and qualitatively demonstrate the effectiveness and superiority of the proposed MSCCNet. We expect this work to inspire more studies on pixel-level face manipulation localization. The codes are available (https://github.com/miaoct/MSCCNet).

Related papers

• Mixture-of-Noises Enhanced Forgery-Aware Predictor for Multi-Face Manipulation Detection and Localization [52.87635234206178]
  This paper proposes a new framework, namely MoNFAP, specifically tailored for multi-face manipulation detection and localization. The framework incorporates two novel modules: the Forgery-aware Unified Predictor (FUP) Module and the Mixture-of-Noises Module (MNM)
  arXiv  Detail & Related papers  (2024-08-05T08:35:59Z)
• Exploring Multi-view Pixel Contrast for General and Robust Image Forgery Localization [4.8454936010479335]
  We propose a Multi-view Pixel-wise Contrastive algorithm (MPC) for image forgery localization. Specifically, we first pre-train the backbone network with the supervised contrastive loss. Then the localization head is fine-tuned using the cross-entropy loss, resulting in a better pixel localizer.
  arXiv  Detail & Related papers  (2024-06-19T13:51:52Z)
• Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training [51.16994853817024]
  This work focuses on designing an effective pre-training framework for 3D radiology images. We introduce Disruptive Autoencoders, a pre-training framework that attempts to reconstruct the original image from disruptions created by a combination of local masking and low-level perturbations. The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-31T17:59:42Z)
• Hierarchical Forgery Classifier On Multi-modality Face Forgery Clues [61.37306431455152]
  We propose a novel Hierarchical Forgery for Multi-modality Face Forgery Detection (HFC-MFFD) The HFC-MFFD learns robust patches-based hybrid representation to enhance forgery authentication in multiple-modality scenarios. The specific hierarchical face forgery is proposed to alleviate the class imbalance problem and further boost detection performance.
  arXiv  Detail & Related papers  (2022-12-30T10:54:29Z)
• Probabilistic Deep Metric Learning for Hyperspectral Image Classification [91.5747859691553]
  This paper proposes a probabilistic deep metric learning framework for hyperspectral image classification. It aims to predict the category of each pixel for an image captured by hyperspectral sensors. Our framework can be readily applied to existing hyperspectral image classification methods.
  arXiv  Detail & Related papers  (2022-11-15T17:57:12Z)
• MSMG-Net: Multi-scale Multi-grained Supervised Metworks for Multi-task Image Manipulation Detection and Localization [1.14219428942199]
  A novel multi-scale multi-grained deep network (MSMG-Net) is proposed to automatically identify manipulated regions. In our MSMG-Net, a parallel multi-scale feature extraction structure is used to extract multi-scale features. The MSMG-Net can effectively perceive the object-level semantics and encode the edge artifact.
  arXiv  Detail & Related papers  (2022-11-06T14:58:21Z)
• Multiscale Convolutional Transformer with Center Mask Pretraining for Hyperspectral Image Classificationtion [14.33259265286265]
  We propose a noval multi-scale convolutional embedding module for hyperspectral images (HSI) to realize effective extraction of spatial-spectral information. Similar to Mask autoencoder, but our pre-training method only masks the corresponding token of the central pixel in the encoder, and inputs the remaining token into the decoder to reconstruct the spectral information of the central pixel.
  arXiv  Detail & Related papers  (2022-03-09T14:42:26Z)
• MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detection [10.594107680952774]
  Key research question for image manipulation detection is how to learn generalizable features that are sensitive to manipulations in novel data. In this paper we address both aspects by multi-view feature learning and multi-scale supervision. Our thoughts are realized by a new network which we term MVSS-Net and its enhanced version MVSS-Net++.
  arXiv  Detail & Related papers  (2021-12-16T15:01:52Z)
• Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images [108.79667788962425]
  salient object detection in optical remote sensing images (RSI-SOD) remains to be a challenging emerging topic. We propose a novel Multi-Content Complementation Network (MCCNet) to explore the complementarity of multiple content for RSI-SOD. In MCCM, we consider multiple types of features that are critical to RSI-SOD, including foreground features, edge features, background features, and global image-level features.
  arXiv  Detail & Related papers  (2021-12-02T04:46:40Z)
• Image Manipulation Detection by Multi-View Multi-Scale Supervision [11.319080833880307]
  Key challenge of image manipulation detection is how to learn generalizable features that are sensitive to manipulations in novel data. In this paper we address both aspects by multi-view feature learning and multi-scale supervision. Our thoughts are realized by a new network which we term MVSS-Net.
  arXiv  Detail & Related papers  (2021-04-14T13:05:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.