CosalPure: Learning Concept from Group Images for Robust Co-Saliency Detection

• URL: http://arxiv.org/abs/2403.18554v2
• Date: Fri, 12 Apr 2024 02:27:09 GMT
• Title: CosalPure: Learning Concept from Group Images for Robust Co-Saliency Detection
• Authors: Jiayi Zhu, Qing Guo, Felix Juefei-Xu, Yihao Huang, Yang Liu, Geguang Pu,
• Abstract summary: Co-salient object detection (CoSOD) aims to identify the common and salient (usually in the foreground) regions across a given group of images. adversarial perturbations could be easily affected by some adversarial perturbations, leading to substantial accuracy reduction. We propose a novel robustness enhancement framework by first learning the concept of the co-salient objects based on the input group images.
• Score: 22.82243087156918
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Co-salient object detection (CoSOD) aims to identify the common and salient (usually in the foreground) regions across a given group of images. Although achieving significant progress, state-of-the-art CoSODs could be easily affected by some adversarial perturbations, leading to substantial accuracy reduction. The adversarial perturbations can mislead CoSODs but do not change the high-level semantic information (e.g., concept) of the co-salient objects. In this paper, we propose a novel robustness enhancement framework by first learning the concept of the co-salient objects based on the input group images and then leveraging this concept to purify adversarial perturbations, which are subsequently fed to CoSODs for robustness enhancement. Specifically, we propose CosalPure containing two modules, i.e., group-image concept learning and concept-guided diffusion purification. For the first module, we adopt a pre-trained text-to-image diffusion model to learn the concept of co-salient objects within group images where the learned concept is robust to adversarial examples. For the second module, we map the adversarial image to the latent space and then perform diffusion generation by embedding the learned concept into the noise prediction function as an extra condition. Our method can effectively alleviate the influence of the SOTA adversarial attack containing different adversarial patterns, including exposure and noise. The extensive results demonstrate that our method could enhance the robustness of CoSODs significantly.

Related papers

• How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization? [91.49559116493414]
  We propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM) It can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Experiments validate that our CIDM surpasses existing custom diffusion models.
  arXiv  Detail & Related papers  (2024-10-23T06:47:29Z)
• Boosting Imperceptibility of Stable Diffusion-based Adversarial Examples Generation with Momentum [13.305800254250789]
  We propose a novel framework, Stable Diffusion-based Momentum Integrated Adversarial Examples (SD-MIAE) It generates adversarial examples that can effectively mislead neural network classifiers while maintaining visual imperceptibility and preserving the semantic similarity to the original class label. Experimental results demonstrate that SD-MIAE achieves a high misclassification rate of 79%, improving by 35% over the state-of-the-art method.
  arXiv  Detail & Related papers  (2024-10-17T01:22:11Z)
• Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion [51.931083971448885]
  We propose a framework named Human Feedback Inversion (HFI), where human feedback on model-generated images is condensed into textual tokens guiding the mitigation or removal of problematic images. Our experimental results demonstrate our framework significantly reduces objectionable content generation while preserving image quality, contributing to the ethical deployment of AI in the public sphere.
  arXiv  Detail & Related papers  (2024-07-17T05:21:41Z)
• Rethinking and Defending Protective Perturbation in Personalized Diffusion Models [21.30373461975769]
  We study the fine-tuning process of personalized diffusion models (PDMs) through the lens of shortcut learning. PDMs are susceptible to minor adversarial perturbations, leading to significant degradation when fine-tuned on corrupted datasets. We propose a systematic defense framework that includes data purification and contrastive decoupling learning.
  arXiv  Detail & Related papers  (2024-06-27T07:14:14Z)
• Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models [58.74606272936636]
  Text-to-image (T2I) diffusion models have shown exceptional capabilities in generating images that closely correspond to textual prompts. The models could be exploited for malicious purposes, such as generating images with violence or nudity, or creating unauthorized portraits of public figures in inappropriate contexts. concept removal methods have been proposed to modify diffusion models to prevent the generation of malicious and unwanted concepts.
  arXiv  Detail & Related papers  (2024-06-21T03:58:44Z)
• Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient [20.091446060893638]
  This paper proposes a concept domain correction framework for unlearning concepts in diffusion models. By aligning the output domains of sensitive concepts and anchor concepts through adversarial training, we enhance the generalizability of the unlearning results.
  arXiv  Detail & Related papers  (2024-05-24T07:47:36Z)
• PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation [50.556961575275345]
  We propose a perception-aware fusion framework to promote segmentation robustness in adversarial scenes. We show that our scheme substantially enhances the robustness, with gains of 15.3% mIOU, compared with advanced competitors.
  arXiv  Detail & Related papers  (2023-08-08T01:55:44Z)
• Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion [106.42918868850249]
  We propose a novel strategy named textbfDegeneration-Tuning (DT) to shield contents of unwanted concepts from SD weights. As this adaptation occurs at the level of the model's weights, the SD, after DT, can be grafted onto other conditional diffusion frameworks like ControlNet to shield unwanted concepts.
  arXiv  Detail & Related papers  (2023-08-02T03:34:44Z)
• Unpaired Deep Image Dehazing Using Contrastive Disentanglement Learning [36.24651058888557]
  We present an effective unpaired learning based image dehazing network from an unpaired set of clear and hazy images. We show that our method performs favorably against existing state-of-the-art unpaired dehazing approaches.
  arXiv  Detail & Related papers  (2022-03-15T06:45:03Z)
• Can You Spot the Chameleon? Adversarially Camouflaging Images from Co-Salient Object Detection [46.95646405874199]
  Co-salient object detection (CoSOD) has recently achieved significant progress and played a key role in retrieval-related tasks. In this paper, we identify a novel task: adversarial co-saliency attack. We propose the very first black-box joint adversarial exposure and noise attack (Jadena)
  arXiv  Detail & Related papers  (2020-09-19T15:43:46Z)
• Stylized Adversarial Defense [105.88250594033053]
  adversarial training creates perturbation patterns and includes them in the training set to robustify the model. We propose to exploit additional information from the feature space to craft stronger adversaries. Our adversarial training approach demonstrates strong robustness compared to state-of-the-art defenses.
  arXiv  Detail & Related papers  (2020-07-29T08:38:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.