GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection

• URL: http://arxiv.org/abs/2603.29295v1
• Date: Tue, 31 Mar 2026 05:59:59 GMT
• Title: GazeCLIP: Gaze-Guided CLIP with Adaptive-Enhanced Fine-Grained Language Prompt for Deepfake Attribution and Detection
• Abstract summary: Current deepfake attribution or deepfake detection works tend to exhibit poor generalization to novel generative methods.<n>We propose a novel gaze-guided CLIP with adaptive-enhanced fine-grained language prompts for fine-grained deepfake attribution and detection (DFAD)<n>We conduct a novel and fine-grained benchmark to evaluate the DFAD performance of networks on novel generators like diffusion and flow models.<n>We introduce a gaze-aware model based on CLIP, which is devised to enhance the generalization to unseen face forgery attacks.
• Score: 80.12497948980378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current deepfake attribution or deepfake detection works tend to exhibit poor generalization to novel generative methods due to the limited exploration in visual modalities alone. They tend to assess the attribution or detection performance of models on unseen advanced generators, coarsely, and fail to consider the synergy of the two tasks. To this end, we propose a novel gaze-guided CLIP with adaptive-enhanced fine-grained language prompts for fine-grained deepfake attribution and detection (DFAD). Specifically, we conduct a novel and fine-grained benchmark to evaluate the DFAD performance of networks on novel generators like diffusion and flow models. Additionally, we introduce a gaze-aware model based on CLIP, which is devised to enhance the generalization to unseen face forgery attacks. Built upon the novel observation that there are significant distribution differences between pristine and forged gaze vectors, and the preservation of the target gaze in facial images generated by GAN and diffusion varies significantly, we design a visual perception encoder to employ the inherent gaze differences to mine global forgery embeddings across appearance and gaze domains. We propose a gaze-aware image encoder (GIE) that fuses forgery gaze prompts extracted via a gaze encoder with common forged image embeddings to capture general attribution patterns, allowing features to be transformed into a more stable and common DFAD feature space. We build a language refinement encoder (LRE) to generate dynamically enhanced language embeddings via an adaptive-enhanced word selector for precise vision-language matching. Extensive experiments on our benchmark show that our model outperforms the state-of-the-art by 6.56% ACC and 5.32% AUC in average performance under the attribution and detection settings, respectively. Codes will be available on GitHub.

Related papers

• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition [51.70817823155725]
  We propose a novel landmark-guided contrastive learning network with vision-language enhancement for Facial Expression Recognition (FER)<n>LaCoVL-FER integrates geometric priors from facial landmarks and semantic priors from a vision-language model.<n>Experiments demonstrate that LaCoVL-FER outperforms state-of-the-art methods on three representative real-world FER datasets.
  arXiv  Detail & Related papers  (2026-05-19T13:15:41Z)
• Deepfake Forensics Adapter: A Dual-Stream Network for Generalizable Deepfake Detection [22.889849855283355]
  Deepfake Forensics Adapter (DFA) is a novel dual-stream framework that synergizes vision-language foundation models with targeted forensics analysis.<n>Our approach integrates a pre-trained CLIP model with three core components to achieve specialized deepfake detection.<n>Our framework not only demonstrates state-of-the-art performance, but also points out a feasible and effective direction for developing a robust deepfake detection system.
  arXiv  Detail & Related papers  (2026-03-02T04:58:00Z)
• AdaptPrompt: Parameter-Efficient Adaptation of VLMs for Generalizable Deepfake Detection [7.76090543025328]
  Recent advances in image generation have led to the widespread availability of highly realistic synthetic media, increasing the difficulty of reliable deepfake detection.<n>A key challenge is generalization, as detectors trained on a narrow class of generators often fail when confronted with unseen models.<n>We address the pressing need for generalizable detection by leveraging large vision-language models, specifically CLIP, to identify synthetic content across diverse generative techniques.
  arXiv  Detail & Related papers  (2025-12-19T16:06:03Z)
• PVLM: Parsing-Aware Vision Language Model with Dynamic Contrastive Learning for Zero-Shot Deepfake Attribution [36.409362073662685]
  We propose a novel parsing-aware vision language model with dynamic contrastive learning (PVLM) method for zero-shot deepfake attribution (ZS-DFA)<n>Specifically, we conduct a novel and fine-grained ZS-DFA benchmark to evaluate the attribution performance of deepfake attributors to unseen advanced generators like diffusion.<n>We also propose an innovative parsing-guided vision language model with dynamic contrastive learning (PVLM) method to capture general and diverse attribution features.
  arXiv  Detail & Related papers  (2025-04-19T01:11:46Z)
• MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection [64.29452783056253]
  The rapid development of photo-realistic face generation methods has raised significant concerns in society and academia.<n>Although existing approaches mainly capture face forgery patterns using image modality, other modalities like fine-grained noises and texts are not fully explored.<n>We propose a novel multi-modal fine-grained CLIP (MFCLIP) model, which mines comprehensive and fine-grained forgery traces across image-noise modalities.
  arXiv  Detail & Related papers  (2024-09-15T13:08:59Z)
• GenFace: A Large-Scale Fine-Grained Face Forgery Benchmark and Cross Appearance-Edge Learning [50.7702397913573]
  The rapid advancement of photorealistic generators has reached a critical juncture where the discrepancy between authentic and manipulated images is increasingly indistinguishable. Although there have been a number of publicly available face forgery datasets, the forgery faces are mostly generated using GAN-based synthesis technology. We propose a large-scale, diverse, and fine-grained high-fidelity dataset, namely GenFace, to facilitate the advancement of deepfake detection.
  arXiv  Detail & Related papers  (2024-02-03T03:13:50Z)
• GazeCLIP: Enhancing Gaze Estimation Through Text-Guided Multimodal Learning [12.706496295933343]
  We propose GazeCLIP, a novel gaze estimation framework that deeply explores text-face collaboration.<n>Specifically, we introduce a meticulously designed linguistic description generator to produce text signals enriched with coarse directional cues.<n>Our findings underscore the potential of using visual-language collaboration to advance gaze estimation and open new avenues for future research in multimodal learning for visual tasks.
  arXiv  Detail & Related papers  (2023-12-30T15:24:50Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.