Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating

• URL: http://arxiv.org/abs/2603.12598v1
• Date: Fri, 13 Mar 2026 03:03:20 GMT
• Title: Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating
• Abstract summary: We introduce Neural Gate, a novel method for mitigating privacy risks through neuron-level model editing.<n>Our method improves a model's privacy safeguards by increasing its rate of refusal for privacy-related questions.
• Score: 71.55435880263238
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) have shown remarkable potential across a wide array of vision-language tasks, leading to their adoption in critical domains such as finance and healthcare. However, their growing deployment also introduces significant security and privacy risks. Malicious actors could potentially exploit these models to extract sensitive information, highlighting a critical vulnerability. Recent studies show that LVLMs often fail to consistently refuse instructions designed to compromise user privacy. While existing work on privacy protection has made meaningful progress in preventing the leakage of sensitive data, they are constrained by limitations in both generalization and non-destructiveness. They often struggle to robustly handle unseen privacy-related queries and may inadvertently degrade a model's performance on standard tasks. To address these challenges, we introduce Neural Gate, a novel method for mitigating privacy risks through neuron-level model editing. Our method improves a model's privacy safeguards by increasing its rate of refusal for privacy-related questions, crucially extending this protective behavior to novel sensitive queries not encountered during the editing process. Neural Gate operates by learning a feature vector to identify neurons associated with privacy-related concepts within the model's representation of a subject. This localization then precisely guides the update of model parameters. Through comprehensive experiments on MiniGPT and LLaVA, we demonstrate that our method significantly boosts the model's privacy protection while preserving its original utility.

Related papers

• Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning [26.034865955638864]
  We propose a privacy-enhanced continual learning framework that forgets what's sensitive and remembers what matters.<n>Our approach first introduces a token-level dynamic Differential Privacy strategy.<n>Second, we integrate a privacy-guided memory sculpting module.
  arXiv  Detail & Related papers  (2025-09-16T11:01:59Z)
• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• A Survey on Privacy Risks and Protection in Large Language Models [13.602836059584682]
  Large Language Models (LLMs) have become increasingly integral to diverse applications, raising privacy concerns.<n>This survey offers a comprehensive overview of privacy risks associated with LLMs and examines current solutions to mitigate these challenges.
  arXiv  Detail & Related papers  (2025-05-04T03:04:07Z)
• Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directions [11.338466798715906]
  Fine-tuning Large Language Models (LLMs) can achieve state-of-the-art performance across various domains.<n>This paper provides a comprehensive survey of privacy challenges associated with fine-tuning LLMs.<n>We highlight vulnerabilities to various privacy attacks, including membership inference, data extraction, and backdoor attacks.
  arXiv  Detail & Related papers  (2024-12-21T06:41:29Z)
• Model Inversion Attacks: A Survey of Approaches and Countermeasures [69.18069679327263]
  Recently, a new type of privacy attack, the model inversion attacks (MIAs), aims to extract sensitive features of private data for training.<n>Despite the significance, there is a lack of systematic studies that provide a comprehensive overview and deeper insights into MIAs.<n>This survey aims to summarize up-to-date MIA methods in both attacks and defenses.
  arXiv  Detail & Related papers  (2024-11-15T08:09:28Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Exploring the Privacy Protection Capabilities of Chinese Large Language Models [19.12726985060863]
  We have devised a three-tiered progressive framework for evaluating privacy in language systems. Our primary objective is to comprehensively evaluate the sensitivity of large language models to private information. Our observations indicate that existing Chinese large language models universally show privacy protection shortcomings.
  arXiv  Detail & Related papers  (2024-03-27T02:31:54Z)
• PAC Privacy Preserving Diffusion Models [6.299952353968428]
  Diffusion models can produce images with both high privacy and visual quality.<n>However, challenges arise such as in ensuring robust protection in privatizing specific data attributes.<n>We introduce the PAC Privacy Preserving Diffusion Model, a model leverages diffusion principles and ensure Probably Approximately Correct (PAC) privacy.
  arXiv  Detail & Related papers  (2023-12-02T18:42:52Z)
• Privacy in Large Language Models: Attacks, Defenses and Future Directions [84.73301039987128]
  We analyze the current privacy attacks targeting large language models (LLMs) and categorize them according to the adversary's assumed capabilities. We present a detailed overview of prominent defense strategies that have been developed to counter these privacy attacks.
  arXiv  Detail & Related papers  (2023-10-16T13:23:54Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Hiding Visual Information via Obfuscating Adversarial Perturbations [47.315523613407244]
  We propose an adversarial visual information hiding method to protect the visual privacy of data. Specifically, the method generates obfuscating adversarial perturbations to obscure the visual information of the data. Experimental results on the recognition and classification tasks demonstrate that the proposed method can effectively hide visual information.
  arXiv  Detail & Related papers  (2022-09-30T08:23:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.