An AI Agent Execution Environment to Safeguard User Data

• URL: http://arxiv.org/abs/2604.19657v1
• Date: Tue, 21 Apr 2026 16:45:30 GMT
• Title: An AI Agent Execution Environment to Safeguard User Data
• Abstract summary: This paper presents (Guaranteed Accounting for Agent Privacy) an execution environment for AI agents that guarantees confidentiality for private user data.<n> Crucially, provides this guarantee deterministically, without trusting the agent with private user data, and without requiring any AI model or the user prompt to be free of attacks.
• Score: 2.2986365465048024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI agents promise to serve as general-purpose personal assistants for their users, which requires them to have access to private user data (e.g., personal and financial information). This poses a serious risk to security and privacy. Adversaries may attack the AI model (e.g., via prompt injection) to exfiltrate user data. Furthermore, sharing private data with an AI agent requires users to trust a potentially unscrupulous or compromised AI model provider with their private data. This paper presents GAAP (Guaranteed Accounting for Agent Privacy), an execution environment for AI agents that guarantees confidentiality for private user data. Through dynamic and directed user prompts, GAAP collects permission specifications from users describing how their private data may be shared, and GAAP enforces that the agent's disclosures of private user data, including disclosures to the AI model and its provider, comply with these specifications. Crucially, GAAP provides this guarantee deterministically, without trusting the agent with private user data, and without requiring any AI model or the user prompt to be free of attacks. GAAP enforces the user's permission specification by tracking how the AI agent accesses and uses private user data. It augments Information Flow Control with novel persistent data stores and annotations that enable it to track the flow of private information both across execution steps within a single task, and also over multiple tasks separated in time. Our evaluation confirms that GAAP blocks all data disclosure attacks, including those that make other state-of-the-art systems disclose private user data to untrusted parties, without a significant impact on agent utility.

Related papers

• How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement [55.966820779528376]
  As AI agents gain prevalence, users are increasingly exposed to the risks such systems entail.<n> Prompt injection attacks, as well as hallucination, can cause agents to leak private information to third parties.<n>As autonomous systems, agents present the more active danger of performing sensitive tasks, such as bank transactions, without the user's intent or authorization.
  arXiv  Detail & Related papers  (2026-07-15T11:36:22Z)
• Your Privacy Depends on Others: Collusion Vulnerabilities in Individual Differential Privacy [50.66105844449181]
  Individual Differential Privacy (iDP) promises users control over their privacy, but this promise can be broken in practice.<n>We reveal a previously overlooked vulnerability in sampling-based iDP mechanisms.<n>We propose $(varepsilon_i,_i,overline)$-iDP a privacy contract that uses $$-divergences to provide users with a hard upper bound on their excess vulnerability.
  arXiv  Detail & Related papers  (2026-01-19T10:26:12Z)
• How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy [52.00934156883483]
  Differential Privacy (DP) is a framework for reasoning about and limiting information leakage.<n>Differentially Private Synthetic data refers to synthetic data that preserves the overall trends of source data.
  arXiv  Detail & Related papers  (2025-12-02T21:14:39Z)
• Towards Usable Privacy Management for IoT TAPs: Deriving Privacy Clusters and Preference Profiles [1.2744523252873352]
  IoT Trigger-Action Platforms (TAPs) typically offer coarse-grained permission controls.<n>This paper contributes to usable privacy management for TAPs by deriving privacy clusters and profiles for different types of users.
  arXiv  Detail & Related papers  (2025-11-14T12:08:58Z)
• AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agents [3.802907024025868]
  AudAgent is a visual framework that monitors AI agents' data practices in real time.<n>AudAgent effectively identifies potential privacy policy violations in real time.
  arXiv  Detail & Related papers  (2025-11-03T17:32:08Z)
• Guarding Your Conversations: Privacy Gatekeepers for Secure Interactions with Cloud-Based AI Models [0.34998703934432673]
  We propose the concept of an "LLM gatekeeper", a lightweight, locally run model that filters out sensitive information from user queries before they are sent to the potentially untrustworthy, though highly capable, cloud-based LLM.<n>Through experiments with human subjects, we demonstrate that this dual-model approach introduces minimal overhead while significantly enhancing user privacy, without compromising the quality of LLM responses.
  arXiv  Detail & Related papers  (2025-08-22T19:49:03Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)
• PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Data [76.21047984886273]
  Personalization is critical in AI assistants, particularly in the context of private AI models that work with individual users.<n>Due to the sensitive nature of such data, there are no publicly available datasets that allow us to assess an AI model's ability to understand users.<n>We introduce a synthetic data generation pipeline that creates diverse, realistic user profiles and private documents simulating human activities.
  arXiv  Detail & Related papers  (2025-02-28T00:43:35Z)
• Air Gap: Protecting Privacy-Conscious Conversational Agents [44.04662124191715]
  We introduce a novel threat model where adversarial third-party apps manipulate the context of interaction to trick LLM-based agents into revealing private information not relevant to the task at hand. We introduce AirGapAgent, a privacy-conscious agent designed to prevent unintended data leakage by restricting the agent's access to only the data necessary for a specific task.
  arXiv  Detail & Related papers  (2024-05-08T16:12:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.