TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

• URL: http://arxiv.org/abs/2606.18996v2
• Date: Thu, 18 Jun 2026 04:53:40 GMT
• Title: TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction
• Abstract summary: We evaluate the trade-off of task accuracy and privacy leakage.<n>We propose structural private field isolation, which replaces private fields with hash keys before they reach the model.
• Score: 26.52125777621677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agents are increasingly deployed in document-intensive workflows where sensitive private information is not an edge case but a routine input, e.g., an agent booking a flight needs passport numbers. In such settings, the agent must use private information to complete tasks accurately while never exposing it in its responses, because it cannot verify who is actually at the keyboard. These two obligations are in fundamental tension. A model capable enough to use private information for task completion can, by the same capability, be induced to reveal it. To evaluate the trade-off of task accuracy and privacy leakage, we introduce Task-completion and Resistance to Active Privacy-extraction (TRAP). Each scenario includes a document containing private information, a task query that requires the agent to invoke the correct tool using private fields, and an attack query that attempts to elicit the same information in natural language. Evaluating 22 models spanning frontier proprietary and open-source models at multiple scales, we find that all model families exhibit non-trivial leakage, and that instruction-following ability correlates with leakage rate. Existing prompt-based defenses reduce leakage but at significant cost to task accuracy. Prompt optimization fails to escape this trade-off. We demonstrate that this failure is not incidental. For any softmax-based model, no soft-constraint defense, e.g., prompt-based defenses, can jointly achieve high task success with zero leakage probability. Motivated by this impossibility result, we propose structural private field isolation, which replaces private fields with hash keys before they reach the model. This approach largely prevents leakage while keeping task accuracy.

Related papers

• Inadvertent Context Leakage in Language Models [44.48836485611056]
  For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data.<n>We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs.<n>We show that this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model.
  arXiv  Detail & Related papers  (2026-08-20T10:05:29Z)
• OCELOT: Inference-Leakage Budgets for Privacy-Preserving LLM Agents [32.5466552510287]
  Leakage is cumulative, as individually innocuous releases accumulate across honest-but-curious or colluding sinks.<n>We present OCELOT, a runtime mediator that budgets how much an adversary's belief about a secret may improve.<n>OCELOT attains significantly lower leakage at higher task utility, resists adaptive injection, jailbreak, cumulative inference, and sink collusion.
  arXiv  Detail & Related papers  (2026-06-10T17:13:35Z)
• From Privacy to Workflow Integrity: Communication-Graph Metadata in Autonomous Agent Interoperability [0.0]
  A2A and MCP protocols standardize what agents say to one another, but assume address-based transport over HTTP(S)<n>What they leave in the clear is the communication graph: which agent contacts which, when, and how often.<n>We give a threat model for the agent communication graph; identify what makes agent metadata distinctively revealing.
  arXiv  Detail & Related papers  (2026-06-05T11:07:55Z)
• PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say [48.84133320567554]
  LLM-based agents are rapidly advancing, autonomously invoking external tools to complete tasks for users.<n> Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context.<n>We introduce emphPrivacyPeek, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents.
  arXiv  Detail & Related papers  (2026-05-29T04:55:12Z)
• MosaicLeaks:Privacy Risks in Querying-in-the-Open for Deep Research Agents [76.60965179634664]
  Deep research agents increasingly combine private local documents with external tools like web retrieval, creating a privacy risk.<n>We introduce MosaicLeaks, a benchmark of 1,001 deep research tasks that chain private enterprise documents and a public web corpus.<n>We find that models across families and sizes frequently leak at all three levels, that zero-shot privacy prompting reduces but does not eliminate leakage.
  arXiv  Detail & Related papers  (2026-05-29T01:44:35Z)
• Do Phone-Use Agents Respect Your Privacy? [97.81424230136075]
  We introduce MyPhoneBench, a verifiable evaluation framework for privacy behavior in mobile agents.<n>We operationalize privacy-respecting phone use as permissioned access, minimal disclosure, and user-controlled memory through a minimal privacy contract.<n>Across five frontier models on 10 mobile apps and 300 tasks, we find that task success, privacy-compliant task completion, and later-session use of saved preferences are distinct capabilities.
  arXiv  Detail & Related papers  (2026-04-01T14:50:50Z)
• Privacy Guard & Token Parsimony by Prompt and Context Handling and LLM Routing [0.0]
  "Privacy Guard" decomposes prompts into focused sub-tasks, re-routing high-risk queries to Zero-Trust or NDA-covered models.<n>This dual mechanism simultaneously eliminates sensitive inference vectors (Zero Leakage) and reduces cloud token payloads (OpEx Reduction)
  arXiv  Detail & Related papers  (2026-03-30T20:16:42Z)
• BlackCATT: Black-box Collusion Aware Traitor Tracing in Federated Learning [51.251962154210474]
  We present a general collusion-resistant embedding method for black-box traitor tracing in Federated Learning: BlackCATT.<n> Experimental results confirm the efficacy of the proposed scheme across different architectures and datasets.<n>For models that would otherwise suffer from update incompatibility on the main task, our proposed BlackCATT+FR incorporates functional regularization.
  arXiv  Detail & Related papers  (2026-02-12T16:26:57Z)
• Evaluating Language Model Reasoning about Confidential Information [95.64687778185703]
  We study whether language models exhibit contextual robustness, or the capability to adhere to context-dependent safety specifications.<n>We develop a benchmark (PasswordEval) that measures whether language models can correctly determine when a user request is authorized.<n>We find that current open- and closed-source models struggle with this seemingly simple task, and that, perhaps surprisingly, reasoning capabilities do not generally improve performance.
  arXiv  Detail & Related papers  (2025-08-27T15:39:46Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)
• Air Gap: Protecting Privacy-Conscious Conversational Agents [44.04662124191715]
  We introduce a novel threat model where adversarial third-party apps manipulate the context of interaction to trick LLM-based agents into revealing private information not relevant to the task at hand. We introduce AirGapAgent, a privacy-conscious agent designed to prevent unintended data leakage by restricting the agent's access to only the data necessary for a specific task.
  arXiv  Detail & Related papers  (2024-05-08T16:12:45Z)
• Can Foundation Models Help Us Achieve Perfect Secrecy? [11.073539163281524]
  A key promise of machine learning is the ability to assist users with personal tasks. A gold standard privacy-preserving system will satisfy perfect secrecy. However, privacy and quality appear to be in tension in existing systems for personal tasks.
  arXiv  Detail & Related papers  (2022-05-27T02:32:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.