PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems

• URL: http://arxiv.org/abs/2607.05318v1
• Date: Mon, 06 Jul 2026 16:57:24 GMT
• Title: PiSAs: Benchmarking Contextual Integrity in Multi-User Agentic Systems
• Abstract summary: We introducePiSAs (Privacy in Shared Agentic systems), a benchmark for assessing unintentional leaks.<n>This enables direct measurement of cross-user spillage across agentic system components and interfaces.<n>We find that, although system design improves CI compliance, results are bottlenecked by incorrect LLM judgment calls.
• Score: 29.778310164003557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As LLM agents evolve from single-user assistants into shared organizational infrastructure, new privacy risks emerge: inappropriate information may not only be exposed through outputs for external recipients, but also internally across users through inter-agent messages, shared memory and agents. These data spillage risks are not captured by existing privacy benchmarks grounded in contextual integrity (CI) as they focus primarily on either single-user settings or interactions between independently owned agents. We introducePiSAs (Privacy in Shared Agentic systems), a benchmark for assessing unintentional leaks with dual CI annotations: whether an information is appropriate for the task, and which users may legitimately access it. This enables direct measurement of cross-user spillage across agentic system components and interfaces, such as outputs, inter-agent communication, and memory. PiSAsis system-agnostic and supports evaluation across different agent topologies and memory regimes. We find that, although system design improves CI compliance, results are bottlenecked by incorrect LLM judgment calls: even state-of-the-art models fail to reliably filter inappropriate content or restrict transmission to authorized users. Our findings underscore the need for privacy-preserving strategies, beyond those studied in this work.

Related papers

• SIR: Self-improving Red-teaming for Compute Use Agents [71.71987044117371]
  Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal.<n>Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI)<n>We present SIR, a black box IPI attack that composes stealthy injections from a small library of reusable principles stated in plain language.
  arXiv  Detail & Related papers  (2026-08-31T03:39:43Z)
• Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? [54.701199583505144]
  Cross-application access creates a privacy risk that has been largely overlooked.<n>We introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios.<n>We evaluate 15 frontier agents and find a surprisingly high failure rate.
  arXiv  Detail & Related papers  (2026-06-22T11:36:58Z)
• It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs [73.46918128880786]
  Self-distillation framework that decouples information suppression from task resolution.<n>Self-distillation framework that decouples information suppression from task resolution.
  arXiv  Detail & Related papers  (2026-05-18T13:57:44Z)
• Information-Theoretic Privacy Control for Sequential Multi-Agent LLM Systems [5.100622189286672]
  We study emphcompositional privacy leakage in sequential multi-agent large language model (LLM) systems.<n>Motivated by this analysis, we propose a privacy-regularized training framework that directly constrains information flow between agent outputs and agent-local sensitive variables.
  arXiv  Detail & Related papers  (2026-02-13T18:23:03Z)
• Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible [12.742325129012576]
  Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks.<n>We propose anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data.<n>Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders.
  arXiv  Detail & Related papers  (2026-02-08T15:50:04Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agents [3.802907024025868]
  AudAgent is a visual framework that monitors AI agents' data practices in real time.<n>AudAgent effectively identifies potential privacy policy violations in real time.
  arXiv  Detail & Related papers  (2025-11-03T17:32:08Z)
• Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactions [31.392818526374963]
  Large Language Models (LLMs) are increasingly deployed as agents that orchestrate tasks and integrate external tools.<n>We demonstrate that these interactive behaviors leave distinctive fingerprints in encrypted traffic exchanged between users and LLM agents.<n>By analyzing traffic patterns associated with agent and tool invocations, adversaries can infer agent activities.
  arXiv  Detail & Related papers  (2025-10-08T16:16:23Z)
• The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration [72.33801123508145]
  Large language models (LLMs) are integral to multi-agent systems.<n>Privacy risks emerge that extend beyond memorization, direct inference, or single-turn evaluations.<n>In particular, seemingly innocuous responses, when composed across interactions, can cumulatively enable adversaries to recover sensitive information.
  arXiv  Detail & Related papers  (2025-09-16T16:57:25Z)
• 1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning [18.751008976082655]
  We introduce a multi-agent framework that decomposes privacy reasoning into specialized subtasks (extraction, classification)<n>We conduct a systematic ablation over information-flow topologies, revealing when and why upstream detection mistakes cascade into downstream leakage.
  arXiv  Detail & Related papers  (2025-08-11T06:34:09Z)
• MAGPIE: A dataset for Multi-AGent contextual PrIvacy Evaluation [54.410825977390274]
  Existing benchmarks to evaluate contextual privacy in LLM-agents primarily assess single-turn, low-complexity tasks.<n>We first present a benchmark - MAGPIE comprising 158 real-life high-stakes scenarios across 15 domains.<n>We then evaluate the current state-of-the-art LLMs on their understanding of contextually private data and their ability to collaborate without violating user privacy.
  arXiv  Detail & Related papers  (2025-06-25T18:04:25Z)
• DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents [52.92354372596197]
  Large Language Models (LLMs) are increasingly central to agentic systems due to their strong reasoning and planning capabilities.<n>This interaction also introduces the risk of prompt injection attacks, where malicious inputs from external sources can mislead the agent's behavior.<n>We propose a Dynamic Rule-based Isolation Framework for Trustworthy agentic systems, which enforces both control and data-level constraints.
  arXiv  Detail & Related papers  (2025-06-13T05:01:09Z)
• Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents [33.26308626066122]
  We characterize the notion of contextual privacy for user interactions with Conversational Agents (LCAs)<n>It aims to minimize privacy risks by ensuring that users (sender) disclose only information that is both relevant and necessary for achieving their intended goals.<n>We propose a locally deployable framework that operates between users and LCAs, identifying and reformulating out-of-context information in user prompts.
  arXiv  Detail & Related papers  (2025-02-22T09:05:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.