On the Suitability of LLM-Driven Agents for Dark Pattern Audits

• URL: http://arxiv.org/abs/2603.03881v1
• Date: Wed, 04 Mar 2026 09:32:49 GMT
• Title: On the Suitability of LLM-Driven Agents for Dark Pattern Audits
• Authors: Chen Sun, Yash Vekaria, Rishab Nithyanand,
• Abstract summary: We evaluate the ability of an auditing agent to consistently locate and complete request flows across a set of data broker websites.<n>Our findings characterize both the feasibility and the limitations of using LLM-driven agents for scalable dark pattern auditing.
• Score: 5.938387201151421
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As LLM-driven agents begin to autonomously navigate the web, their ability to interpret and respond to manipulative interface design becomes critical. A fundamental question that emerges is: can such agents reliably recognize patterns of friction, misdirection, and coercion in interface design (i.e., dark patterns)? We study this question in a setting where the workflows are consequential: website portals associated with the submission of CCPA-related data rights requests. These portals operationalize statutory rights, but they are implemented as interactive interfaces whose design can be structured to facilitate, burden, or subtly discourage the exercise of those rights. We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows, structured evidence gathering, and classification of potential dark patterns. Across a set of 456 data broker websites, we evaluate: (1) the ability of the agent to consistently locate and complete request flows, (2) the reliability and reproducibility of its dark pattern classifications, and (3) the conditions under which it fails or produces poor judgments. Our findings characterize both the feasibility and the limitations of using LLM-driven agents for scalable dark pattern auditing.

Related papers

• Steering LLMs via Scalable Interactive Oversight [74.12746881843044]
  Large Language Models increasingly automate complex, long-horizon tasks such as emphvibe coding, a supervision gap has emerged.<n>It presents a critical challenge in scalable oversight: enabling humans to responsibly steer AI systems on tasks that surpass their own ability to specify or verify.
  arXiv  Detail & Related papers  (2026-02-04T04:52:00Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• SWAP: Towards Copyright Auditing of Soft Prompts via Sequential Watermarking [58.475471437150674]
  We propose sequential watermarking for soft prompts (SWAP)<n>SWAP encodes watermarks through a specific order of defender-specified out-of-distribution classes.<n>Experiments on 11 datasets demonstrate SWAP's effectiveness, harmlessness, and robustness against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-11-05T13:48:48Z)
• Investigating the Impact of Dark Patterns on LLM-Based Web Agents [16.297159088186888]
  We present the first study that investigates the impact of dark patterns on the decision-making process of LLM-based generalist web agents.<n>We introduce LiteAgent, a lightweight framework that automatically prompts agents to execute tasks.<n>We also present TrickyArena, a controlled environment comprising web applications from domains such as e-commerce, streaming services, and news platforms.
  arXiv  Detail & Related papers  (2025-10-20T21:26:26Z)
• Enterprise AI Must Enforce Participant-Aware Access Control [9.68210477539956]
  Large language models (LLMs) are increasingly deployed in enterprise settings where they interact with multiple users and are trained or fine-tuned on sensitive internal data.<n>We show that adversaries can exploit current fine-tuning and RAG architectures to leak sensitive information by leveraging the lack of access control enforcement.<n>We introduce a framework centered on the principle that any content used in training, retrieval, or generation by an LLM is explicitly authorized for emphall users involved in the interaction.
  arXiv  Detail & Related papers  (2025-09-18T04:30:49Z)
• Role-Aware Language Models for Secure and Contextualized Access Control in Organizations [4.122315998598296]
  Large language models (LLMs) are increasingly deployed in enterprise settings.<n>We investigate whether LLMs can be fine-tuned to generate responses that reflect the access privileges associated with different organizational roles.
  arXiv  Detail & Related papers  (2025-07-31T11:41:04Z)
• Permissioned LLMs: Enforcing Access Control in Large Language Models [11.090184334137758]
  We propose Permissioned LLMs that superimpose organizational data access control structures on query responses they generate.<n>We empirically evaluate the efficacy of a PermLLM mechanism.<n>We introduce three novel PermLLM mechanisms that build on Efficient Fine-Tuning to achieve the desired access control.
  arXiv  Detail & Related papers  (2025-05-28T20:47:02Z)
• WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback [78.55946306325914]
  We identify key reasoning skills essential for effective web agents.<n>We reconstruct the agent's reasoning algorithms into chain-of-thought rationales.<n>Our approach yields significant improvements across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T14:03:37Z)
• Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Services [22.700907666937177]
  This position paper highlights emerging accountability challenges in commercial Opaque LLM Services (COLS)<n>We formalize two key risks: textitquantity inflation, where token and call counts may be artificially inflated, and textitquality downgrade, where providers might quietly substitute lower-cost models or tools.<n>We propose a modular three-layer auditing framework for COLS and users that enables trustworthy verification across execution, secure logging, and user-facing auditability without exposing proprietary internals.
  arXiv  Detail & Related papers  (2025-05-24T02:26:49Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.