Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements

• URL: http://arxiv.org/abs/2507.12844v1
• Date: Thu, 17 Jul 2025 07:10:27 GMT
• Title: Machine-Readable Ads: Accessibility and Trust Patterns for AI Web Agents interacting with Online Advertisements
• Abstract summary: multimodal language models are rapidly evolving into web agents that can browse, click, and purchase items on behalf of users.<n>Yet little is known about how these agents interact with ads or which design principles ensure reliable engagement.<n>We ran a controlled experiment using a faithful clone of the news site TT.com, seeded with diverse ads.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous multimodal language models are rapidly evolving into web agents that can browse, click, and purchase items on behalf of users, posing a threat to display advertising designed for human eyes. Yet little is known about how these agents interact with ads or which design principles ensure reliable engagement. To address this, we ran a controlled experiment using a faithful clone of the news site TT.com, seeded with diverse ads: static banners, GIFs, carousels, videos, cookie dialogues, and paywalls. We ran 300 initial trials plus follow-ups using the Document Object Model (DOM)-centric Browser Use framework with GPT-4o, Claude 3.7 Sonnet, Gemini 2.0 Flash, and the pixel-based OpenAI Operator, across 10 realistic user tasks. Our results show these agents display severe satisficing: they never scroll beyond two viewports and ignore purely visual calls to action, clicking banners only when semantic button overlays or off-screen text labels are present. Critically, when sweepstake participation required a purchase, GPT-4o and Claude 3.7 Sonnet subscribed in 100% of trials, and Gemini 2.0 Flash in 70%, revealing gaps in cost-benefit analysis. We identified five actionable design principles-semantic overlays, hidden labels, top-left placement, static frames, and dialogue replacement, that make human-centric creatives machine-detectable without harming user experience. We also evaluated agent trustworthiness through "behavior patterns" such as cookie consent handling and subscription choices, highlighting model-specific risk boundaries and the urgent need for robust trust evaluation frameworks in real-world advertising.

Related papers

• When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents [31.736012043129495]
  We show that users and agents observe consistent information from the same interface.<n>The same UI state can present materially different information to users and agents, a mismatch we term human-agent UI desynchronization.<n>We develop an automated framework that constructs user runtime instruction-agnostic UI desynchronization attacks and realizes them in deployable APKs.
  arXiv  Detail & Related papers  (2026-09-15T07:02:10Z)
• When Ad Networks Misbehave: Understanding Risks of Semi-Drive-By Splash Ads [13.802323882269995]
  We investigate the mobile splash ads ecosystem, where monetization relies on interaction signals that are difficult to verify end-to-end.<n>We uncover a previously less-known form of fraud at the ad-network layer in which splash ads are triggered not by intentional user actions but by incidental or indirect interactions.<n>By translating non-ad interactions into billable engagement events, ad networks can inflate performance metrics, overcharge advertisers, and erode user trust.
  arXiv  Detail & Related papers  (2026-09-09T00:54:43Z)
• Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models [58.59008920408855]
  We propose Adversarial Probing for Implicit VulnErabilities (AdvPIE) to expose implicit vulnerabilities without requiring access to the parameters of target models.<n> AdvPIE adopts a policy agent to generate and refine implicit adversarial prompts based on the feedback from a judge agent.<n>Experiments on standard and safety-aligned T2I models show that AdvPIE1 effectively uncovers implicit vulnerabilities.
  arXiv  Detail & Related papers  (2026-09-05T13:45:09Z)
• MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content [32.239107706539464]
  We present MIRAGE, a pipeline that turns benign mobile screenshots into prompt-injection samples.<n>A key challenge is that an injected screenshot must stay visually indistinguishable from genuine user content while still diverting the agent.<n>On a 1,111-sample benchmark spanning ten applications and eleven attack intents, all five evaluated VLM agents are vulnerable.
  arXiv  Detail & Related papers  (2026-05-27T08:09:53Z)
• Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces [12.062544433817408]
  Vision-language model (VLM) based web agents demonstrate impressive autonomous GUI interaction but remain vulnerable to deceptive interface elements.<n>We formalize deception-aware web agent defense and propose DUDE (Deceptive UI Detector & Evaluator), a two-stage framework combining hybrid-reward learning with asymmetric penalties and experience summarization to distill failure patterns into transferable guidance.<n>Experiments show DUDE reduces deception susceptibility by 53.8% while maintaining task performance, establishing an effective foundation for robust web agent deployment.
  arXiv  Detail & Related papers  (2026-05-10T12:11:52Z)
• SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agents [32.00656418129339]
  We propose SnapGuard, a lightweight yet accurate method for detecting prompt injection attacks from webpage screenshots.<n>We show that SnapGuard achieves an F1 score of 0.75, outperforming GPT-4o-prompt while being 8x faster (1.81s vs. 14.50s) and introducing no additional memory overhead.
  arXiv  Detail & Related papers  (2026-04-28T12:32:21Z)
• WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agents [117.65855863464863]
  Web agents are highly vulnerable to prompt injection attacks.<n>Existing defenses, including system prompt defenses and direct fine-tuning of agents, have shown limited effectiveness.<n>We introduce WebAgentGuard, a reasoning-driven, multimodal guard model for prompt injection detection.
  arXiv  Detail & Related papers  (2026-04-14T04:50:35Z)
• FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback [92.67587639164908]
  We present FronTalk, a benchmark for front-end code generation with multi-modal feedback.<n>We focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues.<n> Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature.
  arXiv  Detail & Related papers  (2025-12-05T23:28:09Z)
• Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent Disambiguation [60.63465682731118]
  The performance of egocentric AI agents is fundamentally limited by multimodal intent ambiguity.<n>We introduce the Plug-and-Play Clarifier, a zero-shot and modular framework that decomposes the problem into discrete, solvable sub-tasks.<n>Our framework improves the intent clarification performance of small language models by approximately 30%, making them competitive with significantly larger counterparts.
  arXiv  Detail & Related papers  (2025-11-12T04:28:14Z)
• When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins [17.279238456399693]
  We present the first large-scale study of 17 third-party plugins used by over 10,000 public websites.<n>8 of these plugins fail to enforce the integrity of the conversation history transmitted in network requests.<n>We found that 13% of e-commerce websites have already exposed their chatbots to third-party content.
  arXiv  Detail & Related papers  (2025-11-08T02:02:24Z)
• Effective and Stealthy One-Shot Jailbreaks on Deployed Mobile Vision-Language Agents [29.62914440645731]
  We present a one-shot jailbreak attack that leverages in-app prompt injections.<n> malicious apps embed short prompts in UI text that remain inert during human interaction but are revealed when an agent drives the UI via ADB.<n>Our framework comprises three crucial components: (1) low-privilege perception-chain targeting, which injects payloads into malicious apps as the agent's visual inputs; (2) user-invisible activation, a touch-based trigger that discriminates agent from human touches using physical touch attributes and exposes the payload only during agent operation; and (3) one-shot prompt efficacy, a stealthy-guided, character-level
  arXiv  Detail & Related papers  (2025-10-09T05:34:57Z)
• BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks [51.803138848305814]
  We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks.<n>We identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs.<n>Our findings surface both the diversity and brittleness of current web agents.
  arXiv  Detail & Related papers  (2025-10-02T15:22:21Z)
• JU-NLP at Touché: Covert Advertisement in Conversational AI-Generation and Detection Strategies [22.324944278737167]
  It explores how subtle promotional content can be crafted within AI-generated responses and introduces methods to identify and mitigate such covert advertising strategies.<n>For generation (Sub-Task1), we propose a novel framework that leverages user context and query intent to produce contextually relevant advertisements.<n>For detection (Sub-Task2), we explore two effective strategies: a fine-tuned CrossEncoder (textttall-mpnet-base-v2) for direct classification, and a prompt-based reformulation using a fine-tuned textttDeBERTa-v3-base
  arXiv  Detail & Related papers  (2025-09-12T14:53:56Z)
• Hijacking JARVIS: Benchmarking Mobile GUI Agents against Unprivileged Third Parties [19.430061128447022]
  We present the first systematic investigation into the vulnerabilities of mobile GUI agents.<n>We introduce a scalable attack simulation framework AgentHazard, which enables flexible and targeted modifications of screen content.<n>We develop a benchmark suite comprising both a dynamic task execution environment and a static dataset of attack scenarios.<n>Our findings reveal that all examined agents are significantly influenced by misleading third-party content.
  arXiv  Detail & Related papers  (2025-07-06T03:31:36Z)
• OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents [34.396536936282175]
  We introduce OS-Harm, a new benchmark for measuring safety of computer use agents.<n> OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior.<n>We evaluate computer use agents based on a range of frontier models and provide insights into their safety.
  arXiv  Detail & Related papers  (2025-06-17T17:59:31Z)
• Screen Hijack: Visual Poisoning of VLM Agents in Mobile Environments [61.808686396077036]
  We present GHOST, the first clean-label backdoor attack specifically designed for mobile agents built upon vision-language models (VLMs)<n>Our method manipulates only the visual inputs of a portion of the training samples without altering their corresponding labels or instructions.<n>We evaluate our method across six real-world Android apps and three VLM architectures adapted for mobile use.
  arXiv  Detail & Related papers  (2025-06-16T08:09:32Z)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents [74.6761188527948]
  Computer-Use Agents (CUAs) with full system access pose significant security and privacy risks.<n>We investigate Visual Prompt Injection (VPI) attacks, where malicious instructions are visually embedded within rendered user interfaces.<n>Our empirical study shows that current CUAs and BUAs can be deceived at rates of up to 51% and 100%, respectively, on certain platforms.
  arXiv  Detail & Related papers  (2025-06-03T05:21:50Z)
• T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models [88.63040835652902]
  Text to video models are vulnerable to jailbreak attacks, where specially crafted prompts bypass safety mechanisms and lead to the generation of harmful or unsafe content.<n>We propose T2VShield, a comprehensive and model agnostic defense framework designed to protect text to video models from jailbreak threats.<n>Our method systematically analyzes the input, model, and output stages to identify the limitations of existing defenses.
  arXiv  Detail & Related papers  (2025-04-22T01:18:42Z)
• Monitoring Viewer Attention During Online Ads [0.5499796332553706]
  Big brands gauge the liking and purchase intent of their new ads, by analyzing the facial responses of viewers recruited online to watch the ads from home or work.<n>Inattentive participants should get flagged and eliminated to avoid skewing the ad-testing process.<n>We introduce an architecture for monitoring viewer attention during online ads.
  arXiv  Detail & Related papers  (2025-04-08T17:34:02Z)
• Are AI Agents interacting with Online Ads? [0.0]
  This study examines how different AI agents interact with online advertising, whether they incorporate ads into their decision-making processes, and which ad formats prove most effective.<n>We analyze interaction patterns, click behavior, and decision-making strategies through experiments with multimodal language models such as OpenAI GPT-4o, Anthropic Claude 3.7 Sonnet, and Google Gemini 2.0 Flash.
  arXiv  Detail & Related papers  (2025-03-20T08:38:57Z)
• 50 Shades of Deceptive Patterns: A Unified Taxonomy, Multimodal Detection, and Security Implications [33.2581087304465]
  Deceptive patterns (DPs) are user interface designs deliberately crafted to manipulate users into unintended decisions.<n>We have expanded the deceptive pattern taxonomy from security and privacy perspectives, refining its categories and scope.<n>We have developed DPGuard, a novel automatic tool leveraging commercial multimodal large language models (MLLMs) for deceptive pattern detection.
  arXiv  Detail & Related papers  (2025-01-23T03:28:38Z)
• Automatically Detecting Online Deceptive Patterns [24.018376492278033]
  Deceptive patterns in digital interfaces manipulate users into making unintended decisions, exploiting cognitive biases and psychological vulnerabilities.<n>We introduce our AutoBot framework to address this gap and help web stakeholders navigate and mitigate online deceptive patterns.<n>AutoBot accurately identifies and localizes deceptive patterns from a screenshot of a website without relying on the underlying HTML code.
  arXiv  Detail & Related papers  (2024-11-11T23:49:02Z)
• Dynamic Analysis and Adaptive Discriminator for Fake News Detection [59.41431561403343]
  We propose a Dynamic Analysis and Adaptive Discriminator (DAAD) approach for fake news detection.<n>For knowledge-based methods, we introduce the Monte Carlo Tree Search algorithm to leverage the self-reflective capabilities of large language models.<n>For semantic-based methods, we define four typical deceit patterns to reveal the mechanisms behind fake news creation.
  arXiv  Detail & Related papers  (2024-08-20T14:13:54Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models [65.18602126334716]
  Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots. We introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups.
  arXiv  Detail & Related papers  (2024-01-25T03:33:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.