FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detection

• URL: http://arxiv.org/abs/2606.16659v1
• Date: Mon, 15 Jun 2026 12:53:55 GMT
• Title: FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detection
• Abstract summary: FraudSMSWalker is a benchmark for URL-masked SMS-to-webpage fraud judgment.<n>It contains 699 bilingual chains, including 332 fraudulent and 367 benign cases, across ten service scenarios.<n>Results show that current agents can detect suspicious cues, but struggle to preserve benign recall and often produce positive predictions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: SMS fraud is increasingly cross-channel: a message directs the user to a webpage, and the final risk depends on how the SMS claim aligns with the page content and requested user action. However, existing evaluations either focus on message-only smishing classification or expose URL and domain cues that allow models to rely on reputation shortcuts. To address this gap, we introduce \textbf{FraudSMSWalker}, a controlled benchmark for URL-masked SMS-to-webpage fraud judgment. FraudSMSWalker contains 699 bilingual chains, including 332 fraudulent and 367 benign cases, across ten service scenarios. The model-visible input consists of the SMS context and sanitized webpage evidence, while raw URLs, hosts, domains, IPs, redirects, and reputation metadata are withheld. The benchmark further includes hard benign cases whose pages contain login, payment, verification, or account-management elements that are plausible under the service context but also appear in scam flows. We evaluate nine web agents under masked browser-agent protocols and conduct URL-visibility ablations. The results show that current agents can detect suspicious cues, but struggle to preserve benign recall and often produce positive predictions that are weakly supported by the observed evidence. These findings position FraudSMSWalker as a benchmark for measuring whether web agents can make fraud judgments that remain both accurate and evidence-grounded when direct reputation shortcuts are suppressed. The associated code and dataset are accessible at the \href{https://anonymous.4open.science/w/FraudMessageWalker-Bench}{anonymous link}.

Related papers

• RiskChainBench: A Benchmark for Obfuscated Platform Message Restoration and Evidence-Grounded Web Investigation [32.78931752588511]
  We introduce RiskChainBench, pairing 3,600 synthetic token-text restoration inputs from 600 source sessions with 600 corresponding human-labeled local web environments.<n>A model first restores the message, operational intent, and destination; the same underlying model then acts as a VLM-driven web agent.<n>We score restoration and correct-routing web investigation separately and compose them offline by applying the frozen primary-entry prediction as a gate to the same Task 2 result.
  arXiv  Detail & Related papers  (2026-09-15T09:32:45Z)
• Breadcrumbing Search Agents [118.20284823674382]
  LLM-based search agents are widely used for information-seeking tasks, but their reliance on external tool returns introduces a critical security risk.<n>We show that the channel delivering search and page observations is a fragile security boundary.<n>Under a constrained tool-intermediary threat model, appending only one controlled result per query can substantially increase attack success.
  arXiv  Detail & Related papers  (2026-08-05T07:57:27Z)
• Send and Pretend: Exploiting Transcript Consistency Issues in End-to-End Encrypted Group Chats [1.8831329126144525]
  Threema, WhatsApp, Signal and iMessage have recently introduced polls to aid agreement processes in groups.<n>This implicitly sets the expectation that all participants see the same outcome and thus have the same view of the conversation.<n>In this work, we demonstrate that today's major E2EE messengers do not guarantee any form of transcript consistency for group chats.
  arXiv  Detail & Related papers  (2026-07-29T22:54:21Z)
• RECEIPT: Deterministic, Reward-Hacking-Resistant Verification for White-Box Agentic XSS Discovery [8.773382962696136]
  Cross-Site Scripting (XSS) remains one of the most prevalent and damaging classes of web vulnerabilities.<n>We present RECEIPT, a verification framework that makes agent-reported XSS findings trustworthy.
  arXiv  Detail & Related papers  (2026-07-20T23:16:16Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity? [54.701199583505144]
  Cross-application access creates a privacy risk that has been largely overlooked.<n>We introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios.<n>We evaluate 15 frontier agents and find a surprisingly high failure rate.
  arXiv  Detail & Related papers  (2026-06-22T11:36:58Z)
• Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents [93.19140872946842]
  Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments.<n>This makes them vulnerable to prompt-injection attacks, in which seemingly benign content embeds adversarial instructions that manipulate agent behaviour.<n>We introduce textbfsysname, a benchmark to systematically categorize and attribute harm in real-world web agent systems.
  arXiv  Detail & Related papers  (2026-06-11T14:12:43Z)
• The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck [18.14752683836217]
  textscPACT is a runtime monitor that assigns semantic roles to tool arguments.<n>textscPACT reframes agent security as authority binding.
  arXiv  Detail & Related papers  (2026-05-11T04:09:11Z)
• TraceScope: Interactive URL Triage via Decoupled Checklist Adjudication [14.375064108289115]
  We present TraceScope, a decoupled triage pipeline that operationalizes this workflow at scale.<n>To prevent the observer effect and ensure safety, a sandboxed operator agent drives a real GUI browser guided by visual motivation to elicit page behavior.<n>An adjudicator agent circumvents LLM context limitations by querying evidence on demand to verify a MITRE ATT&CK checklist, and generates an audit-ready report with extracted indicators of compromise (IOCs)<n>Our evaluation reveals that TraceScope demonstrates superior performance in a real-world scenario as well, successfully detecting sophisticated phishing attempts that current state-of-the-art defenses fail to identify
  arXiv  Detail & Related papers  (2026-04-23T16:31:42Z)
• MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs [16.403811916501077]
  We propose MalURLBench, the first benchmark for evaluating web agents' vulnerabilities to malicious URLs.<n>MalURLBench contains 61,845 attack instances spanning 10 real-world scenarios and 7 categories of real malicious websites.<n> Experiments with 12 popular LLMs reveal that existing models struggle to detect elaborately disguised malicious URLs.
  arXiv  Detail & Related papers  (2026-01-26T03:58:10Z)
• Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety [9.588706798051955]
  Multimodal large language models (MLLMs) are increasingly positioned as AI collaborators for building complex web-related applications.<n>We introduce a comprehensive web understanding benchmark, named WebRSSBench, that jointly evaluates Reasoning, Robustness, and Safety across eight tasks.<n>The benchmark is constructed from 729 websites and contains 3799 question answer pairs that probe multi-step inference over page structure, text, widgets, and safety-critical interactions.
  arXiv  Detail & Related papers  (2025-09-26T02:38:14Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• Verifying the Robustness of Automatic Credibility Assessment [50.55687778699995]
  We show that meaning-preserving changes in input text can mislead the models. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on misinformation detection tasks. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions.
  arXiv  Detail & Related papers  (2023-03-14T16:11:47Z)
• Robust and Verifiable Information Embedding Attacks to Deep Neural Networks via Error-Correcting Codes [81.85509264573948]
  In the era of deep learning, a user often leverages a third-party machine learning tool to train a deep neural network (DNN) classifier. In an information embedding attack, an attacker is the provider of a malicious third-party machine learning tool. In this work, we aim to design information embedding attacks that are verifiable and robust against popular post-processing methods.
  arXiv  Detail & Related papers  (2020-10-26T17:42:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.