HLL: Can Agents Cross Humanity's Last Line of Verification?

• URL: http://arxiv.org/abs/2606.02449v1
• Date: Mon, 01 Jun 2026 16:20:45 GMT
• Title: HLL: Can Agents Cross Humanity's Last Line of Verification?
• Abstract summary: We introduce textbfHumanity's Last Line of Verification (HLL) to evaluate whether agents can cross this boundary through grounded, human-like interaction.<n>HLL covers diverse CAPTCHA interactions and exposes agents to controlled realism stressors.<n>Results show that current agents remain brittle at this human-substitution boundary.
• Score: 27.923163104768758
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal agents are increasingly expected to operate interfaces on behalf of users, raising a central deployment question: can they truly substitute for humans in workflows that services deliberately protect against automation? CAPTCHA verification makes this question concrete. It is not merely a visual puzzle, but a human-verification boundary placed before account creation, content access, form submission, and other protected actions. We introduce \textbf{Humanity's Last Line of Verification (HLL)}, a controlled benchmark that uses interactive CAPTCHA verification to evaluate whether agents can cross this boundary through grounded, human-like interaction rather than recognition alone. HLL covers diverse CAPTCHA interactions and exposes agents to controlled realism stressors, including cluttered webpages, harder task variants, and trace-conditioned validation of the solving process. We evaluate eight frontier multimodal agents in a closed-loop GUI environment. The results show that current agents remain brittle at this human-substitution boundary: performance varies sharply across verification types, degrades under realistic interface conditions, and drops further when correct answers must be supported by valid action traces. By exposing gaps in localization, action calibration, state tracking, and process consistency, HLL provides a concrete testbed for measuring how close multimodal agents are to acting as human substitutes in protected real-world workflows. Our code is available at https://github.com/XinhaoS0101/HLL

Related papers

• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• Collaborative Human-Agent Protocol (CHAP) [0.18352113484137622]
  This paper presents CHAP, the Collaborative Human-Agent Protocol.<n>Under CHAP, the override that used to vanish into a chat thread becomes a structured event carrying a diff, a rationale, and a content hash.<n>The protocol achieves this through a small Core (workspaces, participants, tasks, artefacts, and an append-only evidence log) together with composable profiles that add review, modes, routing, deliberation, handoff, identity, signatures, and transparency-backed audit as deployments require them.
  arXiv  Detail & Related papers  (2026-06-08T17:11:42Z)
• From Privacy to Workflow Integrity: Communication-Graph Metadata in Autonomous Agent Interoperability [0.0]
  A2A and MCP protocols standardize what agents say to one another, but assume address-based transport over HTTP(S)<n>What they leave in the clear is the communication graph: which agent contacts which, when, and how often.<n>We give a threat model for the agent communication graph; identify what makes agent metadata distinctively revealing.
  arXiv  Detail & Related papers  (2026-06-05T11:07:55Z)
• ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use [40.024131729499494]
  We show that agents can exhibit misaligned behavior even in benign settings, taking unsafe actions when those actions are instrumental to task completion.<n>We study this failure mode through the lens of corrigibility, the safety desideratum that agents remain amenable to human correction, interruption, or shutdown.<n>Our work highlights the critical need for principled, corrigibility-focused alignment methods in autonomous agents.
  arXiv  Detail & Related papers  (2026-05-29T20:29:35Z)
• Specifying AI-SDLC Processes: A Protocol Language for Human-Agent Boundaries [0.0]
  We propose a domain-specific language for specifying AI-SDLC processes as protocols.<n>The language distinguishes policy (declared intent) from mechanism (structural enforcement)<n>A working implementation demonstrates feasibility; empirical evaluation is future work.
  arXiv  Detail & Related papers  (2026-05-24T02:52:47Z)
• IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Construction [72.51952455865155]
  We present IMPACT-HOI, a mixed-initiative framework for annotating egocentric procedural video.<n> IMPACT-HOI frames this task as the incremental resolution of a partially specified, onset-anchored event state.<n>A user study with 9 participants shows a 13.5% reduction in manual annotation actions, a 46.67% event match rate, and zero confirmed-field violations.
  arXiv  Detail & Related papers  (2026-05-03T01:37:40Z)
• AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation [71.49152943451328]
  We introduce a benchmark AJ-Bench to evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories.<n>Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification.
  arXiv  Detail & Related papers  (2026-04-20T13:23:38Z)
• Towards Unconstrained Human-Object Interaction [55.123145316966635]
  Human-Object Interaction (HOI) detection is a longstanding computer vision problem.<n>With the advent of Multimodal Large Language Models (MLLMs) it has become feasible to explore more flexible paradigms for interaction recognition.<n>In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection.
  arXiv  Detail & Related papers  (2026-04-15T16:39:28Z)
• Governing Dynamic Capabilities: Cryptographic Binding and Reproducibility Verification for AI Agent Tool Use [0.0]
  No existing security layer verifies what an AI agent can do, whether it executed what it claims, or what happened in a multi-agent interaction.<n>Existing frameworks conflate the two, enabling silent capability escalation and leaving interactions without verifiable provenance.<n>We derive three Agent Governance Requirements: capability integrity (G1), behavioral verifiability (G2), and interaction auditability (G3)<n>We validate with two crypto-agnostic instantiations -- basic (Ed25519, SHA-256; 97 us verify) and enhanced (BBS+ selective disclosure, Groth16 DV-SNARK; 13.8 ms)
  arXiv  Detail & Related papers  (2026-03-15T11:46:57Z)
• aCAPTCHA: Verifying That an Entity Is a Capable Agent via Asymmetric Hardness [5.119165077129504]
  "Is this entity an AI agent?" is a new entity-type verification problem with no established solution.<n>ACAPTCHA provides a composable, infrastructure-free admission gate for any service where entity-type verification is required.
  arXiv  Detail & Related papers  (2026-03-07T09:00:04Z)
• Towards a Humanized Social-Media Ecosystem: AI-Augmented HCI Design Patterns for Safety, Agency & Well-Being [0.0]
  Social platforms connect billions of people, yet their engagement-first algorithms often work on users rather than with them.<n>We propose Human-Layer AI (HL-AI)--user-owned, explainable intermediaries that sit in the browser between platform logic and the interface.<n>HL-AI gives people practical, moment-to-moment control without requiring platform cooperation.
  arXiv  Detail & Related papers  (2025-11-08T06:22:15Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation [61.68049335444254]
  Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments. We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP) With our technical design, our agent achieves new state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios.
  arXiv  Detail & Related papers  (2024-02-19T08:29:03Z)
• Evaluating Multimodal Interactive Agents [14.936474798718653]
  We present a novel approach to evaluation called the Standardised Test Suite (STS) The STS uses behavioural scenarios mined from real human interaction data. It is fast, controlled, interpretable, and representative of naturalistic interactions.
  arXiv  Detail & Related papers  (2022-05-26T11:18:09Z)
• Auditing AI models for Verified Deployment under Semantic Specifications [65.12401653917838]
  AuditAI bridges the gap between interpretable formal verification and scalability. We show how AuditAI allows us to obtain controlled variations for verification and certified training while addressing the limitations of verifying using only pixel-space perturbations.
  arXiv  Detail & Related papers  (2021-09-25T22:53:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.