When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets

• URL: http://arxiv.org/abs/2510.00332v1
• Date: Tue, 30 Sep 2025 22:39:06 GMT
• Title: When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets
• Authors: Zeshi Dai, Zimo Peng, Zerui Cheng, Ryan Yihe Li,
• Abstract summary: We present CAIA, a benchmark exposing a critical blind spot in AI evaluation.<n>We evaluate 17 models on 178 time-anchored tasks requiring agents to distinguish truth from manipulation.<n>Our results reveal a fundamental capability gap: without tools, even frontier models achieve only 28% accuracy on tasks junior analysts routinely handle.
• Score: 0.3069921776214295
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present CAIA, a benchmark exposing a critical blind spot in AI evaluation: the inability of state-of-the-art models to operate in adversarial, high-stakes environments where misinformation is weaponized and errors are irreversible. While existing benchmarks measure task completion in controlled settings, real-world deployment demands resilience against active deception. Using crypto markets as a testbed where $30 billion was lost to exploits in 2024, we evaluate 17 models on 178 time-anchored tasks requiring agents to distinguish truth from manipulation, navigate fragmented information landscapes, and make irreversible financial decisions under adversarial pressure. Our results reveal a fundamental capability gap: without tools, even frontier models achieve only 28% accuracy on tasks junior analysts routinely handle. Tool augmentation improves performance but plateaus at 67.4% versus 80% human baseline, despite unlimited access to professional resources. Most critically, we uncover a systematic tool selection catastrophe: models preferentially choose unreliable web search over authoritative data, falling for SEO-optimized misinformation and social media manipulation. This behavior persists even when correct answers are directly accessible through specialized tools, suggesting foundational limitations rather than knowledge gaps. We also find that Pass@k metrics mask dangerous trial-and-error behavior for autonomous deployment. The implications extend beyond crypto to any domain with active adversaries, e.g. cybersecurity, content moderation, etc. We release CAIA with contamination controls and continuous updates, establishing adversarial robustness as a necessary condition for trustworthy AI autonomy. The benchmark reveals that current models, despite impressive reasoning scores, remain fundamentally unprepared for environments where intelligence must survive active opposition.

Related papers

• Implicit Intelligence -- Evaluating Agents on What Users Don't Say [0.3580891736370874]
  Implicit Intelligence is an evaluation framework testing whether AI agents can move beyond prompt-following to become genuine goal-fulfillers.<n>Our scenarios feature apparent simplicity in user requests, hidden complexity in correct solutions, and discoverability of constraints through environmental exploration.
  arXiv  Detail & Related papers  (2026-02-23T23:46:55Z)
• MAIF: Enforcing AI Trust and Provenance with an Artifact-Centric Agentic Paradigm [0.5495755145898128]
  Current AI systems operate on opaque data structures that lack the audit trails, provenance tracking, or explainability required by emerging regulations like the EU AI Act.<n>We propose an artifact-centric AI agent paradigm where behavior is driven by persistent, verifiable data artifacts rather than ephemeral tasks.<n>Production-ready implementation demonstrates ultra-high-speed streaming (2,720.7 MB/s), optimized video processing (1,342 MB/s), and enterprise-grade security.
  arXiv  Detail & Related papers  (2025-11-19T04:10:32Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testing [23.554239007767276]
  We introduce the first real-world, agent-oriented pentesting benchmark, TermiBench.<n>We propose TermiAgent, a multi-agent penetration testing framework.<n>In evaluations, our work outperforms state-of-the-art agents, exhibiting stronger penetration testing capability.
  arXiv  Detail & Related papers  (2025-09-11T07:30:44Z)
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $τ$-bench [58.114899897566964]
  In a multi-turn conversational environment, large language models (LLMs) often struggle with consistent reasoning and adherence to domain-specific policies.<n>We propose the Input-Reformulation Multi-Agent (IRMA) framework, which automatically reformulates user queries augmented with relevant domain rules.<n>IRMA significantly outperforms ReAct, Function Calling, and Self-Reflection by 16.1%, 12.7%, and 19.1%, respectively.
  arXiv  Detail & Related papers  (2025-08-28T15:57:33Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [101.86739402748995]
  We run the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios.<n>We build the Agent Red Teaming benchmark and evaluate it across 19 state-of-the-art models.<n>Our findings highlight critical and persistent vulnerabilities in today's AI agents.
  arXiv  Detail & Related papers  (2025-07-28T05:13:04Z)
• Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks [1.3124479769761592]
  We introduce a novel prototype designed to employ Large Language Model (LLM)-driven autonomous systems.<n>Our system represents the first demonstration of a fully autonomous, LLM-driven framework capable of compromising accounts.<n>We find that the associated costs are competitive with, and often significantly lower than, those incurred by professional human pen-testers.
  arXiv  Detail & Related papers  (2025-02-06T17:12:43Z)
• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents [44.34340798542]
  Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning. Traditional supervised pre-training on static datasets falls short in enabling autonomous agent capabilities. We propose a framework that combines guided Monte Carlo Tree Search (MCTS) search with a self-critique mechanism and iterative fine-tuning on agent interactions.
  arXiv  Detail & Related papers  (2024-08-13T20:52:13Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• TeD-SPAD: Temporal Distinctiveness for Self-supervised Privacy-preservation for video Anomaly Detection [59.04634695294402]
  Video anomaly detection (VAD) without human monitoring is a complex computer vision task. Privacy leakage in VAD allows models to pick up and amplify unnecessary biases related to people's personal information. We propose TeD-SPAD, a privacy-aware video anomaly detection framework that destroys visual private information in a self-supervised manner.
  arXiv  Detail & Related papers  (2023-08-21T22:42:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.