Herculean: An Agentic Benchmark for Financial Intelligence
- URL: http://arxiv.org/abs/2605.14355v2
- Date: Thu, 21 May 2026 05:16:58 GMT
- Title: Herculean: An Agentic Benchmark for Financial Intelligence
- Abstract summary: We introduce Herculean, the first skilled benchmark for agentic financial intelligence spanning four representative skill sets.<n>Across frontier agents, we find agents perform relatively well on Trading and Market Insights, but struggle substantially on Hedging and Auditing.<n>Overall, our results point to a key gap in current agents in turning financial reasoning into dependable workflow execution.
- Score: 114.33879935277217
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: As AI agents improve, the central question is no longer whether they can solve isolated well-defined financial tasks, but whether they can reliably carry out financial professional work. Existing financial benchmarks offer only a partial view of this ability, as they primarily evaluate static competencies such as question answering, retrieval, summarization, and classification. We introduce Herculean, the first skilled benchmark for agentic financial intelligence spanning four representative workflows, including Trading, Hedging, Market Insights, and Auditing. Each workflow is instantiated as a standardized MCP-based skill environment with its own tools, interaction dynamics, constraints, and success criteria, enabling consistent end-to-end assessment of heterogeneous agent systems. Across frontier agents, we find agents perform relatively well on Trading and Market Insights, but struggle substantially on Hedging and Auditing, where long-horizon coordination, state consistency, and structured verification are critical. Overall, our results point to a key gap in current agents in turning financial reasoning into dependable workflow execution in high-stakes financial workflows.
Related papers
- StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows [63.57092302641323]
We introduce textbfStartupBench, an E2E agent benchmark grounded in market-validated AI startup products.<n>We study AI products with demonstrated adoption, together with their product and users, to identify real-world tasks for which AI has established practical demand.<n>Our results reveal that many market-validated agents remain beyond the reliable capabilities of current general-purpose agents.
arXiv Detail & Related papers (2026-08-18T14:01:32Z) - FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance [0.0]
We introduce FORCE-Bench, which contains 251 expert-annotated queries and evaluates responses using a calibrated-based framework.<n>FORCE-Bench assesses agentic systems on three task types: financial obligation research, financial entity performance research, and business brief generation.<n>We release the dataset, rubrics, harness, and analysis code as open-source to support reproducible comparison and adaptation to other enterprise finance environments.
arXiv Detail & Related papers (2026-07-11T01:16:31Z) - OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents [11.33449680384001]
We introduce OpenFinGym, a unified gym environment for quantitative-finance agent development.<n>It covers forecasting, market generation, real-time trading, and fraud detection under a single execution and verification interface.<n>OpenFinGym additionally provides an automated task-construction pipeline that turns quantitative finance publications into executable task packages.
arXiv Detail & Related papers (2026-06-24T19:42:55Z) - Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
arXiv Detail & Related papers (2026-05-17T10:26:37Z) - FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models [0.33504916505452914]
FINESSE-Bench is a suite of eight specialized benchmarks comprising 3,993 questions for hierarchical evaluation of financial competencies in large language models.<n>We describe a unified evaluation protocol covering multiple-choice questions, numerical answers, and short open-ended responses.
arXiv Detail & Related papers (2026-05-14T23:53:51Z) - FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks [4.444378412245679]
We introduce FrontierFinance, a benchmark of 25 complex financial modeling tasks across five core finance models.<n>We engage human experts to define the tasks, create rubrics, grade LLMs, and perform the tasks themselves as human baselines.<n>We demonstrate that our human experts both receive higher scores on average, and are more likely to provide client-ready outputs than current state-of-the-art systems.
arXiv Detail & Related papers (2026-04-07T14:15:45Z) - FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
arXiv Detail & Related papers (2025-08-07T22:15:22Z) - Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning [12.548390779247987]
We introduce the Agentar-Fin-R1 series of financial large language models.<n>Our optimization approach integrates a high-quality, systematic financial task label system.<n>Our models undergo comprehensive evaluation on mainstream financial benchmarks.
arXiv Detail & Related papers (2025-07-22T17:52:16Z) - FinHEAR: Human Expertise and Adaptive Risk-Aware Temporal Reasoning for Financial Decision-Making [58.04602111184477]
FinHEAR is a framework for Human Expertise and Adaptive Risk-aware reasoning.<n>It orchestrates specialized agents to analyze historical trends, interpret current events, and retrieve expert-informed precedents.<n> Empirical results on financial datasets show that FinHEAR consistently outperforms strong baselines across trend prediction and trading tasks.
arXiv Detail & Related papers (2025-06-10T04:06:51Z) - AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [74.16170899755281]
We introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents.<n>AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit.<n>This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront.
arXiv Detail & Related papers (2024-01-24T01:51:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.