FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents

• URL: http://arxiv.org/abs/2608.11683v1
• Date: Wed, 12 Aug 2026 05:43:26 GMT
• Title: FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents
• Abstract summary: We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow.<n>We find that Samaya's in-house system leads at 56.2%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost.<n>The best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost.
• Score: 5.045937000137832
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI agents are increasingly deployed for professional investment research, yet no benchmark captures the complexity of the full investor workflow. Existing benchmarks mainly target financial data extraction, a narrow slice that current models have largely saturated, while reference-based metrics and generic LLM-as-a-judge scoring fall short on the open-ended, long-form answers that real analyst queries demand. We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow. FrontierFinance is both broader and harder than existing public finance benchmarks. Evaluating frontier models and agent systems under a common harness restricted to publicly available data, we find that the tool harness, not the model alone, strongly shapes quality and efficiency; that Samaya's in-house system leads at 56.0%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost; and that the best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost. Screening & Discovery and Sector, Industry & Macro remain the hardest use cases across all systems, where even the best systems reach only 33% and 39%. We make the dataset and grading code publicly available.

Related papers

• FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models [53.68149869349268]
  Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research.<n>We introduce ourbenchmark, an expert-validated benchmark for evaluating LLMs on frontier, end-to-end TCS research.
  arXiv  Detail & Related papers  (2026-08-20T15:13:41Z)
• Can Open-Weight Models Compete on Financial Text Comprehension? [0.0]
  Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months.<n>Yet their reliability on real-world financial tasks remains largely untested.<n>We update the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports.
  arXiv  Detail & Related papers  (2026-08-09T10:51:52Z)
• FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction [57.909515451254855]
  FinDeepIndicator is the first benchmark dedicated to evaluating Deep Research (DR) agents in end-to-end financial indicator construction.<n>It evaluates DR agents across four stages in indicator construction: formula specification, data collection, indicator calculation, and answer generation.<n>It contains 3,350 curated question-answer pairs derived from both U.S. and Chinese markets, 10 years of historical financial data, and 800 listed companies.
  arXiv  Detail & Related papers  (2026-08-01T16:54:41Z)
• IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO [0.0]
  Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating finance language models.<n>We introduce IPO Finance Agent, which extends the Finance Agent framework along two directions: task domain and retrieval architecture.<n>Best-performing evaluated model, Zhipu GLM-5.2, reaches 79.8% accuracy, and the most cost-efficient model on the resulting frontier, Xiaomi MiMo-2.5 Pro, reaches slightly lower accuracy (77.2%) at 0.05 USD per query.
  arXiv  Detail & Related papers  (2026-06-22T08:42:19Z)
• SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks [65.14594927081983]
  We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning.<n>On a 240-question test set spanning SGI-Reasoning and Scientists' First exam, SciOrch reaches 56.66% average accuracy.<n>It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.
  arXiv  Detail & Related papers  (2026-06-14T15:45:34Z)
• Agents' Last Exam [275.79071391137074]
  Agents' Last Exam (ALE) is a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes.<n>Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018.<n>Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%.
  arXiv  Detail & Related papers  (2026-06-03T20:20:46Z)
• Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs [86.49905745865038]
  Soohak is a 439-problem benchmark newly authored from scratch by 64 mathematicians.<n>The dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.
  arXiv  Detail & Related papers  (2026-05-09T17:14:22Z)
• The AI Productivity Index (APEX) [4.122962658725304]
  We introduce the first version of the AI Productivity Index (APEX), a benchmark for assessing whether frontier AI models can perform knowledge work with high economic value.<n>APEX-v1.0 contains 200 test cases and covers four domains: investment banking, management consulting, law, and primary medical care.<n>We evaluate 23 frontier models on APEX-v1.0 using an LM judge. GPT 5 (Thinking = High) achieves the highest mean score (64.2%), followed by Grok 4 (61.3%) and Gemini 2.5 Flash (Thinking = On) (60.4%)
  arXiv  Detail & Related papers  (2025-09-30T03:26:17Z)
• BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs [7.9458352414205295]
  Large language models excel in general tasks, yet assessing their reliability in logic-heavy, precision-critical domains like finance, law, and healthcare remains challenging.<n>We introduce BizFinBench, the first benchmark specifically designed to evaluate LLMs in real-world financial applications.<n> BizFinBench consists of 6,781 well-annotated queries in Chinese, spanning five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering.
  arXiv  Detail & Related papers  (2025-05-26T03:23:02Z)
• Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications [88.96861155804935]
  We introduce textitOpen-FinLLMs, the first open-source multimodal financial LLMs.<n>FinLLaMA is pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs.<n>We evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings.
  arXiv  Detail & Related papers  (2024-08-20T16:15:28Z)
• BloombergGPT: A Large Language Model for Finance [42.73350054822628]
  We present BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data. We construct a 363 billion token dataset based on Bloomberg's extensive data sources, augmented with 345 billion tokens from general purpose datasets. Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins.
  arXiv  Detail & Related papers  (2023-03-30T17:30:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.