IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO

• URL: http://arxiv.org/abs/2606.23032v3
• Date: Tue, 30 Jun 2026 10:07:05 GMT
• Title: IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO
• Abstract summary: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating finance language models.<n>We introduce IPO Finance Agent, which extends the Finance Agent framework along two directions: task domain and retrieval architecture.<n>Best-performing evaluated model, Zhipu GLM-5.2, reaches 79.8% accuracy, and the most cost-efficient model on the resulting frontier, Xiaomi MiMo-2.5 Pro, reaches slightly lower accuracy (77.2%) at 0.05 USD per query.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language models on financial tasks. However, it narrowly deals with periodic reporting from publicly traded companies (SEC 10-K and 10-Q filings), and its agentic harness relies on naive, unenriched chunk retrieval. Neither the task design nor the retrieval approach addresses the distinct challenges of IPO due diligence. SEC S-1 filings combine historical financial statements, governance structures, pro forma and common-control accounting treatments, capital-formation narratives, and underwriting-sensitive risk disclosures within substantially longer documents than typical periodic filings. That is why we introduce IPO Finance Agent, which extends the Finance Agent v2 framework along two directions: task domain and retrieval architecture. During our experiments, the original Finance Agent v2 harness basically failed to deliver any output related to the SpaceX S-1 filing, due to document length. We therefore had to improve the agentic harness with contextual retrieval, a more realistic and industry-standard approach for long documents. We also built a dataset of 1,000 IPO-diligence questions, and publicly release 70 questions on the SpaceX (SPCX) S-1 filing to support reproducibility, while the remainder are held private to guard against benchmark contamination. In addition, we introduce an evaluator-optimizer pipeline to automatically generate evaluation rubrics for the benchmark: candidate facts are extracted from model answers, consolidated into draft criteria, then automatically audited for omissions, hallucinations, mistiered items, and redundancy, with LLM feedback driving iterative repair, targeted enrichment, and deduplication. Human experts only review final rubrics before deployment. Results show that the best-performing evaluated model, Zhipu GLM-5.2, reaches 79.8% accuracy, and the most cost-efficient model on the resulting Pareto frontier, Xiaomi MiMo-2.5 Pro, reaches slightly lower accuracy (77.2%) at 0.05 USD per query, while exceeding the current Finance Agent v2 leaderboard ceiling, Google Gemini 3.5 Flash at 57.9% for 2.51 USD per query, and undercutting even FABv2's cheapest entry (MiniMax M3: 48.3% at 0.32 USD) on cost-efficiency. Code and data are released on GitHub https://github.com/benstaf/ipoagent

Related papers

• FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models [53.68149869349268]
  Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research.<n>We introduce ourbenchmark, an expert-validated benchmark for evaluating LLMs on frontier, end-to-end TCS research.
  arXiv  Detail & Related papers  (2026-08-20T15:13:41Z)
• Mint-Agent: Introducing Finance-Native Agentic Foundation Models [36.08481007458733]
  Mint-Agent is a family of finance-native agentic models designed around two scales of financial intelligence.<n>Our data engine constructs clean, specialized tasks for atomic financial capabilities and long-horizon agentic execution from real-world financial sources.<n>Our training recipe combines SFT, critical-step OPD, and RLVR to develop separate financial reasoning and agentic execution experts.
  arXiv  Detail & Related papers  (2026-08-17T10:38:54Z)
• FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents [5.045937000137832]
  We introduce FrontierFinance, a fully open benchmark of 220 expert-crafted queries and 11,543 source-attributed rubrics spanning six crucial use cases across the full investor workflow.<n>We find that Samaya's in-house system leads at 56.2%, ahead of the strongest frontier model (Claude Fable 5, 49.2%) at roughly 2.2x lower cost.<n>The best open-weight model (Kimi K3, 46.4%) nearly matches the best proprietary model at 4.5x lower cost.
  arXiv  Detail & Related papers  (2026-08-12T05:43:26Z)
• FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction [57.909515451254855]
  FinDeepIndicator is the first benchmark dedicated to evaluating Deep Research (DR) agents in end-to-end financial indicator construction.<n>It evaluates DR agents across four stages in indicator construction: formula specification, data collection, indicator calculation, and answer generation.<n>It contains 3,350 curated question-answer pairs derived from both U.S. and Chinese markets, 10 years of historical financial data, and 800 listed companies.
  arXiv  Detail & Related papers  (2026-08-01T16:54:41Z)
• AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA [7.135926644099901]
  We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification.<n>On FinMME, AgentFinVQA improves $+7.68$ pp over a primary-backbone matched zero-shot baseline with a proprietary backbone.<n>Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures.
  arXiv  Detail & Related papers  (2026-06-18T04:33:07Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• AgenticRAG: Agentic Retrieval for Enterprise Knowledge Bases [0.7176906280023592]
  We present AgenticRAG, a practical agentic harness for retrieval and analysis over enterprise knowledge bases.<n>Our approach reduces this overdependence by layering a lightweight harness on top of existing enterprise search infrastructure.<n>Our results demonstrate its suitability for real-world enterprise production environments.
  arXiv  Detail & Related papers  (2026-05-07T00:39:22Z)
• Agentic Retrieval-Augmented Generation for Financial Document Question Answering [7.56842616602779]
  FinAgent-RAG is an agentic RAG framework that orchestrates iterative retrieval-reasoning loops with self-verification.<n>It integrates a Contrastive Financial Retriever trained with hard negative mining to distinguish semantically similar but numerically distinct financial passages.<n>It achieves 76.81%, 78.46%, and 74.96% execution accuracy respectively, outperforming the strongest baseline by 5.62--9.32 percentage points.
  arXiv  Detail & Related papers  (2026-05-06T19:59:51Z)
• Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies [0.0]
  We compare four multi-agent orchestration architectures for extracting structured information from financial documents.<n>We find that reflexive architectures achieve the highest field-level F1 (0.943) but at 2.3x the cost of sequential baselines.<n>We present ablation studies on semantic caching, model routing and adaptive retry strategies, demonstrating that hybrid configurations can recover 89% of the reflexive architecture's accuracy gains at only 1.15x baseline cost.
  arXiv  Detail & Related papers  (2026-03-24T00:02:47Z)
• Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models [0.0]
  We present the first systematic evaluation comparing vector-based agentic RAG using hybrid search and metadata filtering.<n>We measure retrieval metrics (MRR, Recall@5), answer quality through LLM-as-a-judge pairwise comparisons, latency, and preprocessing costs.<n>Our findings reveal that applying advanced RAG techniques to financial Q&A systems improves retrieval accuracy, answer quality, and has cost-performance tradeoffs to be considered in production.
  arXiv  Detail & Related papers  (2025-11-22T20:06:25Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents [74.55061622246824]
  Agent Market Arena (AMA) is the first lifelong, real-time benchmark for evaluating Large Language Model (LLM)-based trading agents.<n>AMA integrates verified trading data, expert-checked news, and diverse agent architectures within a unified trading framework.<n>It evaluates agents across GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash.
  arXiv  Detail & Related papers  (2025-10-13T17:54:09Z)
• FinReflectKG - MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence [0.0]
  We present FinReflectKG - MultiHop, a benchmark built on FinReflectKG, a temporally indexed financial Knowledge Graph.<n>We generate financial analyst style questions with exact supporting evidence from the KG.<n>Across both reasoning and non-reasoning models, KG-guided precise retrieval yields substantial gains.
  arXiv  Detail & Related papers  (2025-10-03T11:19:31Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.