AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA

• URL: http://arxiv.org/abs/2606.19782v1
• Date: Thu, 18 Jun 2026 04:33:07 GMT
• Title: AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA
• Abstract summary: We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification.<n>On FinMME, AgentFinVQA improves $+7.68$ pp over a primary-backbone matched zero-shot baseline with a proprietary backbone.<n>Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures.
• Score: 7.135926644099901
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Financial chart question answering in regulated settings demands more than accuracy: practitioners must know which answers to trust before acting on them, and many institutions cannot send client data to external model providers. Yet existing chart-QA agents are accuracy-focused and opaque, and most assume proprietary API access; to our knowledge, none combines auditability with on-premise deployability without significant accuracy compromise. We present AgentFinVQA, a multi-agent pipeline that decomposes each query into planning, OCR, legend grounding, visual inspection, and verification, recording every step in a traceable Model Evaluation Packet (MEP) per sample. On FinMME, AgentFinVQA improves $+7.68$ pp over a primary-backbone matched zero-shot baseline with a proprietary backbone (Gemini-3 Flash; 71.24% vs. 63.56%, McNemar $p \approx 1.1 \times 10^{-16}$), and $+4.84$ pp with open-weights Qwen3.6-27B-FP8 served locally. The verifier's verdict also serves as a useful confidence signal (68.2% vs. 55.6% exact accuracy on confirmed vs. revised answers), enabling human-in-the-loop review routing. Error analysis shows that question misunderstanding, legend confusion and extraction error account for nearly two-thirds of failures and are the categories least detected by the verifier, identifying clear directions for future work. Together these results show that auditable, on-premise financial chart QA is practical and that the open-weights system keeps most of the accuracy gains while enabling full data residency. We release our code to support reproducible evaluation.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• Towards Expert Financial QA via Self-Improving RAG [2.1042118213583394]
  Self-Improving RAG is a framework that decomposes document QA into three specialized agents.<n>Retrieval, Reasoning, and Judge are coordinated by an orchestrator with feedback-driven self-correction.<n>We evaluate on FinanceBench (SEC filing QA) where Self-Improving RAG achieves 86% oracle-guided accuracy with a 36.4% Lazarus Rate.
  arXiv  Detail & Related papers  (2026-08-27T07:01:41Z)
• ARCHER: Agentic Rule and Compliance Harness for Executable Regulations [0.11862655008303459]
  We introduce ARCHER, a test-driven, deterministically orchestrated multi-agent program-synthesis harness.<n>It generates auditable verification code from regulatory Codes of Practice, enabling transparent, adaptable, and scalable compliance checking.<n>Our cost-accuracy analysis shows that using the ARCHER harness, a self-hosted open-weights model can reach 97.8% of frontier-API accuracy at a quarter of the cost.
  arXiv  Detail & Related papers  (2026-07-28T10:54:28Z)
• Toward Auditable Fraud Detection: Combining Graph Features, Model Explanations, and Agentic Case Investigation [0.0]
  Fraud detection systems must scale with rising transaction volume while remaining explainable and reviewable.<n>We study a layered pipeline that combines a gradient-boosted classifier, graph-derived structural features, an autoencoder-based anomaly signal, and an investigation agent.<n>We conclude that each component contributes only under specific conditions, and that a plausible rationale from an investigation agent is not evidence of a better decision.
  arXiv  Detail & Related papers  (2026-07-21T16:37:41Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• FineVerify: Scaling Test-Time Compute with Fine-Grained Self-Verification for Agentic Search [88.16262636915975]
  FineVerify is a fine-grained self-verification framework for agentic search.<n>It decomposes each question into checkable sub-questions, verifies sampled candidates, and selects the candidate with the highest aggregated score.<n>FineVerify consistently outperforms standard scaling baselines.
  arXiv  Detail & Related papers  (2026-05-30T10:21:20Z)
• Design and Evaluation of Multi-Agent AI Oracle Systems for Prediction Market Resolution [0.0]
  Prediction markets aggregate collective intelligence to forecast uncertain events.<n>Existing oracle systems tradeoff fast but brittle automation against accurate but costly human arbitration.<n>We evaluate whether multi-agent LLM architectures can improve oracle resolution accuracy over single-model baselines.
  arXiv  Detail & Related papers  (2026-05-29T03:44:19Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents [0.7699235580548228]
  LLM agents struggle with regulatory audit replay: when asked to reproduce a transaction flagged decision with identical inputs, most deployments fail to return consistent results.<n>This paper introduces the DeterminismFaithfulness Assurance Harness (DFAH), a framework for measuring trajectory determinism and evidence-conditioned faithfulness in tool-using agents deployed in financial services.
  arXiv  Detail & Related papers  (2026-01-17T19:47:55Z)
• Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving [65.02106674311908]
  This paper introduces Intern-S1-MO, a long-horizon math agent that conducts multi-round hierarchical reasoning.<n>By maintaining a compact memory in the form of lemmas, Intern-S1-MO can more freely explore the lemma-rich reasoning spaces.<n> Experiments show that Intern-S1-MO can obtain 26 out of 35 points on the non-geometry problems of IMO2025, matching the performance of silver medalists.
  arXiv  Detail & Related papers  (2025-12-11T15:26:28Z)
• Mapping Smarter, Not Harder: A Test-Time Reinforcement Learning Agent That Improves Without Labels or Model Updates [1.6440434996206623]
  Enterprise Intelligence Platform must integrate logs from numerous third-party vendors.<n>Vendor documentation is often unavailable at test time.<n>We introduce a reinforcement learning agent that can self-improve without labeled examples.
  arXiv  Detail & Related papers  (2025-10-16T17:17:00Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)
• Applying Machine Learning for Duplicate Detection, Throttling and Prioritization of Equipment Commissioning Audits at Fulfillment Network [1.933681537640272]
  VQ (Vendor Qualification) and IOQ (Installation and Operation Qualification) audits are implemented in warehouses to ensure all equipment being turned over meets the quality standards. In this work, Natural Language Processing and Machine Learning are applied to trim a large checklist dataset for a network of warehouses.
  arXiv  Detail & Related papers  (2022-09-28T20:40:32Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)
• Unsupervised Evaluation for Question Answering with Transformers [46.16837670041594]
  We investigate the hidden representations of questions, answers, and contexts in transformer-based QA architectures. We observe a consistent pattern in the answer representations, which we show can be used to automatically evaluate whether or not a predicted answer is correct. We are able to predict whether or not a model's answer is correct with 91.37% accuracy SQuAD, and 80.7% accuracy on SubjQA.
  arXiv  Detail & Related papers  (2020-10-07T07:03:30Z)
• Selective Question Answering under Domain Shift [90.021577320085]
  Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. We train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.
  arXiv  Detail & Related papers  (2020-06-16T19:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.