FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking

• URL: http://arxiv.org/abs/2605.05482v1
• Date: Wed, 06 May 2026 22:04:44 GMT
• Title: FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking
• Abstract summary: We present a unified data-efficient framework for training large language models (LLMs)<n>We optimize answer quality, citation grounding, calibrated and refusal under real-world deployment constraints.<n>The system is deployed at 40+ financial institutions, achieving a 7.1 percentage point improvement in query resolution.
• Score: 0.6356558256426569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are rapidly being adopted across various domains. However, their adoption in banking industry faces resistance due to demands for high accuracy, regulatory compliance, and the need for verifiable and grounded responses. We present a unified, data-efficient framework for training grounded domain-specific LLMs that optimizes answer quality, citation grounding, and calibrated refusal under real-world deployment constraints. First, we describe a data generation pipeline that combines LLM-as-a-Judge filtering, citation annotation, and curriculum learning with only 143M tokens. The resulting 12B model achieves high answer quality outperforming GPT-4.1 on citation grounding, with a modest citation tradeoff versus the untuned base. Second, we propose a calibrated refusal mechanism: training on 22% unanswerable examples yield a 12% "I don't know" rate, substantially improving over the base model's unsafe 4.3% rate while avoiding GPT-4.1's over-refusal (20.2%). Third, we present an end-to-end methodology spanning from data curation to quantized serving. The system is deployed at 40+ financial institutions, achieving a 7.1 percentage point improvement in query resolution (p < 0.001). Additionally, the model delivers 3-5x faster responses at 20-50x lower cost compared to GPT-4.1.

Related papers

• SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking [3.337528987886516]
  SalesLoop is a performance-aware reward that encodes conversion outcomes weighted by ranking position and conversion velocity.<n>SalesLoop improves NDCG@K by +7.9% and P@K by +15.8% over the strongest static baseline.
  arXiv  Detail & Related papers  (2026-07-22T18:26:57Z)
• Domain-Adapted Small Language Models with Hybrid Post-Processing: Achieving Cost-Efficient, Low-Latency Multi-Label Structured Prediction via LoRA Fine-Tuning on Scarce Data [6.3745740668603075]
  Large language models (LLMs) for domain-specific structured evaluation tasks incurs prohibitive latency, cost, and data-privacy overhead.<n>We present a hybrid framework that fine-tunes a small language model on only 219 curated examples and couples it with a deterministic rule-based postprocessing layer.<n>Our system achieves 100% structural validity, 83.0% human-validated overall accuracy, and 100% accuracy on the most critical classification field in blind evaluation on 53 unseen production transcripts.
  arXiv  Detail & Related papers  (2026-06-04T07:09:37Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• GRASP: Gated Regression-Aware Skill Proposer for Self-Improving LLM Agents [17.26555663132631]
  GRASP (Gated Regression-Aware Skill Proposer) treats agent improvement as a sequence of edits to a bounded skill library.<n>We evaluate GRASP across five base models on clinical benchmarks.
  arXiv  Detail & Related papers  (2026-05-28T09:30:09Z)
• FinGround: Detecting and Grounding Financial Hallucinations via Atomic Claim Verification [13.891522069967507]
  FinGround is a verify-then-ground pipeline for financial document QA.<n> Stage 1 performs finance-aware hybrid retrieval over text and tables.<n> Stage 2 decomposes answers into atomic claims classified by a six-type financial taxonomy.<n> Stage 3 rewrites unsupported claims with paragraph- and table-cell-level citations.
  arXiv  Detail & Related papers  (2026-04-26T07:52:59Z)
• Hybrid Deep Learning Approach for Coupled Demand Forecasting and Supply Chain Optimization [0.21990652930491858]
  This paper proposes a Hybrid AI Framework for Demand-Supply Forecasting and Optimization (HAF-DS)<n>It integrates a Long Short-Term Memory (LSTM)-based demand forecasting module with a mixed integer linear programming (MILP) optimization layer.<n>Experiments on textile sales and supply chain datasets show significant performance gains over statistical and deep learning baselines.
  arXiv  Detail & Related papers  (2026-04-23T11:38:59Z)
• PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis [0.0]
  Large language models (LLMs) show strong performance in natural language understanding and generation, but their evaluation and adaptation to domain-specific constraints in telecommunications customer support remain limited.<n>We present a systematic study of parameter-efficient fine-tuning (PEFT) using Low-Rank Adaptation (LoRA) applied to Qwen2.5-3B to build a domain-specific conversational assistant.<n>We introduce a synthetic data generation approach based on a glossary of 52 industry-specific terms, producing approximately 30,000 training examples across 1,560 distinct problem scenarios via a generative pipeline powered by Gemini Gemini.
  arXiv  Detail & Related papers  (2026-04-17T09:56:18Z)
• An Empirical Investigation of Practical LLM-as-a-Judge Improvement Techniques on RewardBench 2 [0.0]
  We present techniques that improve GPT-5.4 judge accuracy on RewardBench 2 without any finetuning.<n>Two techniques account for nearly all available gains: task-specific criteria injection and ensemble scoring.<n>Three further techniques did not reliably improve on criteria + ensembling at comparable cost.
  arXiv  Detail & Related papers  (2026-04-15T10:52:33Z)
• Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration [0.0]
  Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging.<n>We present the first application of MT-GRPO and GTPO for training a tool-calling agent on realistic customer service tasks.
  arXiv  Detail & Related papers  (2026-04-03T08:36:03Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• Enhancing Financial Report Question-Answering: A Retrieval-Augmented Generation System with Reranking Analysis [12.26184264751935]
  Retrieval-Augmented Generation system designed to answer questions about S&P 500 financial reports.<n>System employs hybrid search combining full-text and semantic retrieval, followed by an optional reranking stage.<n>Reranking significantly improves answer quality, achieving 49.0 percent correctness for scores of 8 or above.
  arXiv  Detail & Related papers  (2026-02-18T20:54:23Z)
• PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering [71.15346406323827]
  We introduce PRIME, a benchmark for evaluating verifiers on Process-Outcome Alignment verification.<n>We find that current verifiers frequently fail to detect derivation flaws.<n>We propose a process-aware RLVR training paradigm utilizing verifiers selected via PRIME.
  arXiv  Detail & Related papers  (2026-02-12T04:45:01Z)
• Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulations [49.671779378073886]
  We study question answering in the domain of radio regulations.<n>We propose a telecom-specific Retrieval-Augmented Generation (RAG) pipeline.<n>Our approach consistently improves generation accuracy across all tested models.
  arXiv  Detail & Related papers  (2025-09-11T17:43:42Z)
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [54.05511925104712]
  We propose a simple, effective, and data-efficient method called Step-DPO. Step-DPO treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters.
  arXiv  Detail & Related papers  (2024-06-26T17:43:06Z)
• Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation [56.13803674092712]
  We propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR) CaR employs a two-step process: first, it ranks instruction pairs using a high-accuracy (84.25%) scoring model aligned with expert preferences; second, it preserves dataset diversity through clustering. In our experiment, CaR efficiently selected a mere 1.96% of Alpaca's IT data, yet the resulting AlpaCaR model surpassed Alpaca's performance by an average of 32.1% in GPT-4 evaluations.
  arXiv  Detail & Related papers  (2024-02-28T09:27:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.