GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation

• URL: http://arxiv.org/abs/2608.11787v1
• Date: Wed, 12 Aug 2026 08:28:15 GMT
• Title: GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation
• Abstract summary: We formulate financial advice generation as a reinforcement learning problem.<n>We fine-tune an open-weight language model using Group Relative Policy Optimization.<n>Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple dimensions of advice quality.
• Score: 2.2936251006503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating actionable financial advice from business records demands that models integrate numerical reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline ($0.0228$ vs.\ $0.0104$), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP.

Related papers

• LLM-as-a-Judge for Evaluating System Responses in Conversational Music Recommendation [20.276417246845593]
  We present the first user study to empirically assess the reliability of LLM-as-a-judge for evaluating CRS responses.<n>We collect $n=400$ ratings from 20 domain-expert annotators, who evaluate each response across two dimensions: Personalization Quality and Explanation Quality.<n>We find that LLM-based judges exhibit moderate positive alignment with human assessments and outperform all reference-based baselines.
  arXiv  Detail & Related papers  (2026-07-28T12:26:55Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable [1.8689252029357564]
  This paper introduces ValueBlindBench, a preregistered agreement-gated stress-test protocol.<n>ValueBlindBench decides when LLM-judged investment-rationale claims are publishable, qualified, or invalid.
  arXiv  Detail & Related papers  (2026-04-28T05:04:20Z)
• Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training [75.98427023288052]
  Reasoning LLMs-as-Judges provide a promising path for extending the success of reasoning models to non-verifiable domains.<n>This study investigates the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment.<n>We find that reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs.
  arXiv  Detail & Related papers  (2026-03-12T17:57:06Z)
• Understanding LLM Evaluator Behavior: A Structured Multi-Evaluator Framework for Merchant Risk Assessment [26.786161923794115]
  Large Language Models (LLMs) are increasingly used as evaluators of reasoning quality, yet their reliability and bias in payments-risk settings remain poorly understood.<n>We introduce a structured multi-evaluator framework for assessing LLM reasoning in Merchant Category Code (MCC)-based merchant risk assessment.
  arXiv  Detail & Related papers  (2026-02-04T22:55:16Z)
• Ask a Strong LLM Judge when Your Reward Model is Uncertain [46.41334493044746]
  We propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge.<n>Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification.<n> Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost.
  arXiv  Detail & Related papers  (2025-10-23T09:09:13Z)
• HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment [52.374772443536045]
  HALF (Harm-Aware LLM Fairness) is a framework that assesses model bias in realistic applications and weighs the outcomes by harm severity.<n>We show that HALF exposes a clear gap between previous benchmarking success and deployment readiness.
  arXiv  Detail & Related papers  (2025-10-14T07:13:26Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization [69.23273504123941]
  We train judges to be robust to positional biases that arise in more complex evaluation settings.<n>We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work.<n>We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%.
  arXiv  Detail & Related papers  (2025-05-19T16:50:35Z)
• JudgeLRM: Large Reasoning Models as a Judge [80.07261839142548]
  We introduce JudgeLRM, a family of judgment-oriented Large Language Models (LLMs)<n>We find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios.<n>We show that JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models.
  arXiv  Detail & Related papers  (2025-03-31T02:18:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.