Fin-Bias: Comprehensive Evaluation for LLM Decision-Making under human bias in Finance Domain

• URL: http://arxiv.org/abs/2605.09106v1
• Date: Sat, 09 May 2026 18:28:40 GMT
• Title: Fin-Bias: Comprehensive Evaluation for LLM Decision-Making under human bias in Finance Domain
• Abstract summary: Fin-Bias is a benchmark for evaluating large language models (LLMs) in financial contexts.<n> Fin-Bias includes 8868 long firm-specific analyst reports, including firm aspects summarized and analyzed by sophisticated analysts with investment ratings (Bullish/Neutral/Bearish) spanning from various industries.<n>We present large language models with firm analyst reports with/without analyst investment ratings and even with 'fake' rating, to get investment ratings generated by LLMs.
• Score: 9.104486997354178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed in financial contexts, raising critical concerns about reliability, alignment, and susceptibility to adversarial manipulation. While prior finance-related benchmarks assess LLMs' capabilities in stock trading, they are often restricted to small sample and fail to demonstrate LLM susceptibility to context with potential human bias. We introduce Fin-Bias (financial herding under long and uncertain financial context), a benchmark for evaluating LLM investment decision-making when faced with uncertainty and possible human-biased opinions. Fin-Bias includes 8868 long firm-specific analyst reports, including firm aspects summarized and analyzed by sophisticated analysts with investment ratings (Bullish/Neutral/Bearish) spanning from various industries. We present large language models with firm analyst reports with/without analyst investment ratings and even with 'fake' rating, to get investment ratings generated by LLMs. Our results reveal that LLMs tend to herd the explicit bias in context. We also develop a method to detect potential human opinions, which can encourage LLMs to think independently, some models even exceed human performance in predicting future stock return.

Related papers

• Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection [64.75447949495307]
  Large language models (LLMs) have been widely applied across various domains of finance.<n> behavioral biases can lead to instability and uncertainty in decision-making.<n>mfmdscen is a benchmark for evaluating behavioral biases in mfmd across diverse economic scenarios.
  arXiv  Detail & Related papers  (2026-01-08T22:00:32Z)
• FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain [54.06289302468199]
  FinTrust is a benchmark specifically designed for evaluating the trustworthiness of LLMs in finance applications.<n> proprietary models like o4-mini outperforms in most tasks such as safety.<n>Open-source models like DeepSeek-V3 have advantage in specific areas like industry-level fairness.
  arXiv  Detail & Related papers  (2025-10-17T01:45:49Z)
• StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets? [44.10622904101254]
  Large language models (LLMs) have recently demonstrated strong capabilities as autonomous agents.<n>We introduce StockBench, a benchmark designed to evaluate LLM agents in realistic, multi-month stock trading environments.<n>Our evaluation shows that while most LLM agents struggle to outperform the simple buy-and-hold baseline, several models demonstrate the potential to deliver higher returns and manage risk more effectively.
  arXiv  Detail & Related papers  (2025-10-02T16:54:57Z)
• Your AI, Not Your View: The Bias of LLMs in Investment Analysis [62.388554963415906]
  In finance, Large Language Models (LLMs) face frequent knowledge conflicts arising from discrepancies between their pre-trained parametric knowledge and real-time market data.<n>These conflicts are especially problematic in real-world investment services, where a model's inherent biases can misalign with institutional objectives.<n>We propose an experimental framework to investigate emergent behaviors in such conflict scenarios, offering a quantitative analysis of bias in investment analysis.
  arXiv  Detail & Related papers  (2025-07-28T16:09:38Z)
• Towards Competent AI for Fundamental Analysis in Finance: A Benchmark Dataset and Evaluation [5.892346584607669]
  We propose FinAR-Bench, a benchmark dataset focusing on financial statement analysis.<n>We break this task into three measurable steps: extracting key information, calculating financial indicators, and applying logical reasoning.<n>Our findings offer a clear understanding of LLMs current strengths and limitations in fundamental analysis.
  arXiv  Detail & Related papers  (2025-05-22T07:06:20Z)
• Financial Statement Analysis with Large Language Models [0.0]
  We provide standardized and anonymous financial statements to GPT4 and instruct the model to analyze them.<n>The model outperforms financial analysts in its ability to predict earnings changes directionally.<n>Our trading strategies based on GPT's predictions yield a higher Sharpe ratio and alphas than strategies based on other models.
  arXiv  Detail & Related papers  (2024-07-25T08:36:58Z)
• DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators.<n>The question of how reliable these evaluators are has emerged as a crucial research question.<n>We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• Benchmarking LLMs via Uncertainty Quantification [91.72588235407379]
  The proliferation of open-source Large Language Models (LLMs) has highlighted the urgent need for comprehensive evaluation methods. We introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs.
  arXiv  Detail & Related papers  (2024-01-23T14:29:17Z)
• Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models [11.154814189699735]
  Large Language Models (LLMs) pre-trained on extensive corpora have demonstrated superior performance across various NLP tasks. We introduce a retrieval-augmented LLMs framework for financial sentiment analysis. Our approach achieves 15% to 48% performance gain in accuracy and F1 score.
  arXiv  Detail & Related papers  (2023-10-06T05:40:23Z)
• Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models [53.620827459684094]
  Large Language Models (LLMs) have great potential for credit scoring tasks, with strong generalization ability across multiple tasks. We propose the first open-source comprehensive framework for exploring LLMs for credit scoring. We then propose the first Credit and Risk Assessment Large Language Model (CALM) by instruction tuning, tailored to the nuanced demands of various financial risk assessment tasks.
  arXiv  Detail & Related papers  (2023-10-01T03:50:34Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.