A Methodology for Assessing the Risk of Metric Failure in LLMs Within the Financial Domain

• URL: http://arxiv.org/abs/2510.13524v2
• Date: Thu, 16 Oct 2025 12:21:22 GMT
• Title: A Methodology for Assessing the Risk of Metric Failure in LLMs Within the Financial Domain
• Authors: William Flanagan, Mukunda Das, Rajitha Ramanayake, Swanuja Maslekar, Meghana Mangipudi, Joong Ho Choi, Shruti Nair, Shambhavi Bhusan, Sanjana Dulam, Mouni Pendharkar, Nidhi Singh, Vashisth Doshi, Sachi Shah Paresh,
• Abstract summary: Historical machine learning metrics can oftentimes fail to generalize to GenAI workloads.<n>This paper explains these challenges and provides a Risk Assessment Framework to allow for better application of SME and machine learning Metrics.
• Score: 0.25409967292854213
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As Generative Artificial Intelligence is adopted across the financial services industry, a significant barrier to adoption and usage is measuring model performance. Historical machine learning metrics can oftentimes fail to generalize to GenAI workloads and are often supplemented using Subject Matter Expert (SME) Evaluation. Even in this combination, many projects fail to account for various unique risks present in choosing specific metrics. Additionally, many widespread benchmarks created by foundational research labs and educational institutions fail to generalize to industrial use. This paper explains these challenges and provides a Risk Assessment Framework to allow for better application of SME and machine learning Metrics

Related papers

• Quantifying Uncertainty in Machine Learning-Based Pervasive Systems: Application to Human Activity Recognition [0.2740273306918099]
  We propose to quantify uncertainty in machine learning-based systems.<n>We propose to adapt and jointly utilize a set of selected techniques to evaluate the relevance of model predictions at runtime.<n>The results presented demonstrate the relevance of the approach, and we discuss in detail the assistance provided to domain experts.
  arXiv  Detail & Related papers  (2025-12-10T15:56:05Z)
• A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives [65.3369988566853]
  Recent studies have demonstrated that adversaries can replicate a target model's functionality.<n>Model Extraction Attacks pose threats to intellectual property, privacy, and system security.<n>We propose a novel taxonomy that classifies MEAs according to attack mechanisms, defense approaches, and computing environments.
  arXiv  Detail & Related papers  (2025-08-20T19:49:59Z)
• ELMES: An Automated Framework for Evaluating Large Language Models in Educational Scenarios [23.549720214649476]
  Large Language Models (LLMs) present transformative opportunities for education, generating numerous novel application scenarios.<n>Current benchmarks predominantly measure general intelligence rather than pedagogical capabilities.<n>We introduce ELMES, an open-source automated evaluation framework specifically designed for assessing LLMs in educational settings.
  arXiv  Detail & Related papers  (2025-07-27T15:20:19Z)
• Rethinking Machine Unlearning in Image Generation Models [59.697750585491264]
  CatIGMU is a novel hierarchical task categorization framework.<n>EvalIGMU is a comprehensive evaluation framework.<n>We construct DataIGM, a high-quality unlearning dataset.
  arXiv  Detail & Related papers  (2025-06-03T11:25:14Z)
• Alpha Excel Benchmark [0.0]
  This study presents a novel benchmark for evaluating Large Language Models (LLMs) using challenges derived from the Financial Modeling World Cup (FMWC) Excel competitions.<n>Our findings demonstrate significant variations in performance across different challenge categories with models showing specific strengths in recognition tasks but struggling with complex numerical reasoning.
  arXiv  Detail & Related papers  (2025-05-07T03:56:26Z)
• GEMS: Generative Expert Metric System through Iterative Prompt Priming [18.0413505095456]
  Non-experts can find it unintuitive to create effective measures or transform theories into context-specific metrics. This technical report addresses this challenge by examining software communities within large software corporations. We propose a prompt-engineering framework inspired by neural activities, demonstrating that generative models can extract and summarize theories.
  arXiv  Detail & Related papers  (2024-10-01T17:14:54Z)
• A machine learning workflow to address credit default prediction [0.44943951389724796]
  Credit default prediction (CDP) plays a crucial role in assessing the creditworthiness of individuals and businesses. We propose a workflow-based approach to improve CDP, which refers to the task of assessing the probability that a borrower will default on his or her credit obligations.
  arXiv  Detail & Related papers  (2024-03-06T15:30:41Z)
• Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models [53.620827459684094]
  Large Language Models (LLMs) have great potential for credit scoring tasks, with strong generalization ability across multiple tasks. We propose the first open-source comprehensive framework for exploring LLMs for credit scoring. We then propose the first Credit and Risk Assessment Large Language Model (CALM) by instruction tuning, tailored to the nuanced demands of various financial risk assessment tasks.
  arXiv  Detail & Related papers  (2023-10-01T03:50:34Z)
• Benchmarking Automated Machine Learning Methods for Price Forecasting Applications [58.720142291102135]
  We show the possibility of substituting manually created ML pipelines with automated machine learning (AutoML) solutions. Based on the CRISP-DM process, we split the manual ML pipeline into a machine learning and non-machine learning part. We show in a case study for the industrial use case of price forecasting, that domain knowledge combined with AutoML can weaken the dependence on ML experts.
  arXiv  Detail & Related papers  (2023-04-28T10:27:38Z)
• Truthful Meta-Explanations for Local Interpretability of Machine Learning Models [10.342433824178825]
  We present a local meta-explanation technique which builds on top of the truthfulness metric, which is a faithfulness-based metric. We demonstrate the effectiveness of both the technique and the metric by concretely defining all the concepts and through experimentation.
  arXiv  Detail & Related papers  (2022-12-07T08:32:04Z)
• Panoramic Learning with A Standardized Machine Learning Formalism [116.34627789412102]
  This paper presents a standardized equation of the learning objective, that offers a unifying understanding of diverse ML algorithms. It also provides guidance for mechanic design of new ML solutions, and serves as a promising vehicle towards panoramic learning with all experiences.
  arXiv  Detail & Related papers  (2021-08-17T17:44:38Z)
• Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making [67.72855777115772]
  Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts -- who often have no expertise in ML or data science -- are asked to use ML predictions to make high-stakes decisions. We investigate the ML usability challenges present in the domain of child welfare screening through a series of collaborations with child welfare screeners.
  arXiv  Detail & Related papers  (2021-03-02T22:50:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.