MortarBench: Evaluating Mortgage Loan Origination Agents

• URL: http://arxiv.org/abs/2606.19416v2
• Date: Mon, 22 Jun 2026 21:40:16 GMT
• Title: MortarBench: Evaluating Mortgage Loan Origination Agents
• Abstract summary: We present MortarBench, a loan origination agent benchmark.<n>We find that state-of-the-art large language models (LLMs) perform poorly, with closed-source models achieving at most 77.1% exact match accuracy.<n>Noting these weaknesses, we introduce CRIT, a confidence calibration framework. Our method increases accuracy to 80.5% while improving risk management steering and reducing bias.
• Score: 14.831837275522327
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Loan origination is the process by which a lender creates a new loan, from application and underwriting through approval and funding. This process serves a critical role in evaluating the eligibility and level of risk posed by an applicant. Recently, firms have begun using mortgage loan agents to augment human loan officers, despite a lack of any public benchmark. To fill this gap, we present MortarBench, a loan origination agent benchmark. MortarBench uses a financial data synthesis and mutation pipeline to generate examples with broad edge case coverage that match real-world distributions and questions. We find that state-of-the-art large language models (LLMs) perform poorly, with closed-source models achieving at most 77.1\% exact match accuracy. We also discover systematic biases in LLM perception of foreignness related to non-English names. Noting these weaknesses, we introduce CRIT, a confidence calibration framework. Our method increases accuracy to 80.5\% while improving risk management steering and reducing bias.

Related papers

• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models [52.61023005303122]
  We show that low-confidence remasking improves a proxy for quality while constraining the entropy of the induced sequence distribution.<n>We develop a simple Independent--Hastings sampler that approximately targets this distribution during decoding.
  arXiv  Detail & Related papers  (2026-04-01T02:01:30Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• Standard Benchmarks Fail - Auditing LLM Agents in Finance Must Prioritize Risk [31.43947127076459]
  Standard benchmarks fixate on how well large language model (LLM) agents perform in finance, yet say little about whether they are safe to deploy.<n>We argue that accuracy metrics and return-based scores provide an illusion of reliability, overlooking vulnerabilities such as hallucinated facts, stale data, and adversarial prompt manipulation.
  arXiv  Detail & Related papers  (2025-02-21T12:56:15Z)
• Credit Risk Meets Large Language Models: Building a Risk Indicator from Loan Descriptions in P2P Lending [1.1970409518725493]
  Peer-to-peer (P2P) lending connects borrowers and lenders through online platforms but suffers from significant information asymmetry.<n>This paper addresses this challenge by leveraging BERT, a Large Language Model (LLM) known for its ability to capture contextual nuances in text.<n>We fine-tune BERT to distinguish between defaulted and non-defaulted loans using the loan descriptions provided by the borrowers.
  arXiv  Detail & Related papers  (2024-01-29T10:11:05Z)
• How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation [46.42384207122049]
  We design SimulateBench to evaluate the believability of large language models (LLMs) when simulating human behaviors. Based on SimulateBench, we evaluate the performances of 10 widely used LLMs when simulating characters.
  arXiv  Detail & Related papers  (2023-12-28T16:51:11Z)
• Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models [53.620827459684094]
  Large Language Models (LLMs) have great potential for credit scoring tasks, with strong generalization ability across multiple tasks. We propose the first open-source comprehensive framework for exploring LLMs for credit scoring. We then propose the first Credit and Risk Assessment Large Language Model (CALM) by instruction tuning, tailored to the nuanced demands of various financial risk assessment tasks.
  arXiv  Detail & Related papers  (2023-10-01T03:50:34Z)
• Inclusive FinTech Lending via Contrastive Learning and Domain Adaptation [9.75150920742607]
  FinTech lending has played a significant role in facilitating financial inclusion. There are concerns about the potentially biased algorithmic decision-making during loan screening. We propose a new Transformer-based sequential loan screening model with self-supervised contrastive learning and domain adaptation.
  arXiv  Detail & Related papers  (2023-05-10T01:11:35Z)
• How Costly is Noise? Data and Disparities in Consumer Credit [0.0]
  We show that credit scores are noisier indicators of default risk for historically under-served groups. We find that equalizing the precision of credit scores can reduce disparities in approval rates and in credit misallocation for disadvantaged groups by approximately half.
  arXiv  Detail & Related papers  (2021-05-17T00:42:26Z)
• Explanations of Machine Learning predictions: a mandatory step for its application to Operational Processes [61.20223338508952]
  Credit Risk Modelling plays a paramount role. Recent machine and deep learning techniques have been applied to the task. We suggest to use LIME technique to tackle the explainability problem in this field.
  arXiv  Detail & Related papers  (2020-12-30T10:27:59Z)
• Predicting Bank Loan Default with Extreme Gradient Boosting [0.0]
  We use an Extreme Gradient Boosting algorithm called XGBoost for loan default prediction. The prediction is based on a loan data from a leading bank taking into consideration data sets from both the loan application and the demographic of the applicant.
  arXiv  Detail & Related papers  (2020-01-18T18:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.