The Illusion of Improvement: Reject Inference Strategies in Credit Scoring

• URL: http://arxiv.org/abs/2606.18479v1
• Date: Tue, 16 Jun 2026 20:43:43 GMT
• Title: The Illusion of Improvement: Reject Inference Strategies in Credit Scoring
• Abstract summary: Reject inference methods are widely used to mitigate survival bias in credit scoring.<n>In a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement.<n>We propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions.
• Score: 10.82789277277678
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood. We systematically evaluate several such methods and uncover a structural failure mode: in a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement that leads practitioners to believe the system is getting better when, in fact, its rejection quality -- the ability to correctly screen out defaulters -- is deteriorating. We then propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions: the lender deliberately approves a fraction of rejected applicants and observes their true outcomes. We show that accuracy and rejection quality give opposite recommendations on whether to explore: accuracy favors no exploration, while rejection quality improves with it, confirming that standard evaluation metrics are misleading under selection bias. Even minimal exploration rates (2--5\%) prove sufficient in our experiments to diagnose the severity of the feedback loop at near-zero cost. Our findings are consistent across two machine learning methods and three real-world datasets, and suggest that standard evaluation protocols are inadequate for assessing models trained under survival bias.

Related papers

• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure [23.021647176206972]
  Single-axis mitigations of reward-model biases can rotate optimization pressure onto correlated proxies rather than eliminate it.<n>The failure is enabled by a measurement-versus-optimization gap between audit and policy-induced distributions.<n>We formalize mitigation outcomes into a regime taxonomy and prove that successful mitigation, bias substitution, and overcorrection produce identical observables.
  arXiv  Detail & Related papers  (2026-05-27T05:40:22Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown [4.503897051444244]
  Deep neural networks, despite their high accuracy, often exhibit poor confidence calibration, limiting their reliability in high-stakes applications.<n>Current ad-hoc confidence calibration methods attempt to fix this during training but face a fundamental trade-off.<n>This paper addresses and mitigates this stability-performance trade-off.<n>We propose Socrates Loss, a novel, unified loss function that explicitly leverages uncertainty by incorporating an auxiliary unknown class.
  arXiv  Detail & Related papers  (2026-04-14T03:43:15Z)
• Observationally Informed Adaptive Causal Experimental Design [55.998153710215654]
  We propose Active Residual Learning, a new paradigm that leverages the observational model as a foundational prior.<n>This approach shifts the experimental focus from learning target causal quantities from scratch to efficiently estimating the residuals required to correct observational bias.<n> Experiments on synthetic and semi-synthetic benchmarks demonstrate that R-Design significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-03-04T06:52:37Z)
• Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance [39.88432082145946]
  We propose SCOPE, a novel framework that utilizes Process Reward Models to pinpoint the first erroneous step in suboptimal rollouts.<n>By applying precise refinement on partially correct rollout, our method effectively salvages partially correct trajectories and increases diversity score by 13.5%.<n>Our approach establishes new state-of-the-art results, achieving an average accuracy of 46.6% on math reasoning and exhibiting robust generalization with 53.4% accuracy on out-of-distribution reasoning tasks.
  arXiv  Detail & Related papers  (2026-02-27T15:49:23Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Aurora: Are Android Malware Classifiers Reliable and Stable under Distribution Shift? [51.12297424766236]
  AURORA is a framework to evaluate malware classifiers based on their confidence quality and operational resilience.<n>AURORA is complemented by a set of metrics designed to go beyond point-in-time performance.<n>The fragility in SOTA frameworks across datasets of varying drift suggests the need for a return to the whiteboard.
  arXiv  Detail & Related papers  (2025-05-28T20:22:43Z)
• TrustLoRA: Low-Rank Adaptation for Failure Detection under Out-of-distribution Data [62.22804234013273]
  We propose a simple failure detection framework to unify and facilitate classification with rejection under both covariate and semantic shifts.<n>Our key insight is that by separating and consolidating failure-specific reliability knowledge with low-rank adapters, we can enhance the failure detection ability effectively and flexibly.
  arXiv  Detail & Related papers  (2025-04-20T09:20:55Z)
• Fighting Sampling Bias: A Framework for Training and Evaluating Credit Scoring Models [2.918530881730374]
  This paper addresses the adverse effect of sampling bias on model training and evaluation. We propose bias-aware self-learning and a reject inference framework for scorecard evaluation. Our results suggest a profit improvement of about eight percent, when using Bayesian evaluation to decide on acceptance rates.
  arXiv  Detail & Related papers  (2024-07-17T20:59:54Z)
• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [53.79160907725975]
  We find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance.
  arXiv  Detail & Related papers  (2024-03-05T11:44:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.