Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

• URL: http://arxiv.org/abs/2608.08126v1
• Date: Sat, 08 Aug 2026 13:22:14 GMT
• Title: Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk
• Abstract summary: Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters, in tension with supervisory expectations that adverse decisions be explainable. A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale. We build such a system end to end and test whether the second half of the promise holds. The predictive component is a multi-scale stacking ensemble fusing four differently regularised gradient-boosting learners with a residual network through a neural meta-learner trained on out-of-fold predictions. On a public 32,581-application credit dataset it reaches test ROC-AUC 0.9539 (95% CI [0.9462, 0.9616]) and PR-AUC 0.9137, beating the best single model by Delta-AUC = 0.0143 (p = 0.016 under a conservative independence assumption). Our central finding is asymmetric. The ranking gain is real but operationally small: at the F1-optimal threshold the ensemble avoids only six additional missed defaults out of 1,422 against a tuned random forest, cutting cost-weighted loss by under 2%. The narrative layer fails in a way prompt engineering alone does not fix. In an audited case the model named three factors as risk-increasing that the supplied attributions scored as risk-reducing, omitted the dominant driver, and introduced a feature never given to it. We trace this to properties we measure rather than assume: SHAP and LIME agree on which features matter (overlap@10 = 0.80) but not on their order (tau = 0.43, p = 0.18), and the attribution sign for the model's most sensitive input is near a coin flip across applicants (modal-sign share 0.53). Calibration (ECS = 0.117) and perturbation stability (DPD = 0.078) both fall short of our own thresholds. Constrained prompting is necessary but not sufficient: grounding must be verified after generation, not assumed.

Related papers

• Invocation-Level Reliability of Tool-Using Agents [0.7874708385247353]
  We measure a correct-invocation rate that separates the two, under both a clean teacher-forced context and the model's own free-running context.<n>By depth 6, roughly 70% of a model's own clean-context capability is lost to its own earlier mistakes.
  arXiv  Detail & Related papers  (2026-08-23T09:04:59Z)
• Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs [27.21644231867863]
  Caliber is a defense against model extraction that formulates noise selection as a calibration problem.<n>To defend against an attacker that uses returned scores for knowledge distillation, Caliber adds independent and identically distributed Gaussian noise to the internal logits.<n>Caliber normalizes noise variance by the squared median top-two logit margin and fits the resulting noise-utility relationship with a logistic curve.
  arXiv  Detail & Related papers  (2026-08-02T05:58:00Z)
• One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence [41.99844472131922]
  A single human must audit $N$ LLM agents under a budget of $B ll N$ audits per round.<n>We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $*$ past which confidence-ranked auditing is emphworse than random.
  arXiv  Detail & Related papers  (2026-07-30T14:52:36Z)
• The C-index illusion: discrimination without calibration in published survival models [0.0]
  We reproduce three published survival-ML models across three structurally distinct domains.<n>We test five pre-registered hypotheses under a Holm-corrected family-wise error rate.
  arXiv  Detail & Related papers  (2026-07-21T19:13:54Z)
• Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling [0.0]
  Synthetic-population tools run every individual as an independent large language model (LLM) agent.<n>We show that this paradigm has a basic failure mode, and we set a distribution-first corrective against it.
  arXiv  Detail & Related papers  (2026-07-17T15:45:57Z)
• YUKTI: From Natural-Language Situations to Robust, Verifiable Decisions An Uncertainty-Typed Proposition IR, Assumption-Robust Pareto Frontiers, and a Regret Certificate [0.0]
  We show that YUKTI routes each stage to an exact, nonlinear, or evolutionary solver.<n>We prove a bound making rho an exact factor of decision regret, add auditable traceability, and synthesize a benchmark-faithful data foundation.<n>We validate three ways: under controlled misspecification the robust compromise cuts mean and tail regret by over 90% versus a naive point plan; and on a regulated commercial decision we optimize inside a lawful action space and price the downside in euros.
  arXiv  Detail & Related papers  (2026-06-22T09:29:13Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Knowing but Not Correcting: Routine Task Requests Suppress Factual Correction in LLMs [26.062372963777452]
  LLMs reliably correct false claims when presented in isolation, yet when the same claims are embedded in task-oriented requests, they often comply rather than correct.<n>We term this failure mode emphcorrection suppression and construct a benchmark of 300 false premises to systematically evaluate it across eight models.<n> Suppression rates range from 19% to 90%, with four models exceeding 80%, establishing correction suppression as a prevalent and severe phenomenon.
  arXiv  Detail & Related papers  (2026-05-07T10:04:39Z)
• The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime [0.0]
  Most cited calibration result in deep learning -- post-temperature-scaling ECE of 0.012 on CIFAR-100 -- is below the statistical noise floor.<n>We prove this is not a failure of the experiment but a law: the minimax rate for estimating calibration error with model error rate epsilon is Theta((Lepsilon/m)2/3), and no estimator can beat it.
  arXiv  Detail & Related papers  (2026-04-14T16:48:24Z)
• When LLMs get significantly worse: A statistical approach to detect model degradations [33.63321816712603]
  Minimizing the inference cost and latency of foundation models has become a crucial area of research.<n>We propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations.<n>We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.
  arXiv  Detail & Related papers  (2026-02-09T10:45:13Z)
• Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning [32.32593439144886]
  Behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification.<n>Our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation.
  arXiv  Detail & Related papers  (2025-12-22T22:51:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.