A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction

• URL: http://arxiv.org/abs/2606.17649v1
• Date: Tue, 16 Jun 2026 08:07:42 GMT
• Title: A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction
• Abstract summary: High cost of fine-tuning LLMs poses a significant economic barrier; pre-hoc performance prediction offers a critical solution to substantially reduce this expense.<n>We formulate it as an estimation problem under information constraints, decomposing prediction risk into two components: an intrinsic limit (static data-model compatibility) and a reducible optimization variance.<n>We introduce a predictability phase diagram that organizes tasks into three distinct regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant.
• Score: 7.112791813526235
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The high cost of fine-tuning LLMs poses a significant economic barrier; pre-hoc performance prediction offers a critical solution to substantially reduce this expense. However, the theoretical limits of pre-hoc performance prediction remain unexplored. We formulate it as a stochastic estimation problem under information constraints, decomposing prediction risk into two components: an intrinsic limit (static data-model compatibility) and a reducible optimization variance. We prove that optimization variance admits a necessary lower bound on its decay rate, implying fundamental constraints on how quickly uncertainty dissipates, regardless of the predictor used. Based on these dynamics, we derive a budget-optimal probing principle and introduce a predictability phase diagram that organizes tasks into three distinct regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant. Extensive experiments on synthetic and real-world benchmarks validate these theoretical regimes and demonstrate the efficiency of our probing strategy.

Related papers

• Smart predict-then-robustly-optimize [0.5161531917413708]
  We propose and study a robust variant of the smart predict-then-optimize approach.<n>Our framework consistently yields significant performance improvements over standard methods.
  arXiv  Detail & Related papers  (2026-07-23T19:37:17Z)
• Accelerating Conformal Prediction via Approximate Leave-One-Out [14.660852843318809]
  We accelerate conformal prediction by incorporating approximate leave-one-out (ALO) estimators, and establish coverage and efficiency.<n>Our proof draws on methods developed for analyzing the consistency of ALO cross-validation risk estimators in high-dimensional statistics.<n>We show that the ALO-based methods achieve coverage and efficiency comparable to the exact methods, while significantly reducing the runtime.
  arXiv  Detail & Related papers  (2026-06-30T16:21:48Z)
• Possibilistic Predictive Uncertainty for Deep Learning [75.00607359322863]
  We introduce Dirichlet-approximated possibilistic posterior predictions (DAPPr)<n>DAPPr is a principled framework leveraging possibility theory.<n>This projection-and-approximation strategy yields a simple training objective with closed-form solutions.
  arXiv  Detail & Related papers  (2026-05-01T12:14:01Z)
• Observationally Informed Adaptive Causal Experimental Design [55.998153710215654]
  We propose Active Residual Learning, a new paradigm that leverages the observational model as a foundational prior.<n>This approach shifts the experimental focus from learning target causal quantities from scratch to efficiently estimating the residuals required to correct observational bias.<n> Experiments on synthetic and semi-synthetic benchmarks demonstrate that R-Design significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-03-04T06:52:37Z)
• Unified Inference Framework for Single and Multi-Player Performative Prediction: Method and Asymptotic Optimality [15.289993502701305]
  This paper introduces a unified statistical inference framework that bridges single-agent and multi-agent performativity.<n>It provides a principled toolkit for reliable estimation and decision-making in dynamic, performative environments.
  arXiv  Detail & Related papers  (2026-02-03T03:17:54Z)
• Supervised Optimism Correction: Be Confident When LLMs Are Sure [91.7459076316849]
  We establish a novel theoretical connection between supervised fine-tuning and offline reinforcement learning.<n>We show that the widely used beam search method suffers from unacceptable over-optimism.<n>We propose Supervised Optimism Correction, which introduces a simple yet effective auxiliary loss for token-level $Q$-value estimations.
  arXiv  Detail & Related papers  (2025-04-10T07:50:03Z)
• "Over-optimizing" for Normality: Budget-constrained Uncertainty Quantification for Contextual Decision-making [1.2997358113120427]
  We study uncertainty quantification for contextual optimization using weighted sample average approximation (wSAA)<n>We establish central limit theorems for wSAA and construct algorithm-normality-based confidence intervals for optimal conditional expected costs.<n>We show that over-optimizing'' (running more iterations than the nominal rule) mitigates this misspecification and preserves normality, at the expense of a slight slowdown in the convergence rate of the budget-constrained estimator.
  arXiv  Detail & Related papers  (2025-03-17T02:31:56Z)
• Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning [53.25336975467293]
  We present the first theoretical error decomposition analysis of methods such as perplexity and self-consistency.<n>Our analysis reveals a fundamental trade-off: perplexity methods suffer from substantial model error due to the absence of a proper consistency function.<n>We propose Reasoning-Pruning Perplexity Consistency (RPC), which integrates perplexity with self-consistency, and Reasoning Pruning, which eliminates low-probability reasoning paths.
  arXiv  Detail & Related papers  (2025-02-01T18:09:49Z)
• Smart Predict-then-Optimize Method with Dependent Data: Risk Bounds and Calibration of Autoregression [7.369846475695131]
  We present an autoregressive SPO method directly targeting the optimization problem at the decision stage. We conduct experiments to demonstrate the effectiveness of the SPO+ surrogate compared to the absolute loss and the least squares loss.
  arXiv  Detail & Related papers  (2024-11-19T17:02:04Z)
• Fair Risk Minimization under Causal Path-Specific Effect Constraints [3.0232957374216953]
  This paper introduces a framework for estimating fair optimal predictions using machine learning. We derive closed-form solutions for constrained optimization based on mean squared error and cross-entropy risk criteria.
  arXiv  Detail & Related papers  (2024-08-03T02:05:43Z)
• C-Learner: Constrained Learning for Causal Inference [4.370964009390564]
  We propose a novel debiasing approach that achieves the best weighting of both worlds, producing stable plug-in estimates.<n>Our constrained learning framework solves for the best plug-in estimator under the constraint that the first-order error with respect to the plugged-in quantity is zero.
  arXiv  Detail & Related papers  (2024-05-15T16:38:28Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency [83.02999769628593]
  We offer a theoretical characterization of off-policy evaluation (OPE) in reinforcement learning. We show that the minimax approach enables us to achieve a fast rate of convergence for weights and quality functions. We present the first finite-sample result with first-order efficiency in non-tabular environments.
  arXiv  Detail & Related papers  (2021-02-05T03:20:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.