When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer

• URL: http://arxiv.org/abs/2608.01130v1
• Date: Sun, 02 Aug 2026 10:12:42 GMT
• Title: When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer
• Abstract summary: We ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates.<n>Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A broad range of models face the mismatch where they are updated through trajectory losses but are evaluated by downstream task reward. Here, a trajectory is a training instance that induces a surrogate loss whose reduction might not track the model's decision utility update. Theoretically, we ask when one step of trajectory training reduces both population surrogate loss and decision risk, and how transfer accumulates along repeated updates. To formalize this, we first fix a checkpoint and a restricted update space, and define the reductions in population surrogate risk and decision risk induced by a trajectory as its learnability and decision utility, respectively. On this basis, our theory yields four main results. First, a one-step transfer bound separates their discrepancy into first-order gradient misalignment after nonnegative calibration and second-order curvature; and a pathwise extension accumulates the same terms over repeated updates. Second, when the accessible surrogate gradient is nonzero, universal first-order transfer over every accessible direction holds exactly when the accessible surrogate and decision gradients are positively collinear. Third, the calibration gap bounds the decision regret of learnability-based trajectory selection, while a candidate-difference refinement tightens this guarantee by retaining only directions that affect pairwise rankings. Finally, we establish an approximation--calibration trade-off across nested update spaces. Controlled gridworld and LLM post-training experiments yield results consistent with our predictions.

Related papers

• Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers [40.43356191604634]
  Looped Transformers offer a parameter-efficient route to test-time scaling by reusing shared layers for iterative latent reasoning.<n>Additional iterations can reduce support for a reference answer, leaving unclear whether an update's direction is locally unhelpful or its full displacement moves too far.<n>We study this distinction by analysing reference utility, which measures this support, along the model's own update direction.
  arXiv  Detail & Related papers  (2026-09-15T05:42:02Z)
• One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control [62.82372407840088]
  We show that same-point domain gradients may remain nearly even when consecutive realized updates partially reverse one another in output space.<n>We propose OSOL, which designates a focus domain at each iteration, to rank token-level rebound risk.<n>OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
  arXiv  Detail & Related papers  (2026-09-06T08:32:59Z)
• Decision-Focused Learning via Tangent-Space Projection of Prediction Error [32.655683009031804]
  Decision-Focused Learning (DFL) trains predictors to improve downstream decision quality.<n>We show that, under standard regularity with locally stable active constraints, the regret gradient admits a closed-form geometric characterization.<n>We propose PEAR, which computes regret gradients via a reduced linear system over active constraints.
  arXiv  Detail & Related papers  (2026-05-02T10:10:16Z)
• Accelerating trajectory optimization with Sobolev-trained diffusion policies [12.684196947818103]
  Trajectory Optimization (TO) solvers exploit known system dynamics to compute locally optimal trajectories.<n>A downside is that each new problem instance is solved independently.<n>To improve efficiency, a natural approach is to warm-start TO with initial guesses produced by a learned policy.
  arXiv  Detail & Related papers  (2026-04-21T03:00:52Z)
• Geometric-Disentangelment Unlearning [106.99160454669902]
  gradient ascent on forget samples often harms retained knowledge.<n>We propose the Geometric-disment Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component.<n>Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects.
  arXiv  Detail & Related papers  (2025-11-21T09:58:25Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration [52.70324949884702]
  We quantify the excess risk incurred using approximate posterior probabilities in batch binary decision-making.<n>We identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss.<n>On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost.
  arXiv  Detail & Related papers  (2025-03-23T10:52:36Z)
• To transfer or not transfer: Unified transferability metric and analysis [13.274835852615576]
  In transfer learning, transferability is one of the most fundamental problems, which aims to evaluate the effectiveness of arbitrary transfer tasks. We propose a new analytical approach and metric, Wasserstein Distance based Joint Estimation, for transferability estimation. We approximate the target transfer risk by proposing a non-symmetric, easy-to-understand and easy-to-calculate target risk bound.
  arXiv  Detail & Related papers  (2023-05-12T19:52:11Z)
• Transferability Estimation Based On Principal Gradient Expectation [68.97403769157117]
  Cross-task transferability is compatible with transferred results while keeping self-consistency. Existing transferability metrics are estimated on the particular model by conversing source and target tasks. We propose Principal Gradient Expectation (PGE), a simple yet effective method for assessing transferability across tasks.
  arXiv  Detail & Related papers  (2022-11-29T15:33:02Z)
• Correcting Momentum in Temporal Difference Learning [95.62766731469671]
  We argue that momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale. We show that this phenomenon exists, and then propose a first-order correction term to momentum. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.
  arXiv  Detail & Related papers  (2021-06-07T20:41:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.