Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models

• URL: http://arxiv.org/abs/2604.25859v2
• Date: Sat, 02 May 2026 20:40:07 GMT
• Title: Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models
• Abstract summary: We show that joint training induces an action-conditioned correction that privileged future observations impose on action denoising.<n>We introduce emphPrivileged Foresight Distillation (PFD), which transfers this residual from a training-time teacher into a small adapter on a current-only student.
• Score: 1.5821415295352772
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: World action models jointly predict future video and action during training, raising an open question about what role the future-prediction branch actually plays. A recent finding shows that this branch can be removed at inference with little to no loss on common manipulation benchmarks, suggesting that future information may act merely as a regularizer on the shared visual backbone. We propose instead that joint training induces an action-conditioned correction that privileged future observations impose on action denoising, and that current-only policies capture this correction only partially. Making the account precise, we formulate privileged foresight as a residual in the action-denoising direction -- the difference between what a model predicts given the true future and what it predicts given only the current frame -- and introduce \emph{Privileged Foresight Distillation (PFD)}, which transfers this residual from a training-time teacher into a small adapter on a current-only student. The teacher and student share the same backbone and differ only in the attention mask over video tokens; future video is never generated at inference. Controlled experiments verify that this gain reflects a genuine future-conditioned correction rather than a side effect of capacity or regularization. Empirically, PFD achieves consistent improvements on LIBERO and RoboTwin manipulation benchmarks while preserving the current-only inference interface at negligible added latency. This view reframes the role of future information in world action models: not as a target to predict, nor as a regularizer to absorb, but as a compressible correction to be distilled.

Related papers

• JEPA Policy: Diffusion-Free Imitation Learning via Paired Action and Future Representation Prediction [6.889062419237111]
  JEPA Policy is a diffusion-free framework that uses the action chunk and its observed future representation as paired training targets.<n>Future prediction can therefore shape the representation used to generate actions.<n>Results support paired future-representation supervision as a practical approach to low-latency visuomotor imitation without iterative generative sampling.
  arXiv  Detail & Related papers  (2026-09-09T02:42:20Z)
• Learning to Track from Privileged Target Appearances [67.80976117600649]
  We quantify a bottleneck with a non-deployable oracle that supplies an exact current-frame target crop.<n>We introduce Privileged Appearance Transfer for Tracking (PATT), a teacher-student training framework that transfers these privileged appearances to a deployable tracker.<n>PATT achieves consistent gains under both long- and short-term tracking protocols.
  arXiv  Detail & Related papers  (2026-09-02T11:43:43Z)
• Counterfactual Quotient Models: Learning What Actions Change, Not What the World Does [14.67340173071927]
  Reinforcement-learning models commonly predict complete future states, observations, or feature occupancies.<n>We introduce the Counterfactual Quotient Model, which treats action-conditioned futures as equivalent when they differ only by a component shared across actions.
  arXiv  Detail & Related papers  (2026-08-22T19:55:57Z)
• Foresight Without Seeing: Latent Futures for World Action Models [31.423405133926334]
  We propose ForeWAM, a dynamics-conditioned direct-policy WAM that provides predictive context for action generation without decoding future videos.<n>We also introduce dynamics supervised by a frozen latent action teacher, encouraging the implicit future states to capture interaction-induced transitions such as object motion, contact changes, and task progress.<n>Without embodied robot data pretraining, the standard and accelerated variants of ForeWAM achieve average success rates of 96.7% and 96.9% on LIBERO, respectively.
  arXiv  Detail & Related papers  (2026-08-12T03:27:43Z)
• Learning from the Future: Privileged Self-Distillation for Sequential Recommendation [51.993699802866956]
  We propose a framework that separates learning-time information from inference-time information.<n> Privileged Self-Distillation (PSD) separates learning-time information from inference-time information.<n>PSD is optimized end-to-end in a single stage, leaving the deployed model and inference cost unchanged.
  arXiv  Detail & Related papers  (2026-07-29T15:47:22Z)
• From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation [88.39072412680633]
  We propose MoLA, a control-oriented interface that transforms imagined future videos into executable representations.<n>We evaluate our approach on simulated benchmarks and real-world robot manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-12T14:15:16Z)
• Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback [41.94295877935867]
  In automated driving, predicting trajectories of surrounding vehicles supports reasoning about scene dynamics and enables safe planning for the ego vehicle.<n>Existing models handle predictions as an instantaneous task of forecasting future trajectories based on observed information.<n>We propose a novel retrospection technique to correct its errors during inference and will repeat them.
  arXiv  Detail & Related papers  (2025-04-18T16:35:12Z)
• Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation [42.63061599979695]
  Speech separation (SS) seeks to disentangle a multi-talker speech mixture into single-talker speech streams.<n> Causal separation models, which rely only on past and present information, offer a promising solution for real-time streaming.<n>We introduce a novel that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models.
  arXiv  Detail & Related papers  (2025-04-03T06:18:30Z)
• Generative Regression Based Watch Time Prediction for Short-Video Recommendation [36.95095097454143]
  Watch time prediction has emerged as a pivotal task in short video recommendation systems.<n>Recent studies have attempted to address these issues by converting the continuous watch time estimation into an ordinal regression task.<n>We propose a novel Generative Regression (GR) framework that reformulates WTP as a sequence generation task.
  arXiv  Detail & Related papers  (2024-12-28T16:48:55Z)
• Debiased Fine-Tuning for Vision-language Models by Prompt Regularization [56.48290708901531]
  We present a new paradigm for fine-tuning large-scale vision pre-trained models on downstream task, dubbed Prompt Regularization (ProReg)<n>ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning.<n>We show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-01-29T11:53:55Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• Correcting Momentum in Temporal Difference Learning [95.62766731469671]
  We argue that momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale. We show that this phenomenon exists, and then propose a first-order correction term to momentum. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.
  arXiv  Detail & Related papers  (2021-06-07T20:41:15Z)
• Reducing Representation Drift in Online Continual Learning [87.71558506591937]
  We study the online continual learning paradigm, where agents must learn from a changing distribution with constrained memory and compute. In this work we instead focus on the change in representations of previously observed data due to the introduction of previously unobserved class samples in the incoming data stream.
  arXiv  Detail & Related papers  (2021-04-11T15:19:30Z)
• Unsupervised Video Representation Learning by Bidirectional Feature Prediction [16.074111448606512]
  This paper introduces a novel method for self-supervised video representation learning via feature prediction. We argue that a supervisory signal arising from unobserved past frames is complementary to one that originates from the future frames. We empirically show that utilizing both signals enriches the learned representations for the downstream task of action recognition.
  arXiv  Detail & Related papers  (2020-11-11T19:42:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.