$\text{DT}^2$: Decision-Targeted Digital Twins

• URL: http://arxiv.org/abs/2606.25923v1
• Date: Wed, 24 Jun 2026 15:02:24 GMT
• Title: $\text{DT}^2$: Decision-Targeted Digital Twins
• Abstract summary: A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies.<n>We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function.<n>We introduce $textDT2$, a decision-targeted DT training paradigm.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A digital twin (DT) is a virtual model of a real-world system that can assist decision-making by simulating scenarios induced by different policies. However, typical machine learning-based DTs do not optimise for this use case. We prove that, when model capacity is limited, training DTs to minimise one-step transition errors can produce suboptimal models for ranking sets of policies according to a reward function. We further show that this holds empirically, even with expressive model classes. To address this, we introduce $\text{DT}^2$, a decision-targeted DT training paradigm. Firstly, $\text{DT}^2$ uses fitted Q-evaluation to estimate values of candidate policies from offline data. A DT is then trained to generate rollouts that preserve pairwise policy rankings derived from these proxy ground-truth values with an architecture-agnostic loss function. We empirically demonstrate the efficacy of our method across a range of settings and architectures. $\text{DT}^2$ consistently improves policy ranking and reduces decision regret during policy selection relative to conventional DT training, both for policies used during training and for unseen policies, while maintaining a good level of raw simulation fidelity.

Related papers

• CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning [6.586422759470378]
  We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy.<n>We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2ruped performing a dynamic handstand.
  arXiv  Detail & Related papers  (2026-09-08T15:05:16Z)
• $β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation [85.07190808882523]
  On-policy self-distillation is a promising approach to improve reasoning language models.<n>We show that $$-OPSD consistently outperforms vanilla OPSD.
  arXiv  Detail & Related papers  (2026-07-30T17:41:16Z)
• Bayesian Experimental Design via Score Matching [15.998867609597545]
  We show that the double intractability of the expected information gain can be isolated from the policy learning.<n>This turns the key multiplicative cost into an additive one and reduces the computational burden on the policy training itself.<n>In our experiments we train multiple competitive policies without inducing a multiplicative cost in likelihood evaluations.
  arXiv  Detail & Related papers  (2026-07-09T10:21:06Z)
• Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning [33.899779762210976]
  Supervised fine-tuning (SFT) of large language models can be viewed as an off-policy learning problem.<n>Existing methods mitigate this issue with KL penalties or clipping, which passively updates rather than actively reducing the gap.<n>We propose a simple yet effective data rewriting framework that proactively shrinks the policy gap before training.
  arXiv  Detail & Related papers  (2025-09-18T17:02:30Z)
• IF-GUIDE: Influence Function-Guided Detoxification of LLMs [53.051109450536885]
  We study how training data contributes to the emergence of toxic behaviors in large-language models.<n>We propose a $proactive approach that leverages influence functions to identify harmful tokens within any training data and suppress their impact during training.<n>We present a novel adaptation that measures token-level attributions from training data to model toxicity, along with techniques for selecting toxic training documents and a learning objective.
  arXiv  Detail & Related papers  (2025-06-02T15:32:36Z)
• TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint [11.347808936693152]
  Model-based reinforcement learning algorithms combine model-based planning and learned value/policy prior.<n>Existing methods that rely on standard SAC-style policy iteration for value learning often result in emphpersistent value overestimation.<n>We propose a policy regularization term reducing out-of-distribution (OOD) queries, thereby improving value learning.
  arXiv  Detail & Related papers  (2025-02-05T19:08:42Z)
• Achieving $\widetilde{\mathcal{O}}(\sqrt{T})$ Regret in Average-Reward POMDPs with Known Observation Models [69.1820058966619]
  We tackle average-reward infinite-horizon POMDPs with an unknown transition model.<n>We present a novel and simple estimator that overcomes this barrier.
  arXiv  Detail & Related papers  (2025-01-30T22:29:41Z)
• $Δ\text{-}{\rm OPE}$: Off-Policy Estimation with Pairs of Policies [13.528097424046823]
  We introduce $Deltatext-rm OPE$ methods based on the widely used Inverse Propensity Scoring estimator. Simulated, offline, and online experiments show that our methods significantly improve performance for both evaluation and learning tasks.
  arXiv  Detail & Related papers  (2024-05-16T12:04:55Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Co-learning Planning and Control Policies Constrained by Differentiable Logic Specifications [4.12484724941528]
  This paper presents a novel reinforcement learning approach to solving high-dimensional robot navigation tasks. We train high-quality policies with much fewer samples compared to existing reinforcement learning algorithms. Our approach also demonstrates capabilities for high-dimensional control and avoiding suboptimal policies via policy alignment.
  arXiv  Detail & Related papers  (2023-03-02T15:24:24Z)
• Policy Gradient for Rectangular Robust Markov Decision Processes [62.397882389472564]
  We introduce robust policy gradient (RPG), a policy-based method that efficiently solves rectangular robust Markov decision processes (MDPs) Our resulting RPG can be estimated from data with the same time complexity as its non-robust equivalent.
  arXiv  Detail & Related papers  (2023-01-31T12:40:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.