Dual-Loop Control in DCVerse: Advancing Reliable Deployment of AI in Data Centers via Digital Twins

• URL: http://arxiv.org/abs/2604.07559v1
• Date: Wed, 08 Apr 2026 20:01:34 GMT
• Title: Dual-Loop Control in DCVerse: Advancing Reliable Deployment of AI in Data Centers via Digital Twins
• Abstract summary: Deep Reinforcement Learning (DRL) shows strong potential for intelligent control, but its deployment in mission-critical systems is limited by data scarcity and the lack of real-time pre-evaluation mechanisms.<n>This paper introduces the Dual-Loop Control Framework (DLCF), a digital twin-based architecture designed to overcome these challenges.<n>The framework comprises three core entities: the physical system, a digital twin, and a policy reservoir of diverse DRL agents.
• Score: 9.44096782663678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing scale and complexity of modern data centers present major challenges in balancing energy efficiency with outage risk. Although Deep Reinforcement Learning (DRL) shows strong potential for intelligent control, its deployment in mission-critical systems is limited by data scarcity and the lack of real-time pre-evaluation mechanisms. This paper introduces the Dual-Loop Control Framework (DLCF), a digital twin-based architecture designed to overcome these challenges. The framework comprises three core entities: the physical system, a digital twin, and a policy reservoir of diverse DRL agents. These components interact through a dual-loop mechanism involving real-time data acquisition, data assimilation, DRL policy training, pre-evaluation, and expert verification. Theoretical analysis shows how DLCF can improve sample efficiency, generalization, safety, and optimality. Leveraging DLCF, we implemented the DCVerse platform and validated it through case studies on a real-world data center cooling system. The evaluation shows that our approach achieves up to 4.09% energy savings over conventional control strategies without violating SLA requirements. Additionally, the framework improves policy interpretability and supports more trustworthy DRL deployment. This work provides a foundation for reliable AI-based control in data centers and points toward future extensions for holistic, system-wide optimization.

Related papers

• Securing Autonomous Vehicle Systems via Twin-Aware Federated Reinforcement Learning [18.401916683479676]
  poisoning attacks pose a significant threat to the security and reliability of Federated reinforcement learning (FRL) systems.<n>alg is a framework aimed at enhancing the robustness of FRL systems designed for safety-critical driving scenarios.<n>alg strategically integrates digital twins for rehearsal-based learning and leverages historical aggregated model parameters.
  arXiv  Detail & Related papers  (2026-07-09T06:18:52Z)
• High-Precision Formation Control for Heterogeneous Multi-Robot Systems via Hierarchical Hybrid Physics-Informed Deep Reinforcement Learning [10.865402396881416]
  This paper proposes a hierarchical hybrid physics-informed deep reinforcement learning (HHy-PIDRL) framework.<n>It aims to realize high-precision, highly responsive formation control for heterogeneous multi-robot systems.<n> Experimental results demonstrate that, the success rate of both the upper-layer autonomous navigation policy network and the HM-DRL based formation control policy networks reach 100%.
  arXiv  Detail & Related papers  (2026-07-03T17:38:17Z)
• Fighting Numerical Hallucinations via Data-centric Compilation for Online Financial QA [14.141735278960766]
  Large Language Models (LLMs) have significantly advanced online data services, particularly in the domain of financial question answering (FinQA)<n>However, such systems remain susceptible to numerical reasoning hallucinations, which critically undermine reliability in high-stakes financial applications.<n>We propose a data-centric paradigm and propose a novel framework, the Data-centric Reasoning Compiler (DCRC)<n>We conduct extensive experiments on established offline benchmarks and further validate our framework through deployment in a real-world online financial QA system.
  arXiv  Detail & Related papers  (2026-05-29T09:35:11Z)
• TwinRL-VLA: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation [65.45588646626426]
  We propose TwinRL, a digital twin-real-world collaborative RL framework designed to scale and guide exploration for VLA models.<n>First, a high-fidelity digital twin is efficiently reconstructed from smartphone-captured scenes, enabling realistic bidirectional transfer between real and simulated environments.<n>In our experiments, TwinRL approaches 100% success in both in-distribution regions covered by real-world demonstrations and out-of-distribution regions, delivering at least a 30% speedup over prior real-world RL methods.
  arXiv  Detail & Related papers  (2026-02-09T18:59:52Z)
• Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration [56.074760766965085]
  PRISM achieves a dynamics-aware framework that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge.<n>Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
  arXiv  Detail & Related papers  (2026-01-12T05:43:20Z)
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting [91.38734024438357]
  Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs)<n>Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data.<n>We propose CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting.
  arXiv  Detail & Related papers  (2025-08-15T11:20:03Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Transforming Future Data Center Operations and Management via Physical AI [24.063748316223343]
  Data centers as mission-critical infrastructures are pivotal in powering the growth of artificial intelligence (AI) and the digital economy.<n>The evolution from Internet DC to AI DC has introduced new challenges in operating and managing data centers for improved business resilience and reduced total cost of ownership.<n>We propose and develop a novel Physical AI (PhyAI) framework for advancing DC operations and management.
  arXiv  Detail & Related papers  (2025-04-07T12:09:22Z)
• D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning [99.33607114541861]
  We propose a new benchmark for offline RL that focuses on realistic simulations of robotic manipulation and locomotion environments. Our proposed benchmark covers state-based and image-based domains, and supports both offline RL and online fine-tuning evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:27:00Z)
• Experimental evaluation of offline reinforcement learning for HVAC control in buildings [12.542463083734614]
  Reinforcement learning (RL) techniques have been increasingly investigated for dynamic HVAC control in buildings. This paper comprehensively evaluates the strengths and limitations of state-of-the-art offline RL algorithms.
  arXiv  Detail & Related papers  (2024-08-15T07:25:52Z)
• Digital Twin-Enhanced Deep Reinforcement Learning for Resource Management in Networks Slicing [46.65030115953947]
  We propose a framework consisting of a digital twin and reinforcement learning agents. Specifically, we propose to use the historical data and the neural networks to build a digital twin model to simulate the state variation law of the real environment. We also extend the framework to offline reinforcement learning, where solutions can be used to obtain intelligent decisions based solely on historical data.
  arXiv  Detail & Related papers  (2023-11-28T15:25:14Z)
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
  We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
  arXiv  Detail & Related papers  (2023-04-20T17:11:05Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Federated Deep Reinforcement Learning for the Distributed Control of NextG Wireless Networks [16.12495409295754]
  Next Generation (NextG) networks are expected to support demanding internet tactile applications such as augmented reality and connected autonomous vehicles. Data-driven approaches can improve the ability of the network to adapt to the current operating conditions. Deep RL (DRL) has been shown to achieve good performance even in complex environments.
  arXiv  Detail & Related papers  (2021-12-07T03:13:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.