Interpretable Policy Distillation for Power Grid Topology Control

• URL: http://arxiv.org/abs/2606.00561v1
• Date: Sat, 30 May 2026 06:32:55 GMT
• Title: Interpretable Policy Distillation for Power Grid Topology Control
• Abstract summary: We ask whether a Proximal Policy Optimization (PPO) agent can be compressed into compact tree-based surrogates without losing operational performance.<n>A PPO teacher is trained on Grid2Op's standard 14-bus environment with a stability-oriented reward, using stress-focused data collection on critical, high-loading states.<n>Both surrogates exceed the teacher in mean reward and survival length at a fraction of the inference cost.
• Score: 46.15556541260613
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep reinforcement learning (RL) offers a promising route to real-time power grid operation, yet large neural policies are costly to evaluate, hard to deploy on constrained hardware, and opaque to operators. We ask whether a Proximal Policy Optimization (PPO) agent for grid topology control can be compressed into compact tree-based surrogates without losing operational performance. A PPO teacher is trained on Grid2Op's standard 14-bus environment with a stability-oriented reward, using stress-focused data collection on critical, high-loading states. The policy is then distilled into a decision tree and a random forest. Across held-out validation episodes, both surrogates exceed the teacher in mean reward and survival length at a fraction of the inference cost. The decision tree shows high exact-action agreement with the PPO argmax and near-complete agreement within its top-ranked actions, while remaining small enough to be inspected directly. Feature-importance analysis reveals a representational shift: the PPO policy relies mainly on line-loading signals, while the distilled tree is driven primarily by bus-topology variables. These results suggest that stress-focused distillation can convert a black-box neural controller into a lightweight, auditable rule-like surrogate suited for real-time deployment, while also surfacing risks tied to deterministic actions and topology-specific generalization.

Related papers

• Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization [27.121081058414376]
  Policy-KL constrains response behavior and consumes the action-side exploration budget.<n>We argue for an alternative that breaks the dilemma by moving regularization to the input side.<n>ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift.
  arXiv  Detail & Related papers  (2026-08-24T14:30:59Z)
• SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning [0.0]
  SPOT (Sampling Policy Observation Tree) is a model-agnostic, sampling-based framework for interpreting DRL policies.<n>We show how SPOT constructs an interpretable finite-horizon tree by sampling actions and simulating the resulting successor states.<n>The study illustrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods.
  arXiv  Detail & Related papers  (2026-07-28T07:57:18Z)
• Rethinking the Divergence Regularization in LLM RL [56.952725733148746]
  Reinforcement learning (RL) has become a key component of post-training large language models (LLMs)<n>We propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift.<n>DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights.
  arXiv  Detail & Related papers  (2026-06-08T17:58:23Z)
• gym-invmgmt: An Open Benchmarking Framework for Inventory Management Methods [1.0799969476894555]
  Gym-invmgmt is an extension of the OR-Gym inventory-management lineage for auditable cross-paradigm evaluation.<n>We present gym-invmgmt, a Gymnasium-compatible extension of the OR-Gym inventory-management lineage for auditable cross-paradigm evaluation.
  arXiv  Detail & Related papers  (2026-05-12T00:22:15Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers [5.690649768462432]
  We present ContractionPPO, a framework for robust planning and control of legged robots.<n>ContractionPPO produces a contraction metric that certifies exponential stability of the simulated closed-loop system.<n>Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.
  arXiv  Detail & Related papers  (2026-03-20T04:32:18Z)
• Robust Regularized Policy Iteration under Transition Uncertainty [6.7431287237221085]
  We formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set.<n>We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate.<n>Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines.
  arXiv  Detail & Related papers  (2026-03-10T08:18:27Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models [2.1418081649543157]
  HiPP-Prune is a hierarchical preference-conditioned structured pruning framework that treats pruning as conditional resource allocation.<n>HiPP-Prune makes plan-level decisions by factorizing decisions into an overall sparsity budget and a layer-wise allocation.
  arXiv  Detail & Related papers  (2026-03-06T13:31:54Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG [71.06073770344732]
  Agentic retrieval-augmented generation (RAG) formulates question answering as a multi-step interaction between reasoning and information retrieval.<n>We present TreePS-RAG, an online, tree-based RL framework for agentic RAG that enables step-wise credit assignment while retaining outcome-only rewards.
  arXiv  Detail & Related papers  (2026-01-11T14:07:30Z)
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards [47.557539197058496]
  We introduce Random Policy Valuation for Diverse Reasoning (ROVER)<n>ROVER is a minimalist yet highly effective RL method that samples actions from a softmax over uniform-policy Q-values.<n>It demonstrates superior performance in both textbfquality (textbf+8.2 on pass@1, textbf+16.8 on pass@256) and textbfdiversity (textbf+17.6%)
  arXiv  Detail & Related papers  (2025-09-29T16:09:07Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.