Efficient Geothermal Well-Control Optimization via Diffusion-Surrogate Reinforcement Learning

• URL: http://arxiv.org/abs/2608.28791v1
• Date: Fri, 28 Aug 2026 18:52:57 GMT
• Title: Efficient Geothermal Well-Control Optimization via Diffusion-Surrogate Reinforcement Learning
• Abstract summary: Real-time decision-making for enhanced geothermal systems (EGS) is challenging because long-term production periods involve high-dimensional control spaces.<n>We propose a diffusion-surrogate guided reinforcement learning framework for long-horizon EGS well-control optimization.<n>Experiments on a fractured EGS benchmark show that the diffusion surrogate can accurately reproduce reservoir-state evolution over multiple control stages.
• Score: 5.172885722898411
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time decision-making for enhanced geothermal systems (EGS) is challenging because long-term production periods involve high-dimensional control spaces and a large number of time-consuming high-fidelity hydrothermal simulations. Reinforcement learning provides a natural framework for state-dependent sequential control, but direct policy training with numerical simulators is computationally expensive. To address this issue, we propose a diffusion-surrogate guided reinforcement learning framework for long-horizon EGS well-control optimization. The reservoir temperature and pressure fields are used as system states, while injection rates are selected as control actions. A learned surrogate environment is constructed using conditional diffusion models to predict the evolution of reservoir temperature and pressure fields and a separate reward model to estimate the corresponding economic return. The surrogate environment is then integrated with Proximal Policy Optimization (PPO) for efficient policy training. Experiments on a fractured EGS benchmark show that the diffusion surrogate can accurately reproduce reservoir-state evolution over multiple control stages. The resulting surrogate-assisted PPO policy achieves competitive well-control performance compared with direct simulator-based PPO and existing optimization methods, while substantially reducing the dependence on expensive high-fidelity simulations. These results demonstrate the potential of diffusion-based surrogate environments for efficient reinforcement learning in geothermal well-control optimization.

Related papers

• Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization [85.4863852505905]
  We introduce unsupervised reward optimization of protein language models (PLMs)<n>Our key insight is that task-agnostic rewards, which combine intrinsic model uncertainty with semantic consistency informed by protein representation models, exhibit strong correlation with controllability measures across base models and temperature regimes.<n>Our framework provides a scalable pathway toward controllable biomolecular design in settings where labeled preferences or experimental feedback are scarce or unavailable.
  arXiv  Detail & Related papers  (2026-06-17T11:42:01Z)
• Deep Learning-Accelerated Surrogate Optimization for High-Dimensional Well Control in Stress-Sensitive Reservoirs [0.0]
  Production optimization in stress-sensitive unconventional reservoirs is governed by a nonlinear trade-off between pressure-driven flow and stress-induced degradation of fracture conductivity and matrix permeability.<n>We propose a deep learning-based surrogate optimization framework for high-dimensional well control.
  arXiv  Detail & Related papers  (2026-04-01T01:04:08Z)
• Adaptive Uncertainty-Guided Surrogates for Efficient phase field Modeling of Dendritic Solidification [0.24629531282150874]
  This work presents a surrogate model for dendritic solidification that employs uncertainty-driven adaptive sampling with micro XG and CNNs.<n>The framework systematically investigates how temporal selection, adaptive sampling, and the choice between domain-informed data-driven surrogates affect Xtemporal- model performance.
  arXiv  Detail & Related papers  (2026-02-17T10:43:24Z)
• AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Models [54.56296715999545]
  Reinforcement learning from human feedback shows promise for aligning diffusion and flow models.<n>Policy optimization methods such as GRPO suffer from inefficient and static sampling strategies.<n>We propose Adaptive Entropy-Guided Policy Optimization (AEGPO), a novel dual-signal, dual-level adaptive optimization strategy.
  arXiv  Detail & Related papers  (2026-02-06T16:09:50Z)
• Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving [54.46325690390831]
  We propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment.<n>MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine.<n>MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes.
  arXiv  Detail & Related papers  (2025-11-26T17:01:41Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design [58.8094854658848]
  We address the problem of fine-tuning diffusion models for reward-guided generation in biomolecular design.<n>We propose an iterative distillation-based fine-tuning framework that enables diffusion models to optimize for arbitrary reward functions.<n>Our off-policy formulation, combined with KL divergence minimization, enhances training stability and sample efficiency compared to existing RL-based methods.
  arXiv  Detail & Related papers  (2025-07-01T05:55:28Z)
• Comparison of Model Predictive Control and Proximal Policy Optimization for a 1-DOF Helicopter System [0.7499722271664147]
  This study conducts a comparative analysis of Model Predictive Control (MPC) and Proximal Policy Optimization (PPO), a Deep Reinforcement Learning (DRL) algorithm, applied to a Quanser Aero 2 system. PPO excels in rise-time and adaptability, making it a promising approach for applications requiring rapid response and adaptability.
  arXiv  Detail & Related papers  (2024-08-28T08:35:34Z)
• Extremum-Seeking Action Selection for Accelerating Policy Optimization [18.162794442835413]
  Reinforcement learning for control over continuous spaces typically uses high-entropy policies, such as Gaussian distributions, for local exploration and estimating policy to optimize performance. We propose to improve action selection in this model-free RL setting by introducing additional adaptive control steps based on Extremum-Seeking Control (ESC) Our methods can be easily added in standard policy optimization to improve learning efficiency, which we demonstrate in various control learning environments.
  arXiv  Detail & Related papers  (2024-04-02T02:39:17Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Deep learning based closed-loop optimization of geothermal reservoir production [0.0]
  We propose a closed-loop optimization framework, based on deep learning surrogates, for the well control optimization of geothermal reservoirs. We construct a hybrid convolution-recurrent neural network surrogate, which combines the convolution neural network (CNN) and long short-term memory (LSTM) recurrent network. We show that the proposed framework can achieve efficient and effective real-time optimization and data assimilation in the geothermal reservoir production process.
  arXiv  Detail & Related papers  (2022-04-15T14:37:28Z)
• Variance Reduction based Experience Replay for Policy Optimization [3.0790370651488983]
  Variance Reduction Experience Replay (VRER) is a framework for the selective reuse of relevant samples to improve policy gradient estimation. VRER forms the foundation of our sample efficient off-policy learning algorithm known as Policy Gradient with VRER.
  arXiv  Detail & Related papers  (2021-10-17T19:28:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.