Model-free reinforcement learning with noisy actions for automated experimental control in optics

• URL: http://arxiv.org/abs/2405.15421v2
• Date: Thu, 13 Feb 2025 10:26:45 GMT
• Title: Model-free reinforcement learning with noisy actions for automated experimental control in optics
• Authors: Lea Richtmann, Viktoria-S. Schmiesing, Dennis Wilken, Jan Heine, Aaron Tranter, Avishek Anand, Tobias J. Osborne, Michèle Heurs,
• Abstract summary: We show that reinforcement learning can overcome challenges when coupling laser light into an optical fiber.<n>By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC) or Truncated Quantile Critics (TQC), our agent learns to couple with 90% efficiency, comparable to the human expert.
• Score: 2.3003734964536524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC) or Truncated Quantile Critics (TQC), our agent learns to couple with 90% efficiency, comparable to the human expert. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.

Related papers

• Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems [42.2790464348673]
  We propose a multi-fidelity reinforcement learning framework for controlling instabilities in complex systems. The effect of the proposed framework is demonstrated on two complex dynamics in physics.
  arXiv  Detail & Related papers  (2025-04-08T00:50:15Z)
• Robust Quantum Control using Reinforcement Learning from Demonstration [13.321147424579065]
  We use Reinforcement Learning from Demonstration (RLfD) to leverage the control sequences generated with system models. This approach can increase sample efficiency by reducing the number of samples, which can significantly reduce the training time. We have simulated the preparation of several high-fidelity non-classical states using the RLfD method.
  arXiv  Detail & Related papers  (2025-03-27T02:01:28Z)
• Automating Experimental Optics with Sample Efficient Machine Learning Methods [0.47936618873102926]
  We show how machine learning can be used to achieve autonomous mode-matching of a free-space optical resonator with minimal supervision. In this work, we demonstrate how machine learning can be used to achieve autonomous mode-matching of a free-space optical resonator with minimal supervision.
  arXiv  Detail & Related papers  (2025-03-18T13:50:44Z)
• SHIRE: Enhancing Sample Efficiency using Human Intuition in REinforcement Learning [11.304750795377657]
  We propose SHIRE, a framework for encoding human intuition using Probabilistic Graphical Models (PGMs) SHIRE achieves 25-78% sample efficiency gains across the environments we evaluate at negligible overhead cost.
  arXiv  Detail & Related papers  (2024-09-16T04:46:22Z)
• Benchmarking Reinforcement Learning Methods for Dexterous Robotic Manipulation with a Three-Fingered Gripper [0.7364531214545392]
  Reinforcement Learning (RL) training is predominantly conducted in cost-effective and controlled simulation environments. This research explores the direct training of RL algorithms in controlled yet realistic real-world settings for the execution of dexterous manipulation.
  arXiv  Detail & Related papers  (2024-08-27T02:52:15Z)
• Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight [20.92646531472541]
  We propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) Our framework contains three phases teacher policy using RL with privileged state information distilling it into a student policy via IL, and adaptive fine-tuning via RL. Tests show our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance.
  arXiv  Detail & Related papers  (2024-03-18T19:25:57Z)
• On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning [71.44986275228747]
  In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs) However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
  arXiv  Detail & Related papers  (2023-12-21T11:55:10Z)
• Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition [10.072992621244042]
  We propose a novel method for skill learning in robotic manipulation called Tactile Active Inference Reinforcement Learning (Tactile-AIRL) To enhance the performance of reinforcement learning (RL), we introduce active inference, which integrates model-based techniques and intrinsic curiosity into the RL process. We demonstrate that our method achieves significantly high training efficiency in non-prehensile objects pushing tasks.
  arXiv  Detail & Related papers  (2023-11-19T10:19:22Z)
• Hindsight States: Blending Sim and Real Task Elements for Efficient Reinforcement Learning [61.3506230781327]
  In robotics, one approach to generate training data builds on simulations based on dynamics models derived from first principles. Here, we leverage the imbalance in complexity of the dynamics to learn more sample-efficiently. We validate our method on several challenging simulated tasks and demonstrate that it improves learning both alone and when combined with an existing hindsight algorithm.
  arXiv  Detail & Related papers  (2023-03-03T21:55:04Z)
• SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering [49.78647219715034]
  We propose a sensing-aware model-based reinforcement learning system called SAM-RL. With the sensing-aware learning pipeline, SAM-RL allows a robot to select an informative viewpoint to monitor the task process. We apply our framework to real world experiments for accomplishing three manipulation tasks: robotic assembly, tool manipulation, and deformable object manipulation.
  arXiv  Detail & Related papers  (2022-10-27T05:30:43Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Toward Fast, Flexible, and Robust Low-Light Image Enhancement [87.27326390675155]
  We develop a new Self-Calibrated Illumination (SCI) learning framework for fast, flexible, and robust brightening images in real-world low-light scenarios. Considering the computational burden of the cascaded pattern, we construct the self-calibrated module which realizes the convergence between results of each stage. We make comprehensive explorations to SCI's inherent properties including operation-insensitive adaptability and model-irrelevant generality.
  arXiv  Detail & Related papers  (2022-04-21T14:40:32Z)
• Accelerated Policy Learning with Parallel Differentiable Simulation [59.665651562534755]
  We present a differentiable simulator and a new policy learning algorithm (SHAC) Our algorithm alleviates problems with local minima through a smooth critic function. We show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms.
  arXiv  Detail & Related papers  (2022-04-14T17:46:26Z)
• Autonomous Reinforcement Learning: Formalism and Benchmarking [106.25788536376007]
  Real-world embodied learning, such as that performed by humans and animals, is situated in a continual, non-episodic world. Common benchmark tasks in RL are episodic, with the environment resetting between trials to provide the agent with multiple attempts. This discrepancy presents a major challenge when attempting to take RL algorithms developed for episodic simulated environments and run them on real-world platforms.
  arXiv  Detail & Related papers  (2021-12-17T16:28:06Z)
• Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation [91.05073136215886]
  "Actor-Learner Distillation" transfers learning progress from a large capacity learner model to a small capacity actor model. We demonstrate in several challenging memory environments that using Actor-Learner Distillation recovers the clear sample-efficiency gains of the transformer learner model.
  arXiv  Detail & Related papers  (2021-04-04T17:56:34Z)
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning [12.193817049957733]
  Insufficient learning (due to convergence to local optima) results in under-performing policies whilst redundant learning wastes time and resources. We propose an auto-tuning technique based on the Evidence Lower Bound (ELBO) for self-supervised reinforcement learning. Our method can auto-tune online and yields the best performance at a fraction of the time and computational resources.
  arXiv  Detail & Related papers  (2020-10-16T08:58:24Z)
• Information Theoretic Model Predictive Q-Learning [64.74041985237105]
  We present a novel theoretical connection between information theoretic MPC and entropy regularized RL. We develop a Q-learning algorithm that can leverage biased models.
  arXiv  Detail & Related papers  (2019-12-31T00:29:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.