Offline Reinforcement Learning for Learning to Dispatch for Job Shop Scheduling

• URL: http://arxiv.org/abs/2409.10589v1
• Date: Mon, 16 Sep 2024 15:18:10 GMT
• Title: Offline Reinforcement Learning for Learning to Dispatch for Job Shop Scheduling
• Authors: Jesse van Remmerden, Zaharah Bukhsh, Yingqian Zhang,
• Abstract summary: We introduce Offline Reinforcement Learning for Learning to Dispatch (Offline-LD), a novel approach for Job Shop Scheduling Problem (JSSP) Offline-LD adapts two CQL-based Q-learning methods for maskable action spaces, introduces a new entropy bonus modification for discrete SAC, and exploits reward normalization through preprocessing. Our experiments show that Offline-LD outperforms online RL on both generated and benchmark instances.
• Score: 0.9831489366502301
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Job Shop Scheduling Problem (JSSP) is a complex combinatorial optimization problem. There has been growing interest in using online Reinforcement Learning (RL) for JSSP. While online RL can quickly find acceptable solutions, especially for larger problems, it produces lower-quality results than traditional methods like Constraint Programming (CP). A significant downside of online RL is that it cannot learn from existing data, such as solutions generated from CP, requiring them to train from scratch, leading to sample inefficiency and making them unable to learn from more optimal examples. We introduce Offline Reinforcement Learning for Learning to Dispatch (Offline-LD), a novel approach for JSSP that addresses these limitations. Offline-LD adapts two CQL-based Q-learning methods (mQRDQN and discrete mSAC) for maskable action spaces, introduces a new entropy bonus modification for discrete SAC, and exploits reward normalization through preprocessing. Our experiments show that Offline-LD outperforms online RL on both generated and benchmark instances. By introducing noise into the dataset, we achieve similar or better results than those obtained from the expert dataset, indicating that a more diverse training set is preferable because it contains counterfactual information.

Related papers

• Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration [54.8229698058649]
  We study how unlabeled prior trajectory data can be leveraged to learn efficient exploration strategies. Our method SUPE (Skills from Unlabeled Prior data for Exploration) demonstrates that a careful combination of these ideas compounds their benefits. We empirically show that SUPE reliably outperforms prior strategies, successfully solving a suite of long-horizon, sparse-reward tasks.
  arXiv  Detail & Related papers  (2024-10-23T17:58:45Z)
• Equivariant Offline Reinforcement Learning [7.822389399560674]
  We investigate the use of $SO(2)$-equivariant neural networks for offline RL with a limited number of demonstrations. Our experimental results show that equivariant versions of Conservative Q-Learning (CQL) and Implicit Q-Learning (IQL) outperform their non-equivariant counterparts.
  arXiv  Detail & Related papers  (2024-06-20T03:02:49Z)
• ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories [27.5648276335047]
  Training autonomous agents with sparse rewards is a long-standing problem in online reinforcement learning (RL) We propose a novel approach that leverages offline data to learn a generative diffusion model, coined as Adaptive Trajectory diffuser (ATraDiff) ATraDiff consistently achieves state-of-the-art performance across a variety of environments, with particularly pronounced improvements in complicated settings.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale [27.02990488317357]
  Given an offline demonstration dataset from an imperfect expert, what is the best way to leverage it to bootstrap online learning performance in MDPs? We first propose an Informed Posterior Sampling-based RL (iPSRL) algorithm that uses the offline dataset, and information about the expert's behavioral policy used to generate the offline dataset. Since this algorithm is computationally impractical, we then propose the iRLSVI algorithm that can be seen as a combination of the RLSVI algorithm for online RL, and imitation learning.
  arXiv  Detail & Related papers  (2023-03-20T18:16:25Z)
• Efficient Online Reinforcement Learning with Offline Data [78.92501185886569]
  We show that we can simply apply existing off-policy methods to leverage offline data when learning online. We extensively ablate these design choices, demonstrating the key factors that most affect performance. We see that correct application of these simple recommendations can provide a $mathbf2.5times$ improvement over existing approaches.
  arXiv  Detail & Related papers  (2023-02-06T17:30:22Z)
• Boosting Offline Reinforcement Learning via Data Rebalancing [104.3767045977716]
  offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets. We propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged. We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time.
  arXiv  Detail & Related papers  (2022-10-17T16:34:01Z)
• Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations [5.760034336327491]
  We study the problem of offline Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. We introduce an additional discriminator to distinguish expert and non-expert data. Our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.
  arXiv  Detail & Related papers  (2022-07-20T17:29:04Z)
• RvS: What is Essential for Offline RL via Supervised Learning? [77.91045677562802]
  Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. In every environment suite we consider simply maximizing likelihood with two-layer feedforward is competitive. They also probe the limits of existing RvS methods, which are comparatively weak on random data.
  arXiv  Detail & Related papers  (2021-12-20T18:55:16Z)
• Offline Meta-Reinforcement Learning with Online Self-Supervision [66.42016534065276]
  We propose a hybrid offline meta-RL algorithm, which uses offline data with rewards to meta-train an adaptive policy. Our method uses the offline data to learn the distribution of reward functions, which is then sampled to self-supervise reward labels for the additional online data. We find that using additional data and self-generated rewards significantly improves an agent's ability to generalize.
  arXiv  Detail & Related papers  (2021-07-08T17:01:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.