Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers

• URL: http://arxiv.org/abs/2212.11498v2
• Date: Fri, 7 Jul 2023 17:20:25 GMT
• Title: Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers
• Authors: Aleksandar Krnjaic, Raul D. Steleac, Jonathan D. Thomas, Georgios Papoudakis, Lukas Sch\"afer, Andrew Wing Keung To, Kuan-Ho Lao, Murat Cubuktepe, Matthew Haley, Peter B\"orsting, Stefano V. Albrecht
• Abstract summary: We envision a warehouse in which dozens of mobile robots and human pickers work together to collect and deliver items within the warehouse. We develop hierarchical MARL algorithms in which a manager assigns goals to worker agents, and the policies of the manager and workers are co-trained toward maximising a global objective. Our hierarchical algorithms achieve significant gains in sample efficiency and overall pick rates over baseline MARL algorithms in diverse warehouse configurations.
• Score: 51.71901155656791
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We envision a warehouse in which dozens of mobile robots and human pickers work together to collect and deliver items within the warehouse. The fundamental problem we tackle, called the order-picking problem, is how these worker agents must coordinate their movement and actions in the warehouse to maximise performance (e.g. order throughput). Established industry methods using heuristic approaches require large engineering efforts to optimise for innately variable warehouse configurations. In contrast, multi-agent reinforcement learning (MARL) can be flexibly applied to diverse warehouse configurations (e.g. size, layout, number/types of workers, item replenishment frequency), as the agents learn through experience how to optimally cooperate with one another. We develop hierarchical MARL algorithms in which a manager assigns goals to worker agents, and the policies of the manager and workers are co-trained toward maximising a global objective (e.g. pick rate). Our hierarchical algorithms achieve significant gains in sample efficiency and overall pick rates over baseline MARL algorithms in diverse warehouse configurations, and substantially outperform two established industry heuristics for order-picking systems.

Related papers

• Learning Manipulation Tasks in Dynamic and Shared 3D Spaces [2.4892784882130132]
  Learning automated pick-and-place operations can be efficiently done by introducing collaborative autonomous systems. In this paper, we propose a deep reinforcement learning strategy to learn the place task of multi-categorical items.
  arXiv  Detail & Related papers  (2024-04-26T19:40:19Z)
• Dynamic AGV Task Allocation in Intelligent Warehouses [1.519321208145928]
  The booming AGV industry is witnessing widespread adoption due to its efficiency, reliability, and cost-effectiveness. This paper focuses on enhancing the picker-to-parts system, prevalent in small to medium-sized warehouses, through the strategic use of AGVs. We propose a novel approach Neural Dynamic Programming approach for coordinating a mixed team of human AGV workers.
  arXiv  Detail & Related papers  (2023-12-26T12:28:25Z)
• MASP: Scalable GNN-based Planning for Multi-Agent Navigation [17.788592987873905]
  We propose a goal-conditioned hierarchical planner for navigation tasks with a substantial number of agents. We also leverage graph neural networks (GNN) to model the interaction between agents and goals, improving goal achievement. The results demonstrate that MASP outperforms classical planning-based competitors and RL baselines.
  arXiv  Detail & Related papers  (2023-12-05T06:05:04Z)
• Multi-Robot Coordination and Layout Design for Automated Warehousing [55.150593161240444]
  We show that, even with state-of-the-art MAPF algorithms, commonly used human-designed layouts can lead to congestion for warehouses with large numbers of robots. We extend existing automatic scenario generation methods to optimize warehouse layouts. Results show that our optimized warehouse layouts (1) reduce traffic congestion and thus improve throughput, (2) improve the scalability of the automated warehouses by doubling the number of robots in some cases, and (3) are capable of generating layouts with user-specified diversity measures.
  arXiv  Detail & Related papers  (2023-05-10T20:00:06Z)
• Multi-Agent Reinforcement Learning for Microprocessor Design Space Exploration [71.95914457415624]
  Microprocessor architects are increasingly resorting to domain-specific customization in the quest for high-performance and energy-efficiency. We propose an alternative formulation that leverages Multi-Agent RL (MARL) to tackle this problem. Our evaluation shows that the MARL formulation consistently outperforms single-agent RL baselines.
  arXiv  Detail & Related papers  (2022-11-29T17:10:24Z)
• Multi-agent Deep Covering Skill Discovery [50.812414209206054]
  We propose Multi-agent Deep Covering Option Discovery, which constructs the multi-agent options through minimizing the expected cover time of the multiple agents' joint state space. Also, we propose a novel framework to adopt the multi-agent options in the MARL process. We show that the proposed algorithm can effectively capture the agent interactions with the attention mechanism, successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options.
  arXiv  Detail & Related papers  (2022-10-07T00:40:59Z)
• LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning [122.47938710284784]
  We propose a novel framework for learning dynamic subtask assignment (LDSA) in cooperative MARL. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy. We show that LDSA learns reasonable and effective subtask assignment for better collaboration.
  arXiv  Detail & Related papers  (2022-05-05T10:46:16Z)
• Containerized Distributed Value-Based Multi-Agent Reinforcement Learning [18.79371121484969]
  We propose a containerized multi-agent reinforcement learning framework. To own knowledge, our method is the first to solve the challenging Google Research Football full game $5_v_5$. On the StarCraft II micromanagement benchmark, our method gets $4$-$18times$ better results compared to state-of-the-art non-distributed MARL algorithms.
  arXiv  Detail & Related papers  (2021-10-15T15:54:06Z)
• Dif-MAML: Decentralized Multi-Agent Meta-Learning [54.39661018886268]
  We propose a cooperative multi-agent meta-learning algorithm, referred to as MAML or Dif-MAML. We show that the proposed strategy allows a collection of agents to attain agreement at a linear rate and to converge to a stationary point of the aggregate MAML. Simulation results illustrate the theoretical findings and the superior performance relative to the traditional non-cooperative setting.
  arXiv  Detail & Related papers  (2020-10-06T16:51:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.