A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health

• URL: http://arxiv.org/abs/2402.14807v4
• Date: Fri, 25 Oct 2024 13:34:14 GMT
• Title: A Decision-Language Model (DLM) for Dynamic Restless Multi-Armed Bandit Tasks in Public Health
• Authors: Nikhil Behari, Edwin Zhang, Yunfan Zhao, Aparna Taneja, Dheeraj Nagaraj, Milind Tambe,
• Abstract summary: Large Language Models (LLMs) have emerged as adept automated planners across domains of robotic control and navigation. We propose a Decision Language Model (DLM) for RMABs, enabling dynamic fine-tuning of RMAB policies using human-language commands.
• Score: 29.894488663882328
• License:
• Abstract: Restless multi-armed bandits (RMAB) have demonstrated success in optimizing resource allocation for large beneficiary populations in public health settings. Unfortunately, RMAB models lack flexibility to adapt to evolving public health policy priorities. Concurrently, Large Language Models (LLMs) have emerged as adept automated planners across domains of robotic control and navigation. In this paper, we propose a Decision Language Model (DLM) for RMABs, enabling dynamic fine-tuning of RMAB policies in public health settings using human-language commands. We propose using LLMs as automated planners to (1) interpret human policy preference prompts, (2) propose reward functions as code for a multi-agent RMAB environment, and (3) iterate on the generated reward functions using feedback from grounded RMAB simulations. We illustrate the application of DLM in collaboration with ARMMAN, an India-based non-profit promoting preventative care for pregnant mothers, that currently relies on RMAB policies to optimally allocate health worker calls to low-resource populations. We conduct a technology demonstration in simulation using the Gemini Pro model, showing DLM can dynamically shape policy outcomes using only human prompts as input.

Related papers

• Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards [41.140822259857266]
  We present a principled method termed Social Choice Language Model for dealing with tradeoffs for reward functions based on human preferences. Our experiments demonstrate that our model reliably selects more effective, aligned, and balanced reward functions.
  arXiv  Detail & Related papers  (2024-08-22T03:54:08Z)
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts [23.27203570485055]
  Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models with human preferences. We propose a two-stage approach to train a reward model (RM) with multi-dimensional absolute-rating data. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow on top of the ArmoRM.
  arXiv  Detail & Related papers  (2024-06-18T17:58:28Z)
• Fine-Tuning Language Models with Reward Learning on Policy [68.70065254564642]
  Reinforcement learning from human feedback (RLHF) has emerged as an effective approach to aligning large language models (LLMs) to human preferences. Despite its popularity, (fixed) reward models may suffer from inaccurate off-distribution. We propose reward learning on policy (RLP), an unsupervised framework that refines a reward model using policy samples to keep it on-distribution.
  arXiv  Detail & Related papers  (2024-03-28T10:02:10Z)
• WARM: On the Benefits of Weight Averaged Reward Models [63.08179139233774]
  We propose Weight Averaged Reward Models (WARM) to mitigate reward hacking. Experiments on summarization tasks, using best-of-N and RL methods, shows that WARM improves the overall quality and alignment of LLM predictions.
  arXiv  Detail & Related papers  (2024-01-22T18:27:08Z)
• Personalized Reinforcement Learning with a Budget of Policies [9.846353643883443]
  Personalization in machine learning (ML) tailors models' decisions to the individual characteristics of users. We propose a novel framework termed represented Markov Decision Processes (r-MDPs) that is designed to balance the need for personalization with the regulatory constraints. In an r-MDP, we cater to a diverse user population, each with unique preferences, through interaction with a small set of representative policies. We develop two deep reinforcement learning algorithms that efficiently solve r-MDPs.
  arXiv  Detail & Related papers  (2024-01-12T11:27:55Z)
• Nash Learning from Human Feedback [86.09617990412941]
  We introduce an alternative pipeline for the fine-tuning of large language models using pairwise human feedback. We term this approach Nash learning from human feedback (NLHF) We present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent.
  arXiv  Detail & Related papers  (2023-12-01T19:26:23Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)
• Guide Your Agent with Adaptive Multimodal Rewards [107.08768813632032]
  This work presents Adaptive Return-conditioned Policy (ARP), an efficient framework to enhance the agent's generalization ability. Our key idea is to calculate a similarity between visual observations and natural language instructions in the pre-trained multimodal embedding space. Because the multimodal rewards provide adaptive signals at each timestep, our ARP effectively mitigates the goal misgeneralization.
  arXiv  Detail & Related papers  (2023-09-19T17:39:20Z)
• Equitable Restless Multi-Armed Bandits: A General Framework Inspired By Digital Health [23.762981395335217]
  Restless multi-armed bandits (RMABs) are a popular framework for algorithmic decision making in sequential settings with limited resources. RMABs are increasingly being used for sensitive decisions such as in public health, treatment scheduling, anti-poaching, and -- the motivation for this work -- digital health. We study equitable objectives for RMABs for the first time. We consider two equity-aligned objectives from the fairness literature, minimax reward and max Nash welfare. We develop efficient algorithms for solving each -- a water filling algorithm for the former, and a greedy algorithm with theoretically motivated nuance to balance disparate group sizes
  arXiv  Detail & Related papers  (2023-08-17T13:00:27Z)
• Limited Resource Allocation in a Non-Markovian World: The Case of Maternal and Child Healthcare [27.812174610119452]
  We consider the problem of scheduling interventions in low resource settings to increase adherence and/or engagement. Past works have successfully developed several classes of Restless Multi-armed Bandit (RMAB) based solutions for this problem. We demonstrate significant deviations from the Markov assumption on real-world data on a maternal health awareness program from our partner NGO, ARMMAN. To tackle the generalised non-Markovian RMAB setting we (i) model each participant's trajectory as a time-series, (ii) leverage the power of time-series forecasting models to predict future states, and (iii) propose the Time
  arXiv  Detail & Related papers  (2023-05-22T02:26:29Z)
• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization [73.74371798168642]
  We introduce an open-source modular library, RL4LMs, for optimizing language generators with reinforcement learning. Next, we present the GRUE benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO, that learns to effectively reduce the action space in language generation.
  arXiv  Detail & Related papers  (2022-10-03T21:38:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.