Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessing

• URL: http://arxiv.org/abs/2501.06366v2
• Date: Tue, 14 Jan 2025 04:42:08 GMT
• Title: Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessing
• Authors: Jitao Wang, Chengchun Shi, John D. Piette, Joshua R. Loftus, Donglin Zeng, Zhenke Wu,
• Abstract summary: Counterfactual fairness (CF) offers a promising statistical tool grounded in causal inference to formulate and study fairness. We theoretically characterize the optimal CF policy and prove its stationarity, which greatly simplifies the search for optimal CF policies. We prove and then validate our policy learning approach in controlling unfairness and attaining optimal value through simulations.
• Score: 13.34215548232296
• License:
• Abstract: When applied in healthcare, reinforcement learning (RL) seeks to dynamically match the right interventions to subjects to maximize population benefit. However, the learned policy may disproportionately allocate efficacious actions to one subpopulation, creating or exacerbating disparities in other socioeconomically-disadvantaged subgroups. These biases tend to occur in multi-stage decision making and can be self-perpetuating, which if unaccounted for could cause serious unintended consequences that limit access to care or treatment benefit. Counterfactual fairness (CF) offers a promising statistical tool grounded in causal inference to formulate and study fairness. In this paper, we propose a general framework for fair sequential decision making. We theoretically characterize the optimal CF policy and prove its stationarity, which greatly simplifies the search for optimal CF policies by leveraging existing RL algorithms. The theory also motivates a sequential data preprocessing algorithm to achieve CF decision making under an additive noise assumption. We prove and then validate our policy learning approach in controlling unfairness and attaining optimal value through simulations. Analysis of a digital health dataset designed to reduce opioid misuse shows that our proposal greatly enhances fair access to counseling.

Related papers

• Counterfactual Fairness by Combining Factual and Counterfactual Predictions [18.950415688199993]
  In high-stake domains such as healthcare and hiring, the role of machine learning (ML) in decision-making raises significant fairness concerns. This work focuses on Counterfactual Fairness (CF), which posits that an ML model's outcome on any individual should remain unchanged if they had belonged to a different demographic group. We provide a theoretical study on the inherent trade-off between CF and predictive performance in a model-agnostic manner.
  arXiv  Detail & Related papers  (2024-09-03T15:21:10Z)
• Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Preference-Based Multi-Agent Reinforcement Learning (PbMARL) We identify the Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for PbMARL.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data [102.16105233826917]
  Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning.
  arXiv  Detail & Related papers  (2024-04-22T17:20:18Z)
• Policy Learning with Distributional Welfare [1.0742675209112622]
  Most literature on treatment choice has considered utilitarian welfare based on the conditional average treatment effect (ATE) This paper proposes an optimal policy that allocates the treatment based on the conditional quantile of individual treatment effects (QoTE)
  arXiv  Detail & Related papers  (2023-11-27T14:51:30Z)
• Optimal and Fair Encouragement Policy Evaluation and Learning [11.712023983596914]
  We study causal identification and robust estimation of optimal treatment rules, including under potential violations of positivity. We develop a two-stage algorithm for solving over parametrized policy classes under general constraints to obtain variance-sensitive regret bounds. We illustrate the methods in three case studies based on data from reminders of SNAP benefits, randomized encouragement to enroll in insurance, and from pretrial supervised release with electronic monitoring.
  arXiv  Detail & Related papers  (2023-09-12T20:45:30Z)
• Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequality [94.89246810243053]
  This paper studies offline policy learning, which aims at utilizing observations collected a priori to learn an optimal individualized decision rule. Existing policy learning methods rely on a uniform overlap assumption, i.e., the propensities of exploring all actions for all individual characteristics must be lower bounded. We propose Pessimistic Policy Learning (PPL), a new algorithm that optimize lower confidence bounds (LCBs) instead of point estimates.
  arXiv  Detail & Related papers  (2022-12-19T22:43:08Z)
• Reinforcement Learning with Stepwise Fairness Constraints [50.538878453547966]
  We introduce the study of reinforcement learning with stepwise fairness constraints. We provide learning algorithms with strong theoretical guarantees in regard to policy optimality and fairness violation.
  arXiv  Detail & Related papers  (2022-11-08T04:06:23Z)
• Federated Offline Reinforcement Learning [55.326673977320574]
  We propose a multi-site Markov decision process model that allows for both homogeneous and heterogeneous effects across sites. We design the first federated policy optimization algorithm for offline RL with sample complexity. We give a theoretical guarantee for the proposed algorithm, where the suboptimality for the learned policies is comparable to the rate as if data is not distributed.
  arXiv  Detail & Related papers  (2022-06-11T18:03:26Z)
• Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency [0.0]
  This paper focuses on the fairness concepts of PPV parity, false omission rate (FOR) parity, and sufficiency. We show that group-specific threshold rules are optimal for PPV parity and FOR parity. We also provide a solution for the optimal decision rules satisfying the fairness constraint sufficiency.
  arXiv  Detail & Related papers  (2022-06-05T18:47:34Z)
• False Correlation Reduction for Offline Reinforcement Learning [115.11954432080749]
  We propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL)
  arXiv  Detail & Related papers  (2021-10-24T15:34:03Z)
• Inherent Trade-offs in the Fair Allocation of Treatments [2.6143568807090696]
  Explicit and implicit bias clouds human judgement, leading to discriminatory treatment of minority groups. We propose a causal framework that learns optimal intervention policies from data subject to fairness constraints.
  arXiv  Detail & Related papers  (2020-10-30T17:55:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.