Operationalizing Counterfactual Metrics: Incentives, Ranking, and Information Asymmetry

• URL: http://arxiv.org/abs/2305.14595v2
• Date: Wed, 29 Nov 2023 20:56:41 GMT
• Title: Operationalizing Counterfactual Metrics: Incentives, Ranking, and Information Asymmetry
• Authors: Serena Wang, Stephen Bates, P. M. Aronow, Michael I. Jordan
• Abstract summary: We analyze the incentive misalignments that arise from such average treated outcome metrics. We show how counterfactual metrics can be modified to behave reasonably in patient-facing ranking systems.
• Score: 62.53919624802853
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: From the social sciences to machine learning, it has been well documented that metrics to be optimized are not always aligned with social welfare. In healthcare, Dranove et al. (2003) showed that publishing surgery mortality metrics actually harmed the welfare of sicker patients by increasing provider selection behavior. We analyze the incentive misalignments that arise from such average treated outcome metrics, and show that the incentives driving treatment decisions would align with maximizing total patient welfare if the metrics (i) accounted for counterfactual untreated outcomes and (ii) considered total welfare instead of averaging over treated patients. Operationalizing this, we show how counterfactual metrics can be modified to behave reasonably in patient-facing ranking systems. Extending to realistic settings when providers observe more about patients than the regulatory agencies do, we bound the decay in performance by the degree of information asymmetry between principal and agent. In doing so, our model connects principal-agent information asymmetry with unobserved heterogeneity in causal inference.

Related papers

• PRISM: Leveraging Prototype Patient Representations with Feature-Missing-Aware Calibration for EHR Data Sparsity Mitigation [7.075420686441701]
  PRISM is a framework that indirectly imputes data by leveraging prototype representations of similar patients. PRISM also includes a feature confidence module, which evaluates the reliability of each feature considering missing statuses.
  arXiv  Detail & Related papers  (2023-09-08T07:01:38Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Policy Learning with Asymmetric Counterfactual Utilities [0.6138671548064356]
  We consider optimal policy learning with asymmetric counterfactual utility functions. We derive minimax decision rules by minimizing the maximum expected utility loss. We show that one can learn minimax loss decision rules from observed data by solving intermediate classification problems.
  arXiv  Detail & Related papers  (2022-06-21T15:44:49Z)
• Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics [64.81682222169113]
  How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice.
  arXiv  Detail & Related papers  (2022-04-21T15:52:14Z)
• Estimating and Improving Fairness with Adversarial Learning [65.99330614802388]
  We propose an adversarial multi-task training strategy to simultaneously mitigate and detect bias in the deep learning-based medical image analysis system. Specifically, we propose to add a discrimination module against bias and a critical module that predicts unfairness within the base classification model. We evaluate our framework on a large-scale public-available skin lesion dataset.
  arXiv  Detail & Related papers  (2021-03-07T03:10:32Z)
• Learning $\mathbf{\mathit{Matching}}$ Representations for Individualized Organ Transplantation Allocation [98.43063331640538]
  We formulate the problem of learning data-driven rules for organ matching using observational data for organ allocations and transplant outcomes. We propose a model based on representation learning to predict donor-recipient compatibility. Our model outperforms state-of-art allocation methods and policies executed by human experts.
  arXiv  Detail & Related papers  (2021-01-28T01:33:21Z)
• MCU-Net: A framework towards uncertainty representations for decision support system patient referrals in healthcare contexts [0.0]
  We present a framework of uncertainty representation evaluated for medical image segmentation, using MCU-Net. The framework augments this by adding a human-in-the-loop aspect based on an uncertainty threshold for automated referral of uncertain cases to a medical professional.
  arXiv  Detail & Related papers  (2020-07-08T09:54:56Z)
• Using Deep Learning and Explainable Artificial Intelligence in Patients' Choices of Hospital Levels [10.985001960872264]
  This study used nationwide insurance data, accumulated possible features discussed in existing literature, and used a deep neural network to predict the patients choices of hospital levels. The results showed that the model was able to predict with high area under the receiver operating characteristics curve (AUC) (0.90), accuracy (0.90), sensitivity (0.94), and specificity (0.97) with highly imbalanced label.
  arXiv  Detail & Related papers  (2020-06-24T02:15:15Z)
• Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics [64.88815792555451]
  We show that current methods for judging metrics are highly sensitive to the translations used for assessment. We develop a method for thresholding performance improvement under an automatic metric against human judgements.
  arXiv  Detail & Related papers  (2020-06-11T09:12:53Z)
• Robust Policies For Proactive ICU Transfers [3.9286045166400685]
  Patients whose transfer to the Intensive Care Unit (ICU) is unplanned are prone to higher mortality rates than those who were admitted directly to the ICU. Recent advances in machine learning to predict patient deterioration have introduced the possibility of emphproactive transfer from the ward to the ICU. We study the problem of finding emphrobust patient transfer policies which account for uncertainty in statistical estimates due to data limitations when optimizing to improve overall patient care.
  arXiv  Detail & Related papers  (2020-02-14T21:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.