Auditing the Fairness of COVID-19 Forecast Hub Case Prediction Models

• URL: http://arxiv.org/abs/2405.14891v1
• Date: Fri, 17 May 2024 21:07:19 GMT
• Title: Auditing the Fairness of COVID-19 Forecast Hub Case Prediction Models
• Authors: Saad Mohammad Abrar, Naman Awasthi, Daniel Smolyak, Vanessa Frias-Martinez,
• Abstract summary: The COVID-19 Forecast Hub is used by the Centers for Disease Control and Prevention (CDC) for their official COVID-19 communications. By focusing exclusively on prediction accuracy, the Forecast Hub fails to evaluate whether the proposed models have similar performance across social determinants. We show statistically significant predictive performance across social determinants, with minority racial and ethnic groups as well as less urbanized areas often associated with higher prediction errors.
• Score: 0.24999074238880484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The COVID-19 Forecast Hub, a repository of COVID-19 forecasts from over 50 independent research groups, is used by the Centers for Disease Control and Prevention (CDC) for their official COVID-19 communications. As such, the Forecast Hub is a critical centralized resource to promote transparent decision making. Nevertheless, by focusing exclusively on prediction accuracy, the Forecast Hub fails to evaluate whether the proposed models have similar performance across social determinants that have been known to play a role in the COVID-19 pandemic including race, ethnicity and urbanization level. In this paper, we carry out a comprehensive fairness analysis of the Forecast Hub model predictions and we show statistically significant diverse predictive performance across social determinants, with minority racial and ethnic groups as well as less urbanized areas often associated with higher prediction errors. We hope this work will encourage COVID-19 modelers and the CDC to report fairness metrics together with accuracy, and to reflect on the potential harms of the models on specific social groups and contexts.

Related papers

• A Multilateral Attention-enhanced Deep Neural Network for Disease Outbreak Forecasting: A Case Study on COVID-19 [0.6874745415692134]
  We propose a novel approach to address the challenges of infectious disease forecasting. We introduce a Multilateral Attention-enhanced GRU model that leverages information from multiple sources. By incorporating attention mechanisms within a GRU framework, our model can effectively capture complex relationships and temporal dependencies in the data.
  arXiv  Detail & Related papers  (2024-08-26T06:31:53Z)
• Performative Prediction on Games and Mechanism Design [69.7933059664256]
  We study a collective risk dilemma where agents decide whether to trust predictions based on past accuracy. As predictions shape collective outcomes, social welfare arises naturally as a metric of concern. We show how to achieve better trade-offs and use them for mechanism design.
  arXiv  Detail & Related papers  (2024-08-09T16:03:44Z)
• DemOpts: Fairness corrections in COVID-19 case prediction models [0.24999074238880484]
  We show that state of the art deep learning models output mean prediction errors that are significantly different across racial and ethnic groups. We propose a novel de-biasing method, DemOpts, to increase the fairness of deep learning based forecasting models trained on potentially biased datasets.
  arXiv  Detail & Related papers  (2024-05-15T16:22:46Z)
• Using Pre-training and Interaction Modeling for ancestry-specific disease prediction in UK Biobank [69.90493129893112]
  Recent genome-wide association studies (GWAS) have uncovered the genetic basis of complex traits, but show an under-representation of non-European descent individuals. Here, we assess whether we can improve disease prediction across diverse ancestries using multiomic data.
  arXiv  Detail & Related papers  (2024-04-26T16:39:50Z)
• Learning for Counterfactual Fairness from Observational Data [62.43249746968616]
  Fairness-aware machine learning aims to eliminate biases of learning models against certain subgroups described by certain protected (sensitive) attributes such as race, gender, and age. A prerequisite for existing methods to achieve counterfactual fairness is the prior human knowledge of the causal model for the data. In this work, we address the problem of counterfactually fair prediction from observational data without given causal models by proposing a novel framework CLAIRE.
  arXiv  Detail & Related papers  (2023-07-17T04:08:29Z)
• Evaluating Probabilistic Classifiers: The Triptych [62.997667081978825]
  We propose and study a triptych of diagnostic graphics that focus on distinct and complementary aspects of forecast performance. The reliability diagram addresses calibration, the receiver operating characteristic (ROC) curve diagnoses discrimination ability, and the Murphy diagram visualizes overall predictive performance and value.
  arXiv  Detail & Related papers  (2023-01-25T19:35:23Z)
• A spatiotemporal machine learning approach to forecasting COVID-19 incidence at the county level in the United States [2.9822184411723645]
  We present COVID-LSTM, a data-driven model based on a Long Short-term memory architecture for forecasting COVID-19 incidence at the county-level in the US. We use the weekly number of new cases as temporal input, and hand-engineered spatial features from Facebook to capture the spread of the disease in time and space. Over the 4-week forecast horizon, our model is on average 50 cases per county more accurate than the COVIDhub-ensemble.
  arXiv  Detail & Related papers  (2021-09-24T17:40:08Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• STELAR: Spatio-temporal Tensor Factorization with Latent Epidemiological Regularization [76.57716281104938]
  We develop a tensor method to predict the evolution of epidemic trends for many regions simultaneously. STELAR enables long-term prediction by incorporating latent temporal regularization through a system of discrete-time difference equations. We conduct experiments using both county- and state-level COVID-19 data and show that our model can identify interesting latent patterns of the epidemic.
  arXiv  Detail & Related papers  (2020-12-08T21:21:47Z)
• Inter-Series Attention Model for COVID-19 Forecasting [23.58411106491221]
  We develop a new neural forecasting model, called Attention Crossing Time Series (textbfACTS), that makes forecasts via comparing patterns across time series obtained from multiple regions. Among 13 out of 18 testings including forecasting newly confirmed cases, hospitalizations and deaths, textbfACTS outperforms all the leading COVID-19 forecasters highlighted by CDC.
  arXiv  Detail & Related papers  (2020-10-25T00:11:49Z)
• Forecasting COVID-19 daily cases using phone call data [0.0]
  We propose a simple Multiple Linear Regression model optimised to use call data to forecast the number of daily confirmed cases. Our proposed approach outperforms ARIMA, ETS and a regression model without call data, evaluated by three point forecast error metrics, one prediction interval and two probabilistic forecast accuracy measures.
  arXiv  Detail & Related papers  (2020-10-05T18:07:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.