All Models Are Wrong, But Can They Be Useful? Lessons from COVID-19 Agent-Based Models: A Systematic Review

• URL: http://arxiv.org/abs/2509.13346v1
• Date: Fri, 12 Sep 2025 21:16:50 GMT
• Title: All Models Are Wrong, But Can They Be Useful? Lessons from COVID-19 Agent-Based Models: A Systematic Review
• Abstract summary: The COVID-19 pandemic prompted a surge in computational models to simulate disease dynamics and guide interventions.<n>We systematically reviewed 536 COVID-19 ABM studies published from January 2020 to December 2023.<n>Overall, COVID-19 ABMs advanced quickly, but lacked transparency, accessibility, and participatory engagement.<n>Stronger standards are needed for ABMs to serve as reliable decision-support tools in future public health crises.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The COVID-19 pandemic prompted a surge in computational models to simulate disease dynamics and guide interventions. Agent-based models (ABMs) are well-suited to capture population and environmental heterogeneity, but their rapid deployment raised questions about utility for health policy. We systematically reviewed 536 COVID-19 ABM studies published from January 2020 to December 2023, retrieved from Web of Science, PubMed, and Wiley on January 30, 2024. Studies were included if they used ABMs to simulate COVID-19 transmission, where reviews were excluded. Studies were assessed against nine criteria of model usefulness, including transparency and re-use, interdisciplinary collaboration and stakeholder engagement, and evaluation practices. Publications peaked in late 2021 and were concentrated in a few countries. Most models explored behavioral or policy interventions (n = 294, 54.85%) rather than real-time forecasting (n = 9, 1.68%). While most described model assumptions (n = 491, 91.60%), fewer disclosed limitations (n = 349, 65.11%), shared code (n = 219, 40.86%), or built on existing models (n = 195, 36.38%). Standardized reporting protocols (n = 36, 6.72%) and stakeholder engagement were rare (13.62%, n = 73). Only 2.24% (n = 12) described a comprehensive validation framework, though uncertainty was often quantified (n = 407, 75.93%). Limitations of this review include underrepresentation of non-English studies, subjective data extraction, variability in study quality, and limited generalizability. Overall, COVID-19 ABMs advanced quickly, but lacked transparency, accessibility, and participatory engagement. Stronger standards are needed for ABMs to serve as reliable decision-support tools in future public health crises.

Related papers

• BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases [59.19934491064188]
  Artificial intelligence (AI) has achieved remarkable success in medical imaging.<n>These models often perform inconsistently across real-world clinical settings.<n>We develop a benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models.
  arXiv  Detail & Related papers  (2026-06-23T17:58:59Z)
• Predicting Neuromodulation Outcome for Parkinson's Disease with Generative Virtual Brain Model [95.41752463487008]
  Parkinson's disease affects over ten million people worldwide.<n>Inter-individual variability limits empirical treatment selection, increasing non-negligible surgical risk and cost.<n>We bridge this gap with a pretraining-finetuning framework to predict outcomes directly from resting-state fMRI.
  arXiv  Detail & Related papers  (2026-03-31T02:36:10Z)
• From Data to Behavior: Predicting Unintended Model Behaviors Before Training [78.37660873165284]
  We introduce Data2Behavior, a new task for predicting unintended model behaviors prior to training.<n>We also propose Manipulating Data Features (MDF), a lightweight approach that summarizes candidate data through their mean representations.<n>Experiments on Qwen3-14B, Qwen2.5-32B-Instruct, and Gemma-3-12b-it confirm that MDF can anticipate unintended behaviors and provide insight into pre-training vulnerabilities.
  arXiv  Detail & Related papers  (2026-02-04T16:37:17Z)
• CRTRE: Causal Rule Generation with Target Trial Emulation Framework [47.2836994469923]
  We introduce a novel method called causal rule generation with target trial emulation framework (CRTRE) CRTRE applies randomize trial design principles to estimate the causal effect of association rules. We then incorporate such association rules for the downstream applications such as prediction of disease onsets.
  arXiv  Detail & Related papers  (2024-11-10T02:40:06Z)
• Machine Learning Approaches for Mental Illness Detection on Social Media: A Systematic Review of Biases and Methodological Challenges [0.037693031068634524]
  This systematic review examines machine learning models for detecting mental illness using social media data.<n>It highlights biases and methodological challenges encountered throughout the machine learning lifecycle.<n>By overcoming these challenges, future research can develop more robust and generalizable ML models for depression detection on social media.
  arXiv  Detail & Related papers  (2024-10-21T17:05:50Z)
• Using Pre-training and Interaction Modeling for ancestry-specific disease prediction in UK Biobank [69.90493129893112]
  Recent genome-wide association studies (GWAS) have uncovered the genetic basis of complex traits, but show an under-representation of non-European descent individuals. Here, we assess whether we can improve disease prediction across diverse ancestries using multiomic data.
  arXiv  Detail & Related papers  (2024-04-26T16:39:50Z)
• Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imaging [47.99192239793597]
  We evaluated the effect of privacy-preserving training of AI models regarding accuracy and fairness compared to non-private training. Our study shows that -- under the challenging realistic circumstances of a real-life clinical dataset -- the privacy-preserving training of diagnostic deep learning models is possible with excellent diagnostic accuracy and fairness.
  arXiv  Detail & Related papers  (2023-02-03T09:49:13Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)
• Strict baselines for Covid-19 forecasting and ML perspective for USA and Russia [105.54048699217668]
  Covid-19 allows researchers to gather datasets accumulated over 2 years and to use them in predictive analysis. We present the results of a consistent comparative study of different types of methods for predicting the dynamics of the spread of Covid-19 based on regional data for two countries: the United States and Russia.
  arXiv  Detail & Related papers  (2022-07-15T18:21:36Z)
• Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence [79.038671794961]
  We launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution. Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK.
  arXiv  Detail & Related papers  (2021-11-18T00:43:41Z)
• From SIR to SEAIRD: a novel data-driven modeling approach based on the Grey-box System Theory to predict the dynamics of COVID-19 [6.131772929312605]
  Common compartmental modeling for COVID-19 is based on a priori knowledge and numerous assumptions. Our study aimed at providing a framework for data-driven approaches, by leveraging the strengths of the grey-box system theory or grey-box identification. The incidence rate of COVID-19 was as low as 3 infected cases per 100,000 exposed persons in Brazil and France.
  arXiv  Detail & Related papers  (2021-05-29T21:25:09Z)
• Combining Graph Neural Networks and Spatio-temporal Disease Models to Predict COVID-19 Cases in Germany [0.0]
  Several experts have called for the necessity to account for human mobility to explain the spread of COVID-19. Most statistical or epidemiological models cannot directly incorporate unstructured data sources, including data that may encode human mobility. We propose a trade-off between both research directions and present a novel learning approach that combines the advantages of statistical regression and machine learning models.
  arXiv  Detail & Related papers  (2021-01-03T16:39:00Z)
• Short-term forecasting COVID-19 cumulative confirmed cases: Perspectives for Brazil [3.0711362702464675]
  The new Coronavirus (COVID-19) is an emerging disease responsible for infecting millions of people since the first notification until nowadays. In this paper, autoregressive integrated moving average (ARIMA), cubist (CUBIST), random forest (RF), ridge regression (RIDGE), and stacking-ensemble learning are evaluated. The developed models can generate accurate forecasting, achieving errors in a range of 0.87% - 3.51%, 1.02% - 5.63%, and 0.95% - 6.90% in one, three, and six-days-ahead, respectively.
  arXiv  Detail & Related papers  (2020-07-21T17:58:58Z)
• Forecasting Brazilian and American COVID-19 cases based on artificial intelligence coupled with climatic exogenous variables [3.0711362702464675]
  coronavirus disease (COVID-19) is a public health problem once according to the World Health Organization up to June 10th, 2020. In the current scenario, the Brazil and the United States of America present a high daily incidence of new cases and deaths. It is important to forecast the number of new cases in a time window of one week, once this can help the public health system developing strategic planning to deals with the COVID-19.
  arXiv  Detail & Related papers  (2020-07-21T17:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.