Scrutinizing Index-Based Risk Assessments: A Case Study in NYC Decision-making for Heat Emergency Management

• URL: http://arxiv.org/abs/2605.17697v1
• Date: Sun, 17 May 2026 23:36:31 GMT
• Title: Scrutinizing Index-Based Risk Assessments: A Case Study in NYC Decision-making for Heat Emergency Management
• Abstract summary: We study hand-crafted indices for geographic targeting and decision-making in emergency management.<n>We examine the challenges that practitioners may face in selecting an index for preparedness and response actions.
• Score: 8.2333980008702
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cities are increasingly turning to large-scale data analysis and machine learning to make consequential decisions. While the algorithmic fairness community has focused on analyzing the risks and benefits associated with these complex methods, there has been much less scrutiny of the many simpler, but still widely used, data-driven tools that support government decision-making in a variety of settings. In this work, we study hand-crafted indices for geographic targeting and decision-making in emergency management -- a field responsible for coordinating preparedness and response efforts to hazards ranging from natural disasters to human threats. Indices, which capture abstract principles and overarching priorities (e.g., reducing social vulnerability), are low-complexity models that statistically aggregate chosen variables. They are generally flexible and interpretable, but can also be sensitive to key design choices and require strong assumptions. Through a case study of decision-making for extreme heat emergencies in NYC, we examine the challenges that practitioners may face in selecting an index for preparedness and response actions. We map empirical findings from index-based simulations to concerns related to validity and reliability from the measurement literature and show via sensitivity analyses that different reasonable choices of input variables or spatial scale can result in substantive differences to index risk scores, thereby affecting downstream government decision-making. We contrast these challenges with considerations for developing predictive algorithms that more narrowly relate to concrete, measurable outcomes. Ultimately, we provide generalizable recommendations that practitioners and public-sector technologists can use for navigating the trade-offs between indices and predictive algorithms in other government settings.

Related papers

• Uncertainty-Aware Generation and Decision-Making Under Ambiguity [57.15357080072861]
  We present and evaluate various uncertainty-aware decision-making algorithms based on Bayesian decision theory and risk-averse decision making.<n>We find empirically that these algorithms can improve the utility of the generations but need to be carefully implemented when ambiguity is high.
  arXiv  Detail & Related papers  (2026-06-29T17:20:43Z)
• Strategic Feature Selection [62.74494413516484]
  We study strategic classification through feature selection and its interaction with ridge regularization.<n>We provide a fine-grained characterization of the performance of a feature subset under optimal regularization.<n>Motivated by this characterization, we develop a practical algorithm for jointly choosing the feature set and the level of ridge regularization.
  arXiv  Detail & Related papers  (2026-06-17T09:47:15Z)
• Actionable Counterfactual Explanations Using Bayesian Networks and Path Planning with Applications to Environmental Quality Improvement [4.041834517339835]
  We develop a method for actionable counterfactual explanations.<n>Data is only used to learn a density estimator.<n>We test our algorithm with a real-world Environmental Protection Agency dataset.
  arXiv  Detail & Related papers  (2025-08-04T17:20:50Z)
• Adapting Probabilistic Risk Assessment for AI [0.0]
  General-purpose artificial intelligence (AI) systems present an urgent risk management challenge.<n>Current methods often rely on selective testing and undocumented assumptions about risk priorities.<n>This paper introduces the probabilistic risk assessment (PRA) for AI framework.
  arXiv  Detail & Related papers  (2025-04-25T17:59:14Z)
• Predicting Human Choice Between Textually Described Lotteries [0.0]
  This study conducts the first large-scale exploration of human decision-making in such tasks.<n>We evaluate multiple computational approaches, including fine-tuning Large Language Models.
  arXiv  Detail & Related papers  (2025-03-18T08:10:33Z)
• ABI Approach: Automatic Bias Identification in Decision-Making Under Risk based in an Ontology of Behavioral Economics [46.57327530703435]
  Risk seeking preferences for losses, driven by biases such as loss aversion, pose challenges and can result in severe negative consequences. This research introduces the ABI approach, a novel solution designed to support organizational decision-makers by automatically identifying and explaining risk seeking preferences.
  arXiv  Detail & Related papers  (2024-05-22T23:53:46Z)
• On strategies for risk management and decision making under uncertainty shared across multiple fields [55.2480439325792]
  The paper finds more than 110 examples of such strategies and this approach to risk is termed RDOT: Risk-reducing Design and Operations Toolkit.<n>RDOT strategies fall into six broad categories: structural, reactive, formal, adversarial, multi-stage and positive.<n>Overall, RDOT represents an overlooked class of versatile responses to uncertainty.
  arXiv  Detail & Related papers  (2023-09-06T16:14:32Z)
• Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks [142.67349734180445]
  Existing algorithms that provide risk-awareness to deep neural networks are complex and ad-hoc. Here we present capsa, a framework for extending models with risk-awareness.
  arXiv  Detail & Related papers  (2023-08-01T02:07:47Z)
• On solving decision and risk management problems subject to uncertainty [91.3755431537592]
  Uncertainty is a pervasive challenge in decision and risk management. This paper develops a systematic understanding of such strategies, determine their range of application, and develop a framework to better employ them.
  arXiv  Detail & Related papers  (2023-01-18T19:16:23Z)
• Causal Fairness Analysis [68.12191782657437]
  We introduce a framework for understanding, modeling, and possibly solving issues of fairness in decision-making settings. The main insight of our approach will be to link the quantification of the disparities present on the observed data with the underlying, and often unobserved, collection of causal mechanisms. Our effort culminates in the Fairness Map, which is the first systematic attempt to organize and explain the relationship between different criteria found in the literature.
  arXiv  Detail & Related papers  (2022-07-23T01:06:34Z)
• On the Fairness of Machine-Assisted Human Decisions [3.4069627091757178]
  We show that the inclusion of a biased human decision-maker can revert common relationships between the structure of the algorithm and the qualities of resulting decisions. In the lab experiment, we demonstrate how predictions informed by gender-specific information can reduce average gender disparities in decisions.
  arXiv  Detail & Related papers  (2021-10-28T17:24:45Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)
• AIRSENSE-TO-ACT: A Concept Paper for COVID-19 Countermeasures based on Artificial Intelligence algorithms and multi-sources Data Processing [0.0]
  This paper describes a new tool to support institutions in the implementation of targeted countermeasures, based on quantitative and multi-scale elements, for the fight and prevention of emergencies, such as the current COVID-19 pandemic. The tool is a centralized system (web application), single multi-user platform, which relies on Artificial Intelligence (AI) algorithms for the processing of heterogeneous data, and which can produce an output level of risk. The model includes a specific neural network which will be first trained to learn the correlation between selected inputs, related to the case of interest: environmental variables (chemical-physical, such as meteorological), human activity
  arXiv  Detail & Related papers  (2020-11-07T17:50:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.