MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

• URL: http://arxiv.org/abs/2608.22460v1
• Date: Sun, 23 Aug 2026 15:28:10 GMT
• Title: MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification
• Abstract summary: Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices.<n>We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases.<n>We evaluate cross-source risk classification using leave-one-dataset-out evaluation.
• Score: 2.1067460775364286
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.

Related papers

• The Accuracy Paradox: Empirical Diagnostic of Default Decision Thresholds in Multi-Label Enzyme Commission Prediction [With Code] [0.7530103765625609]
  We present a systematic empirical diagnostic of uncalibrated fixed decision boundaries operating under severe class imbalance.<n>Our results highlight a pronounced Accuracy Paradox: while the multi-label system a achieves a deceivingly high mean accuracy of 77.16%, the macro F1-score (0.3976) and macro recall (0.3872) reveal severe predictive breakdown.
  arXiv  Detail & Related papers  (2026-09-07T19:00:17Z)
• CASCADE: An Agentic Regulatory Network Framework for Patient-Data-Validated Downstream Perturbation Prediction [0.0]
  We use focal-gene copy-number amplification as a proxy for the inverse of knockdown against real TCGA patient tumor data.<n>For MYC, CASCADE's predicted knockdown targets show strong concordance with real amplified-vs-non-amplified tumor expression.<n>Its accuracy is not shown to exceed existing public knowledge of MYC- or E2F-driven biology.
  arXiv  Detail & Related papers  (2026-08-05T19:30:44Z)
• Cross-Fitted Residual Utility for Primary-Preserving Cognitive Decision Correction in Automatic Modulation Classification [1.547549252134621]
  We study a post-inference problem through cross-fitted residual utility and a primary-preserving cognitive decision policy.<n>A structured KAN-Fourier classifier supplies the default probability, while neural and non-neural candidates provide observable evidence.<n>On RMLA, RMLB, and HISAR, the complete system improves overall accuracy from 63.632% to 66.332%, 65.161% to 66.168%, and 77.769% to 79.867%.
  arXiv  Detail & Related papers  (2026-08-03T11:07:37Z)
• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It [0.0]
  Conformal prediction is being adopted in drug discovery to put an honest number on model reliability.<n>We show this guarantee can be dangerous on imbalanced datasets.<n>Our contribution is demonstrating on real chemistry how severe and invisible this known conformal-theory gap becomes under imbalance.
  arXiv  Detail & Related papers  (2026-07-07T04:31:56Z)
• Confidence is Not Reliability: Rethinking MC Dropout in Brain Tumour Segmentation [2.104759809264928]
  A segmentation model that fails silently on treatment-critical sub-regions represents a patient safety risk.<n>We ask whether voxel-level uncertainty estimation via Monte Carlo (MC) Dropout can reliably identify segmentation errors.<n>We evaluate a high-performance pretrained SegResNet and a locally trained UNet with residual units.
  arXiv  Detail & Related papers  (2026-06-17T17:24:41Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• Residual Stream Analysis of Overfitting And Structural Disruptions [26.444437808784482]
  We show that safety data exhibits substantially lower token entropy and 2-gram diversity compared to general instruction data.<n>To uncover the root cause, we introduce FlowLens, a stable PCA-based tool for residual-stream geometry analysis.<n>We propose Variance Concentration Loss, an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals.
  arXiv  Detail & Related papers  (2026-03-04T11:13:58Z)
• From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation [43.148402136307716]
  Cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal.<n>We transcribed 50 publicly available medical audio clips with eight ASR systems spanning commercial APIs and open-source engines.<n>Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass.
  arXiv  Detail & Related papers  (2026-03-02T13:02:13Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Evaluating AI systems under uncertain ground truth: a case study in dermatology [43.8328264420381]
  We show that ignoring uncertainty leads to overly optimistic estimates of model performance.<n>In skin condition classification, we find that a large portion of the dataset exhibits significant ground truth uncertainty.
  arXiv  Detail & Related papers  (2023-07-05T10:33:45Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.