Breaking the Statistical Similarity Trap in Extreme Convection Detection

• URL: http://arxiv.org/abs/2509.09195v1
• Date: Thu, 11 Sep 2025 07:10:45 GMT
• Title: Breaking the Statistical Similarity Trap in Extreme Convection Detection
• Authors: Md Tanveer Hossain Munim,
• Abstract summary: Current evaluation metrics for deep learning weather models reward blurry predictions while missing rare, high-impact events.<n>We introduce DART, a framework addressing the challenge of transforming coarse atmospheric forecasts into high-resolution satellite temperature fields.<n>DART employs dual-decoder architecture with explicit background/extreme decomposition, physically motivated oversampling, and task-specific loss functions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current evaluation metrics for deep learning weather models create a "Statistical Similarity Trap", rewarding blurry predictions while missing rare, high-impact events. We provide quantitative evidence of this trap, showing sophisticated baselines achieve 97.9% correlation yet 0.00 CSI for dangerous convection detection. We introduce DART (Dual Architecture for Regression Tasks), a framework addressing the challenge of transforming coarse atmospheric forecasts into high-resolution satellite brightness temperature fields optimized for extreme convection detection (below 220 K). DART employs dual-decoder architecture with explicit background/extreme decomposition, physically motivated oversampling, and task-specific loss functions. We present four key findings: (1) empirical validation of the Statistical Similarity Trap across multiple sophisticated baselines; (2) the "IVT Paradox", removing Integrated Water Vapor Transport, widely regarded as essential for atmospheric river analysis, improves extreme convection detection by 270%; (3) architectural necessity demonstrated through operational flexibility (DART achieves CSI = 0.273 with bias = 2.52 vs. 6.72 for baselines at equivalent CSI), and (4) real-world validation with the August 2023 Chittagong flooding disaster as a case study. To our knowledge, this is the first work to systematically address this hybrid conversion-segmentation-downscaling task, with no direct prior benchmarks identified in existing literature. Our validation against diverse statistical and deep learning baselines sufficiently demonstrates DART's specialized design. The framework enables precise operational calibration through beta-tuning, trains in under 10 minutes on standard hardware, and integrates seamlessly with existing meteorological workflows, demonstrating a pathway toward trustworthy AI for extreme weather preparedness.

Related papers

• Information-Theoretic Digital Twins for Stealthy Attack Detection in Industrial Control Systems: A Closed-Form KL Divergence Approach [0.0]
  Digital twins (DTs) are increasingly used to monitor and secure Industrial Control Systems (ICS)<n>But detecting stealthy False Data Injection Attacks (FDIAs) that manipulate system states within normal physical bounds remains challenging.<n>We propose a closed-loop Information-Theoretic Digital Twin (IT-DT) framework for real-time anomaly detection.
  arXiv  Detail & Related papers  (2026-03-02T08:56:11Z)
• Function-Space Decoupled Diffusion for Forward and Inverse Modeling in Carbon Capture and Storage [65.51149575007149]
  We present Fun-DDPS, a generative framework that combines function-space diffusion models with differentiable neural operator surrogates for both forward and inverse modeling.<n>Fun-DDPS produces physically consistent realizations free from the high-frequency artifacts observed in joint-state baselines.
  arXiv  Detail & Related papers  (2026-02-12T18:58:12Z)
• Demystifying Data-Driven Probabilistic Medium-Range Weather Forecasting [63.8116386935854]
  We demonstrate that state-of-the-art probabilistic skill requires neither intricate architectural constraints nor specialized trainings.<n>We introduce a scalable framework for learning multi-scale atmospheric dynamics by combining a directly downsampled latent space with a history-conditioned local projector.<n>We find that our framework design is robust to the choice of probabilistic estimators, seamlessly supporting interpolants, diffusion models, and CRPS-based ensemble training.
  arXiv  Detail & Related papers  (2026-01-26T03:52:16Z)
• Diffuse to Detect: A Generalizable Framework for Anomaly Detection with Diffusion Models Applications to UAVs and Beyond [2.4449457537548036]
  Anomaly detection in complex, high-dimensional data, such as UAV sensor readings, is essential for operational safety.<n>We propose the Diffuse to Detect (DTD) framework, a novel approach that adapts diffusion models for anomaly detection.<n>DTD employs a single-step diffusion process to predict noise patterns, enabling rapid and precise identification of anomalies without reconstruction errors.
  arXiv  Detail & Related papers  (2025-10-27T02:08:08Z)
• A Multimodal Approach to Heritage Preservation in the Context of Climate Change [0.0]
  We propose a lightweight multimodal architecture that fuses sensor data (temperature, humidity) with visual imagery to predict severity at heritage sites.<n>On data from Strasbourg Cathedral, our model achieves 76.9% accu- racy, a 43% improvement over standard multimodal architectures.
  arXiv  Detail & Related papers  (2025-10-15T22:07:57Z)
• Contrastive-KAN: A Semi-Supervised Intrusion Detection Framework for Cybersecurity with scarce Labeled Data [0.0]
  We propose a real-time intrusion detection system based on a semi-supervised contrastive learning framework.<n>Our method leverages abundant unlabeled data to effectively distinguish between normal and attack behaviors.<n> Experimental results show that our method outperforms existing contrastive learning-based approaches.
  arXiv  Detail & Related papers  (2025-07-14T21:02:34Z)
• ReconMOST: Multi-Layer Sea Temperature Reconstruction with Observations-Guided Diffusion [48.540756751934836]
  ReconMOST is a data-driven guided diffusion model framework for multi-layer sea temperature reconstruction.<n>Our method extends ML-based SST reconstruction to a global, multi-layer setting, handling over 92.5% missing data.
  arXiv  Detail & Related papers  (2025-06-12T06:27:22Z)
• Accurate and Reliable Predictions with Mutual-Transport Ensemble [46.368395985214875]
  We propose a co-trained auxiliary model and adaptively regularizes the cross-entropy loss using Kullback-Leibler (KL) We show that MTE can simultaneously enhance both accuracy and uncertainty calibration. For example, on the CIFAR-100 dataset, our MTE method on ResNet34/50 achieved significant improvements compared to previous state-of-the-art method.
  arXiv  Detail & Related papers  (2024-05-30T03:15:59Z)
• Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling [58.456404022536425]
  State of the art for physical hazard prediction from weather and climate requires expensive km-scale numerical simulations driven by coarser resolution global inputs. Here, a generative diffusion architecture is explored for downscaling such global inputs to km-scale, as a cost-effective machine learning alternative. The model is trained to predict 2km data from a regional weather model over Taiwan, conditioned on a 25km global reanalysis.
  arXiv  Detail & Related papers  (2023-09-24T19:57:22Z)
• Inference from Real-World Sparse Measurements [21.194357028394226]
  Real-world problems often involve complex and unstructured sets of measurements, which occur when sensors are sparsely placed in either space or time. Deep learning architectures capable of processing sets of measurements with positions varying from set to set and extracting readouts anywhere are methodologically difficult. We propose an attention-based model focused on applicability and practical robustness, with two key design contributions.
  arXiv  Detail & Related papers  (2022-10-20T13:42:20Z)
• The KFIoU Loss for Rotated Object Detection [115.334070064346]
  In this paper, we argue that one effective alternative is to devise an approximate loss who can achieve trend-level alignment with SkewIoU loss. Specifically, we model the objects as Gaussian distribution and adopt Kalman filter to inherently mimic the mechanism of SkewIoU. The resulting new loss called KFIoU is easier to implement and works better compared with exact SkewIoU.
  arXiv  Detail & Related papers  (2022-01-29T10:54:57Z)
• T$^2$-Net: A Semi-supervised Deep Model for Turbulence Forecasting [65.498967509424]
  Air turbulence forecasting can help airlines avoid hazardous turbulence, guide routes that keep passengers safe, maximize efficiency, reduce costs. Traditional forecasting approaches rely on painstakingly customized turbulence indexes, which are less effective in dynamic and complex weather conditions. We propose a machine learning based turbulence forecasting system due to two challenges: (1) Complex-temporal correlations, and (2) scarcity, very limited turbulence labels can be obtained.
  arXiv  Detail & Related papers  (2020-10-26T21:14:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.