Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration

• URL: http://arxiv.org/abs/2605.00972v1
• Date: Fri, 01 May 2026 17:03:33 GMT
• Title: Toward a Scientific Discovery Engine for Weather and Climate Data: A Visual Analytics Workbench for Embedding-Based Exploration
• Abstract summary: Earth system science is producing increasingly large, high-dimensional datasets from physics based Earth system models to AI-based weather and climate models.<n> Embedding-based representations can make these data searchable through similarity search and analog retrieval, but nearest neighbors in latent space are not automatically scientifically meaningful.<n>We present an open-source visual analytics workbench for each of these steps.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Earth system science is producing increasingly large, high-dimensional datasets from physics based Earth system models to AI-based weather and climate models. Embedding-based representations can make these data searchable through similarity search and analog retrieval, but nearest neighbors in latent space are not automatically scientifically meaningful: it may reflect real weather structure, or preprocessing, geography, or model bias. Researchers therefore need ways to inspect how embeddings organize meteorological data, compare representation models, develop retrieval strategies, and verify results against physical evidence. We present an open-source visual analytics workbench for each of these steps. The system links embedding experiments to source data, metadata, spatial context, and model configurations, so latent-space results can be traced back to the physics. Users can explore latent spaces for different models, issue global or localized queries, and inspect analogs through familiar meteorological views. This enables a discovery workflow in which scientists characterize a phenomenon of interest in a well-understood dataset, identifying its signature in latent space, and then use that signature to probe larger, less-labeled archives or ensembles for similar events. We demonstrate the workbench through tropical-cyclone retrieval using ERA5-derived embeddings and IBTrACS metadata, and evaluate its out-of-core retrieval backend to show that large embedding collections can be searched beyond in-memory limits on commodity workstation hardware.

Related papers

• Towards a Foundation Model for the Martian Atmosphere [7.786694785229998]
  The martian atmosphere hosts dynamical phenomena ranging from planet-encircling dust storms to mesoscale orographic clouds and nocturnal low-level jets.<n>General circulation model show capability to simulate these phenomena, but is computationally expensive at resolution needed to resolve mesoscale features.<n>These constraints motivate the development of a data-driven foundation model for the Martian atmosphere.
  arXiv  Detail & Related papers  (2026-05-16T20:37:05Z)
• ArtifactLinker: Linking Scientific Artifacts for Automatic State-of-the-Art Discovery [58.48859167777581]
  ArtifactLinker is a framework to automatically discover state-of-the-art (SOTA) models for datasets.<n>A benchmark named ArtifactBench with 14,053 artifacts and 51,337 relations is presented.<n>Results show that (1) graph structures between existing artifacts are effective for missing link prediction; (2) end-to-end ranking and verification with ArtifactLinker help discover potential SOTA results and research insights.
  arXiv  Detail & Related papers  (2026-05-16T09:26:08Z)
• Earth-o1: A Grid-free Observation-native Atmospheric World Model [81.55899748753434]
  We present Earth-o1, an observation-native atmospheric world model that overcomes structural limitations.<n>By integrating diverse sensor inputs into a unified, grid-free dynamical field, the model autonomously advances the atmospheric state in space and time.<n>We show that this fundamentally distinct paradigm enables direct, real-time forecasting and cross-sensor inference without the overhead of explicit numerical solvers.
  arXiv  Detail & Related papers  (2026-05-07T14:27:48Z)
• Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecasting [52.53742727060265]
  Foundation models (FMs) for the Earth system learn statistical relationships between physical variables across massive datasets.<n>Here, we introduce Earth System Foundation Model (ESFM), a fully open model building on the 3Dwin UNet backbone of the pioneering model.<n>We present findings using dense gridded data (ERA5, CMIP6), regionally masked dense data, sparse gridded MODIS satellite data, and station data.<n>Results demonstrate competitive or superior performance relative to state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2026-04-20T11:40:39Z)
• ReSearch: A Multi-Stage Machine Learning Framework for Earth Science Data Discovery [6.780086370528623]
  We introduce textbfReSearch, a multi-stage, reasoning-enhanced search framework that formulates Earth Science data discovery.<n>ReSearch integrates lexical search, semantic embeddings, abbreviation expansion, and large language model reranking within a unified architecture.<n>Experiments demonstrate that ReSearch consistently improves recall and ranking performance over baseline methods.
  arXiv  Detail & Related papers  (2026-01-20T17:27:12Z)
• Zephyrus: An Agentic Framework for Weather Science [47.611521052984365]
  Foundation models for weather science are pre-trained on vast amounts of structured numerical data and outperform traditional weather forecasting systems.<n>Large language models (LLMs) excel at understanding and generating text but cannot reason about high-dimensional meteorological datasets.<n>We bridge this gap by building a novel agentic framework for weather science.<n>We design Zephyrus, a multi-turn LLM-based weather agent that iteratively analyzes weather datasets, observes results, and refines its approach through conversational feedback loops.
  arXiv  Detail & Related papers  (2025-10-05T03:34:08Z)
• TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy [44.85881816317044]
  We show how to permuting 360-degree panoramic imagery into rich interconnected observation, state and action nodes.<n>We benchmark this dataset via TARDIS, a transformer-based generative world model.<n>We demonstrate robust performance across a range of agentic tasks such as controllable image synthesis, instruction following, autonomous self-control, and state-of-the-art georeferencing.
  arXiv  Detail & Related papers  (2025-06-12T21:08:11Z)
• Enabling High Data Throughput Reinforcement Learning on GPUs: A Domain Agnostic Framework for Data-Driven Scientific Research [90.91438597133211]
  We introduce WarpSci, a framework designed to overcome crucial system bottlenecks in the application of reinforcement learning. We eliminate the need for data transfer between the CPU and GPU, enabling the concurrent execution of thousands of simulations.
  arXiv  Detail & Related papers  (2024-08-01T21:38:09Z)
• DIRESA, a distance-preserving nonlinear dimension reduction technique based on regularized autoencoders [0.0]
  In meteorology, finding similar weather patterns or analogs in historical datasets can be useful for data assimilation, forecasting, and postprocessing.<n>We propose a dimension reduction technique based on autoencoder (AE) neural networks to compress the datasets and perform the search in an interpretable, compressed latent space.<n>A distance-regularized Siamese twin autoencoder (DIRESA) architecture is designed to preserve distance in latent space while capturing the nonlinearities in the datasets.
  arXiv  Detail & Related papers  (2024-04-28T20:54:57Z)
• Federated Prompt Learning for Weather Foundation Models on Devices [37.88417074427373]
  On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing. This paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD) FedPoD enables devices to obtain highly customized models while maintaining communication efficiency.
  arXiv  Detail & Related papers  (2023-05-23T16:59:20Z)
• Multimodal Dataset from Harsh Sub-Terranean Environment with Aerosol Particles for Frontier Exploration [55.41644538483948]
  This paper introduces a multimodal dataset from the harsh and unstructured underground environment with aerosol particles. It contains synchronized raw data measurements from all onboard sensors in Robot Operating System (ROS) format. The focus of this paper is not only to capture both temporal and spatial data diversities but also to present the impact of harsh conditions on captured data.
  arXiv  Detail & Related papers  (2023-04-27T20:21:18Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Gaze-based Object Detection in the Wild [23.923563888749108]
  In human-robot collaboration, one challenging task is to teach a robot new yet unknown objects. We investigate if it is possible to detect objects (object or no object) from gaze data and determine their bounding box parameters.
  arXiv  Detail & Related papers  (2022-03-29T15:10:17Z)
• Latent Feature Representation via Unsupervised Learning for Pattern Discovery in Massive Electron Microscopy Image Volumes [4.278591555984395]
  In particular, we give an unsupervised deep learning approach to learning a latent representation that captures semantic similarity in the data set. We demonstrate the utility of our method applied to nano-scale electron microscopy data, where even relatively small portions of animal brains can require terabytes of image data.
  arXiv  Detail & Related papers  (2020-12-22T17:14:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.