MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understanding

• URL: http://arxiv.org/abs/2605.26320v1
• Date: Mon, 25 May 2026 20:35:48 GMT
• Title: MULTISEISMO: A Multimodal Seismic Dataset and Model for Cross-Modal Seismic Understanding
• Abstract summary: In seismology, understanding earthquake phenomena requires the synthesis of timeseries waveform data, geographical imagery, and contextual metadata.<n>We present MultiSeismo, a large scale structured multimodal dataset spanning over 16K seismic events spanning 13 years (2010 to 2023) across diverse geographical regions.<n>Each event data integrates waveform recordings from global station networks, intensity maps, population exposure visualizations, and a comprehensive textual format.<n>We leverage finetune an existing multimodal model (Unified IO 2) enhanced with a specialized timeseries encoder, which yields SeisModal, the first domain specific multimodal model for comprehensive seismic analysis.
• Score: 1.0407464689506312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The application of generalist multimodal models (GMMs) to specialized scientific domains remains limited due to the scarcity of comprehensive domain-specific datasets that integrate multiple data modalities beyond text and images. In seismology, understanding earthquake phenomena requires the synthesis of timeseries waveform data, geographical imagery, and contextual metadata, a multimodal integration absent in existing seismic datasets. We present MultiSeismo, a large scale structured multimodal seismic dataset, comprising over 16K seismic events spanning 13 years (2010 to 2023) across diverse geographical regions. Each event data integrates waveform recordings from global station networks, intensity maps, population exposure visualizations, and a comprehensive textual description within a standardized JSON format. We additionally develop MISCE, a multimodal instruction set on top of raw data to enable supervised training and evaluation of GMMs on seismic reasoning tasks ranging from basic information retrieval to complex cross modal analysis. We leverage MISCE to finetune an existing multimodal model (Unified IO 2) enhanced with a specialized timeseries encoder, which yields SeisModal, the first domain specific multimodal model for comprehensive seismic analysis. Evaluation of state of the art multimodal models on MultiSeismo reveals significant challenges, particularly with time-series data processing for general purpose models, while demonstrating SeisModal's superior performance on seismic multimodal reasoning tasks. These results prove that MultiSeismo provides a rigorous benchmark for future multimodal research in seismology and validate the success of our domain specific architectural adaptations.

Related papers

• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe [52.210953881548996]
  We show that a vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks.<n>Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding.<n>Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks.
  arXiv  Detail & Related papers  (2026-07-17T13:25:44Z)
• RMISC: A Large-scale Real-world Multivariate Corpus for Time Series Foundation Models [11.251202834387682]
  We establish a large-scale, high-quality, real-world, and multivariate time series archive that contains around 200 datasets and 142 billion time points.<n> Experimental results show that incorporating real-world multivariate data predominantly improves the generalization performance.
  arXiv  Detail & Related papers  (2026-07-07T17:04:44Z)
• Empowering Time Series Analysis with Large-Scale Multimodal Pretraining [34.22079919081765]
  Building multimodal foundation models is a natural next step, but it faces key challenges.<n>Lack of a unified multimodal pretraining paradigm and large-scale multimodal corpora for time series analysis.<n>How to effectively integrate heterogeneous modalities and enhance model generalization.
  arXiv  Detail & Related papers  (2026-02-05T13:26:35Z)
• A General-Purpose Diversified 2D Seismic Image Dataset from NAMSS [1.9685228653779]
  The Unicamp-NAMSS dataset is a large, diverse, and geographically distributed collection of migrated 2D seismic sections.<n>We constructed the dataset from the National Archive of Marine Seismic Surveys (NAMSS), which contains decades of publicly available marine seismic data.
  arXiv  Detail & Related papers  (2026-01-27T14:01:29Z)
• GridMind: A Multi-Agent NLP Framework for Unified, Cross-Modal NFL Data Insights [0.0]
  This paper introduces GridMind, a framework that unifies structured, semi-structured, and unstructured data through Retrieval-Augmented Generation (RAG) and large language models (LLMs)<n>This approach aligns with the evolving field of multimodal representation learning, where unified models are increasingly essential for real-time, cross-modal interactions.
  arXiv  Detail & Related papers  (2025-03-24T18:33:36Z)
• Multi-modal Time Series Analysis: A Tutorial and Survey [36.93906365779472]
  Multi-modal time series analysis has emerged as a prominent research area in data mining.<n>However, effective analysis of multi-modal time series is hindered by data heterogeneity, modality gap, misalignment, and inherent noise.<n>Recent advancements in multi-modal time series methods have exploited the multi-modal context via cross-modal interactions.
  arXiv  Detail & Related papers  (2025-03-17T20:30:02Z)
• MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments [49.45034796115852]
  Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment.<n>Current datasets fall short in scale, realism and do not capture the nature of OR scenes, limiting multimodal in OR modeling.<n>We introduce MM-OR, a realistic and large-scale multimodal OR dataset, and first dataset to enable multimodal scene graph generation.
  arXiv  Detail & Related papers  (2025-03-04T13:00:52Z)
• mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data [71.352883755806]
  Multimodal embedding models have gained significant attention for their ability to map data from different modalities, such as text and images, into a unified representation space.<n>However, the limited labeled multimodal data often hinders embedding performance.<n>Recent approaches have leveraged data synthesis to address this problem, yet the quality of synthetic data remains a critical bottleneck.
  arXiv  Detail & Related papers  (2025-02-12T15:03:33Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [79.58755811919366]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• UniTraj: Learning a Universal Trajectory Foundation Model from Billion-Scale Worldwide Traces [64.24594320103066]
  Building a universal trajectory foundation model is a promising solution to address the limitations of existing trajectory modeling approaches.<n>We introduce UniTraj, a Universal Trajectory foundation model that aims to address these limitations through three key innovations.<n>First, we construct WorldTrace, an unprecedented dataset of 2.45 million trajectories with billions of GPS points spanning 70 countries.
  arXiv  Detail & Related papers  (2024-11-06T12:06:43Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction [16.01738433164131]
  We develop SeisCLIP, a seismology foundation model trained through contrastive learning from multi-modal data. It consists of a transformer encoder for extracting crucial features from time-frequency seismic spectrum and an foundational encoder for integrating the phase and source information of the same event. Notably, SeisCLIP's performance surpasses that of baseline methods in event classification, localization, and focal mechanism analysis tasks.
  arXiv  Detail & Related papers  (2023-09-05T15:40:13Z)
• Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework [89.8609061423685]
  We propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks. We demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies.
  arXiv  Detail & Related papers  (2023-02-23T18:59:05Z)
• High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning [112.51498431119616]
  This paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. A single model, HighMMT, scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas.
  arXiv  Detail & Related papers  (2022-03-02T18:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.