Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

• URL: http://arxiv.org/abs/2609.07798v1
• Date: Mon, 07 Sep 2026 17:41:28 GMT
• Title: Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining
• Abstract summary: We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base.<n>We compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints.<n>Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources.
• Score: 8.17889232497434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints. Across nine climate NLP benchmarks, our best model achieves 76.3 average F_1, improving significantly over a vanilla ModernBERT baseline by 2.8 points. Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources, while parameter-space merging improves over joint multi-source training and better preserves complementary information from heterogeneous climate corpora. We release all Climate-ModernBERT variants and training checkpoints to support future research in climate NLP and domain-adaptive pretraining.

Related papers

• Climate-Eval: A Comprehensive Benchmark for NLP Tasks Related to Climate Change [2.8680187920555635]
  Climate-Eval aggregates existing datasets along with a newly developed news classification dataset.<n>This results in a benchmark of 25 tasks based on 13 datasets, covering key aspects of climate discourse.
  arXiv  Detail & Related papers  (2025-05-24T11:45:46Z)
• ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method [61.76389719956301]
  We contribute a multi-modal climate benchmark, i.e., ClimateBench-M, which aligns time series climate data from ERA5, extreme weather events data from NOAA, and satellite image data from NASA.<n>Under each data modality, we also propose a simple but strong generative method that could produce competitive performance in weather forecasting, thunderstorm alerts, and crop segmentation tasks.
  arXiv  Detail & Related papers  (2025-04-10T02:22:23Z)
• ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models [38.05357439484919]
  We develop ClimaGen, an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop.<n>We present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science.
  arXiv  Detail & Related papers  (2024-10-22T05:12:19Z)
• ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change [21.827936253363603]
  This paper introduces ClimateGPT, a model family of domain-specific large language models that synthesize interdisciplinary research on climate change. We trained two 7B models from scratch on a science-oriented dataset of 300B tokens. ClimateGPT-7B, 13B and 70B are continuously pre-trained from Llama2 on a domain-specific dataset of 4.2B tokens.
  arXiv  Detail & Related papers  (2024-01-17T23:29:46Z)
• Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM [77.17254959695218]
  Large Language Models (LLMs) like ChatGPT and Bard have shown impressive conversational abilities and excel in a wide variety of NLP tasks. We propose a light-weight Arabic Mini-ClimateGPT that is built on an open-source LLM and is specifically fine-tuned on a conversational-style instruction tuning Arabic dataset Clima500-Instruct. Our model surpasses the baseline LLM in 88.3% of cases during ChatGPT-based evaluation.
  arXiv  Detail & Related papers  (2023-12-14T22:04:07Z)
• Federated Prompt Learning for Weather Foundation Models on Devices [37.88417074427373]
  On-device intelligence for weather forecasting uses local deep learning models to analyze weather patterns without centralized cloud computing. This paper propose Federated Prompt Learning for Weather Foundation Models on Devices (FedPoD) FedPoD enables devices to obtain highly customized models while maintaining communication efficiency.
  arXiv  Detail & Related papers  (2023-05-23T16:59:20Z)
• ClimaX: A foundation model for weather and climate [51.208269971019504]
  ClimaX is a deep learning model for weather and climate science. It can be pre-trained with a self-supervised learning objective on climate datasets. It can be fine-tuned to address a breadth of climate and weather tasks.
  arXiv  Detail & Related papers  (2023-01-24T23:19:01Z)
• Towards Answering Climate Questionnaires from Unstructured Climate Reports [26.036105166376284]
  Activists and policymakers need NLP tools to process the vast and rapidly growing unstructured textual climate reports into structured form. We introduce two new large-scale climate questionnaire datasets and use their existing structure to train self-supervised models. We then use these models to help align texts from unstructured climate documents to the semi-structured questionnaires in a human pilot study.
  arXiv  Detail & Related papers  (2023-01-11T00:22:56Z)
• Spatiotemporal modeling of European paleoclimate using doubly sparse Gaussian processes [61.31361524229248]
  We build on recent scale sparsetemporal GPs to reduce the computational burden. We successfully employ such a doubly sparse GP to construct a probabilistic model of paleoclimate.
  arXiv  Detail & Related papers  (2022-11-15T14:15:04Z)
• Analyzing Sustainability Reports Using Natural Language Processing [68.8204255655161]
  In recent years, companies have increasingly been aiming to both mitigate their environmental impact and adapt to the changing climate context. This is reported via increasingly exhaustive reports, which cover many types of climate risks and exposures under the umbrella of Environmental, Social, and Governance (ESG) We present this tool and the methodology that we used to develop it in the present article.
  arXiv  Detail & Related papers  (2020-11-03T21:22:42Z)
• Dynamical Landscape and Multistability of a Climate Model [64.467612647225]
  We find a third intermediate stable state in one of the two climate models we consider. The combination of our approaches allows to identify how the negative feedback of ocean heat transport and entropy production drastically change the topography of Earth's climate.
  arXiv  Detail & Related papers  (2020-10-20T15:31:38Z)
• HECT: High-Dimensional Ensemble Consistency Testing for Climate Models [1.7587442088965226]
  Climate models play a crucial role in understanding the effect of environmental changes on climate to help mitigate climate risks and inform decisions. Large global climate models such as the Community Earth System Model (CESM), are very complex with millions of lines of code describing interactions of the atmosphere, land, oceans, and ice. Our work uses probabilistics like tree-based algorithms and deep neural networks to perform a statistically rigorous goodness-of-fit test of high-dimensional and man-made data.
  arXiv  Detail & Related papers  (2020-10-08T15:16:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.