GeoLangBind: Unifying Earth Observation with Agglomerative Vision-Language Foundation Models

• URL: http://arxiv.org/abs/2503.06312v1
• Date: Sat, 08 Mar 2025 19:10:04 GMT
• Title: GeoLangBind: Unifying Earth Observation with Agglomerative Vision-Language Foundation Models
• Authors: Zhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Jie Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, Xiao Xiang Zhu,
• Abstract summary: GeoLangBind is a novel agglomerative vision--language foundation model.<n>It bridges the gap between heterogeneous EO data modalities using language as a unifying medium.<n>Our approach aligns different EO data types into a shared language embedding space.
• Score: 27.878058177228727
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Earth observation (EO) data, collected from diverse sensors with varying imaging principles, present significant challenges in creating unified analytical frameworks. We present GeoLangBind, a novel agglomerative vision--language foundation model that bridges the gap between heterogeneous EO data modalities using language as a unifying medium. Our approach aligns different EO data types into a shared language embedding space, enabling seamless integration and complementary feature learning from diverse sensor data. To achieve this, we construct a large-scale multimodal image--text dataset, GeoLangBind-2M, encompassing six data modalities. GeoLangBind leverages this dataset to develop a zero-shot foundation model capable of processing arbitrary numbers of EO data channels as input. Through our designed Modality-aware Knowledge Agglomeration (MaKA) module and progressive multimodal weight merging strategy, we create a powerful agglomerative foundation model that excels in both zero-shot vision--language comprehension and fine-grained visual understanding. Extensive evaluation across 23 datasets covering multiple tasks demonstrates GeoLangBind's superior performance and versatility in EO applications, offering a robust framework for various environmental monitoring and analysis tasks. The dataset and pretrained models will be publicly available.

Related papers

• OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence [51.0456395687016]
  multimodal large language models (LLMs) have opened new frontiers in artificial intelligence. We propose a MLLM (OmniGeo) tailored to geospatial applications. By combining the strengths of natural language understanding and spatial reasoning, our model enhances the ability of instruction following and the accuracy of GeoAI systems.
  arXiv  Detail & Related papers  (2025-03-20T16:45:48Z)
• EarthView: A Large Scale Remote Sensing Dataset for Self-Supervision [72.84868704100595]
  This paper presents a dataset specifically designed for self-supervision on remote sensing data, intended to enhance deep learning applications on Earth monitoring tasks.<n>The dataset spans 15 tera pixels of global remote-sensing data, combining imagery from a diverse range of sources, including NEON, Sentinel, and a novel release of 1m spatial resolution data from Satellogic.<n>Accompanying the dataset is EarthMAE, a tailored Masked Autoencoder developed to tackle the distinct challenges of remote sensing data.
  arXiv  Detail & Related papers  (2025-01-14T13:42:22Z)
• Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models [55.25892137362187]
  We introduce a new dataset featuring Multimodal Multi-Grained Concept annotations (MMGiC) for MLLMs.<n>Our analyses reveal that multi-grained concept annotations integrate and complement each other, under our structured template and a general MLLM framework.<n>We further validate our hypothesis by investigating the fair comparison and effective collaboration between MMGiC and image--caption data on 12 multimodal comprehension and generation benchmarks.
  arXiv  Detail & Related papers  (2024-12-08T13:45:44Z)
• FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension [10.482908189805872]
  Referring Expression (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding.<n>We have established a new REC dataset characterized by two key features.<n>It includes negative text and images created through fine-grained editing and generation based on existing data.
  arXiv  Detail & Related papers  (2024-09-23T06:56:51Z)
• Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations [5.065947993017157]
  This study introduces an approach to curate vision-language datasets by employing an image decoding machine learning model. We amassed approximately 9.6 million vision-language paired datasets in VHR imagery. The resultant model outperformed counterparts that did not leverage publicly available vision-language datasets.
  arXiv  Detail & Related papers  (2024-09-11T06:36:08Z)
• Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework [51.26566634946208]
  We introduce smileGeo, a novel visual geo-localization framework. By inter-agent communication, smileGeo integrates the inherent knowledge of these agents with additional retrieved information. Results show that our approach significantly outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-21T03:31:30Z)
• Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models [49.439311430360284]
  We introduce a novel data synthesis method inspired by contrastive learning and image difference captioning.<n>Our key idea involves challenging the model to discern both matching and distinct elements.<n>We leverage this generated dataset to fine-tune state-of-the-art (SOTA) MLLMs.
  arXiv  Detail & Related papers  (2024-08-08T17:10:16Z)
• Towards Vision-Language Geo-Foundation Model: A Survey [65.70547895998541]
  Vision-Language Foundation Models (VLFMs) have made remarkable progress on various multimodal tasks. This paper thoroughly reviews VLGFMs, summarizing and analyzing recent developments in the field.
  arXiv  Detail & Related papers  (2024-06-13T17:57:30Z)
• General Object Foundation Model for Images and Videos at Scale [99.2806103051613]
  We present GLEE, an object-level foundation model for locating and identifying objects in images and videos. GLEE accomplishes detection, segmentation, tracking, grounding, and identification of arbitrary objects in the open world scenario. We employ an image encoder, text encoder, and visual prompter to handle multi-modal inputs, enabling to simultaneously solve various object-centric downstream tasks.
  arXiv  Detail & Related papers  (2023-12-14T17:26:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.