GAIR: Improving Multimodal Geo-Foundation Model with Geo-Aligned Implicit Representations

• URL: http://arxiv.org/abs/2503.16683v1
• Date: Thu, 20 Mar 2025 19:59:39 GMT
• Title: GAIR: Improving Multimodal Geo-Foundation Model with Geo-Aligned Implicit Representations
• Abstract summary: Key challenge in multimodal GeoFM development is to explicitly model geospatial relationships across modalities.<n>We propose GAIR, a novel multimodal GeoFM architecture integrating overhead RS data, street view imagery, and their geolocation metadata.<n>We evaluate GAIR across 10 geospatial tasks spanning RS image-based, SV image-based, and location embedding-based benchmarks.
• Score: 5.439918212297177
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advancements in vision and language foundation models have inspired the development of geo-foundation models (GeoFMs), enhancing performance across diverse geospatial tasks. However, many existing GeoFMs primarily focus on overhead remote sensing (RS) data while neglecting other data modalities such as ground-level imagery. A key challenge in multimodal GeoFM development is to explicitly model geospatial relationships across modalities, which enables generalizability across tasks, spatial scales, and temporal contexts. To address these limitations, we propose GAIR, a novel multimodal GeoFM architecture integrating overhead RS data, street view (SV) imagery, and their geolocation metadata. We utilize three factorized neural encoders to project an SV image, its geolocation, and an RS image into the embedding space. The SV image needs to be located within the RS image's spatial footprint but does not need to be at its geographic center. In order to geographically align the SV image and RS image, we propose a novel implicit neural representations (INR) module that learns a continuous RS image representation and looks up the RS embedding at the SV image's geolocation. Next, these geographically aligned SV embedding, RS embedding, and location embedding are trained with contrastive learning objectives from unlabeled data. We evaluate GAIR across 10 geospatial tasks spanning RS image-based, SV image-based, and location embedding-based benchmarks. Experimental results demonstrate that GAIR outperforms state-of-the-art GeoFMs and other strong baselines, highlighting its effectiveness in learning generalizable and transferable geospatial representations.

Related papers

• Geo-Embed: Towards Unified Multimodal Embeddings for Urban Understanding [18.530343758279493]
  We introduce GeoMEB, a large-scale multimodal embedding benchmark.<n>We present Geo-Embed, a unified embedding model that adapts a shared vision-language backbone to instruction-conditioned query-target matching.<n>Results motivate future geospatial embedders that organize training and evaluation around explicit query-target relations.
  arXiv  Detail & Related papers  (2026-08-04T15:36:17Z)
• Textual Supervision Enhances Geospatial Representations in Vision-Language Models [47.5986642837928]
  Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems.<n>We analyze the geospatial representations acquired by three model families: vision-only architectures (e.g., ViT), vision-language models (e.g., CLIP), and large-scale multimodal foundation models (e.g., LLaVA, Qwen, and Gemma)
  arXiv  Detail & Related papers  (2026-06-05T11:40:13Z)
• Learning to Wander: Improving the Global Image Geolocation Ability of LMMs via Actionable Reasoning [72.13218601075958]
  We introduce textbfWanderBench, the first open access global geolocation benchmark for actionable geolocation reasoning in embodied scenarios.<n>We propose textbfGeoAoT (Action of Thought), a underlineGeolocation framework with underlineAction of underlineThough, which couples reasoning with embodied actions.<n>Experiments on 19 large multimodal models show that GeoAoT achieves superior fine-grained localization and stronger generalization in dynamic environments.
  arXiv  Detail & Related papers  (2026-03-11T06:24:10Z)
• Global Cross-Modal Geo-Localization: A Million-Scale Dataset and a Physical Consistency Learning Framework [13.268717213871147]
  Cross-modal Geo-localization (CMGL) matches ground-level text descriptions with geo-tagged aerial imagery.<n>We introduce CORE, the first million-scale dataset dedicated to global CMGL.<n>We propose a physical-law-aware network (PLANET) for cross-modal geo-localization.
  arXiv  Detail & Related papers  (2026-03-09T15:27:19Z)
• GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding [23.253648429114236]
  We propose GeoViS, a progressive search-and-reasoning framework for remote sensing visual grounding.<n>We show that GeoViS consistently surpasses existing methods across key visual grounding metrics.
  arXiv  Detail & Related papers  (2025-12-02T12:45:52Z)
• GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data [13.450535648972682]
  This study presents GeoLink, a framework that leverages OpenStreetMap (OSM) data to enhance remote sensing (RS) foundation models (FMs)<n> Specifically, GeoLink enhances RS self-supervised pretraining using multi-granularity learning signals derived from OSM data.<n>For downstream tasks, GeoLink generates both unimodal and multimodal fine-grained encodings to support a wide range of applications.
  arXiv  Detail & Related papers  (2025-09-30T09:45:52Z)
• Geolocating Earth Imagery from ISS: Integrating Machine Learning with Astronaut Photography for Enhanced Geographic Mapping [0.0]
  This paper presents a novel approach to geolocating images captured from the International Space Station (ISS) using advanced machine learning algorithms. Our research addresses this gap by employing three distinct image processing pipelines: a Neural Network based approach, a SIFT based method, and GPT-4 model. Through extensive evaluation on a diverse dataset of over 140 ISS images, our methods demonstrate significant promise in automated geolocation with varied levels of success.
  arXiv  Detail & Related papers  (2025-04-29T22:00:02Z)
• OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence [51.0456395687016]
  multimodal large language models (LLMs) have opened new frontiers in artificial intelligence.<n>We propose a MLLM (OmniGeo) tailored to geospatial applications.<n>By combining the strengths of natural language understanding and spatial reasoning, our model enhances the ability of instruction following and the accuracy of GeoAI systems.
  arXiv  Detail & Related papers  (2025-03-20T16:45:48Z)
• Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework [59.42946541163632]
  We introduce a comprehensive geolocation framework with three key components.<n>GeoComp, a large-scale dataset; GeoCoT, a novel reasoning method; and GeoEval, an evaluation metric.<n>We demonstrate that GeoCoT significantly boosts geolocation accuracy by up to 25% while enhancing interpretability.
  arXiv  Detail & Related papers  (2025-02-19T14:21:25Z)
• Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework [51.26566634946208]
  We introduce smileGeo, a novel visual geo-localization framework. By inter-agent communication, smileGeo integrates the inherent knowledge of these agents with additional retrieved information. Results show that our approach significantly outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-21T03:31:30Z)
• ConGeo: Robust Cross-view Geo-localization across Ground View Variations [34.192775134189965]
  Cross-view geo-localization aims at localizing a ground-level query image by matching it to its corresponding geo-referenced aerial view. Existing learning pipelines are orientation-specific or FoV-specific, demanding separate model training for different ground view variations. We propose ConGeo, a single- and cross-view Contrastive method for Geo-localization.
  arXiv  Detail & Related papers  (2024-03-20T20:37:13Z)
• On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence [39.86997089245117]
  Foundations models (FMs) can be adapted to a wide range of downstream tasks by fine-tuning, few-shot, or zero-shot learning. We propose that one of the major challenges of developing a FM for GeoAI is to address the multimodality nature of geospatial tasks.
  arXiv  Detail & Related papers  (2023-04-13T19:50:17Z)
• Cross-View Visual Geo-Localization for Outdoor Augmented Reality [11.214903134756888]
  We address the problem of geo-pose estimation by cross-view matching of query ground images to a geo-referenced aerial satellite image database. We propose a new transformer neural network-based model and a modified triplet ranking loss for joint location and orientation estimation. Experiments on several benchmark cross-view geo-localization datasets show that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-03-28T01:58:03Z)
• A General Purpose Neural Architecture for Geospatial Systems [142.43454584836812]
  We present a roadmap towards the construction of a general-purpose neural architecture (GPNA) with a geospatial inductive bias. We envision how such a model may facilitate cooperation between members of the community.
  arXiv  Detail & Related papers  (2022-11-04T09:58:57Z)
• Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching [102.39635336450262]
  We address the problem of ground-to-satellite image geo-localization by matching a query image captured at the ground level against a large-scale database with geotagged satellite images. Our new method is able to achieve the fine-grained location of a query image, up to pixel size precision of the satellite image.
  arXiv  Detail & Related papers  (2022-03-26T20:10:38Z)
• Multi-Level Graph Convolutional Network with Automatic Graph Learning for Hyperspectral Image Classification [63.56018768401328]
  We propose a Multi-level Graph Convolutional Network (GCN) with Automatic Graph Learning method (MGCN-AGL) for HSI classification. By employing attention mechanism to characterize the importance among spatially neighboring regions, the most relevant information can be adaptively incorporated to make decisions. Our MGCN-AGL encodes the long range dependencies among image regions based on the expressive representations that have been produced at local level.
  arXiv  Detail & Related papers  (2020-09-19T09:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.