Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models

• URL: http://arxiv.org/abs/2606.28369v2
• Date: Tue, 30 Jun 2026 19:39:46 GMT
• Title: Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models
• Abstract summary: This work presents a novel framework that leverages AI foundation models, including Large Language Models (LLMs) and Vision-Language Models (VLMs)<n>By automatically linking relevant event reports, the proposed framework helps both data curators and the general public gain deeper insights into environmental change and its localized impacts.<n>These findings highlight the potential of AI foundation models to advance Geographic Information Retrieval (GIR) through multifaceted, intelligent analysis that integrates key geographic concepts: space, time, scale, and semantics.
• Score: 27.517725318193627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic search and recommendation of similar documents, such as news and reports about unusual environmental events (e.g., a dead whale washed ashore in Alaska) that contain spatial and temporal information, is a critical task in Geographic Information Retrieval (GIR). This work presents a novel framework that leverages AI foundation models, including Large Language Models (LLMs) and Vision-Language Models (VLMs), to enable effective similarity search and ranking for such event documents. To support this goal, we introduce two new strategies: (1) CAMERA (Context-Aware Multimodal Event Retrieval Algorithm), which fuses textual and visual information to generate richer embeddings than those derived from text alone; and (2) ASTRA (Adaptive Spatial and Temporal Re-ranking Algorithm), which improves similarity ranking by incorporating scale-dependent spatiotemporal relevance alongside semantic similarity. Experimental results, using a dataset from the Local Environmental Observer Network, demonstrate that our VLM-enhanced methods outperform unimodal, LLM-based approaches in similarity ranking effectiveness. By automatically linking relevant event reports, the proposed framework helps both data curators and the general public gain deeper insights into environmental change and its localized impacts. These findings highlight the potential of AI foundation models to advance GIR through multifaceted, intelligent analysis that integrates key geographic concepts: space, time, scale, and semantics.

Related papers

• Bridging the Agent-World Gap: Text World Models for LLM-based Agents [83.8237017249529]
  Large language model (LLM)-based agents are increasingly used in interactive textual environments.<n>Many remain largely reactive, mapping observations to actions without an explicit model of how these environments are structured and evolve.<n>This motivates text world models (TWMs): transition models over textual states that, given a state and a candidate action, predict the resulting webpage, terminal output, API response, or user reply.<n>We systematically review text world models for LLM-based agents, organized around a formal framework and the agent lifecycle.
  arXiv  Detail & Related papers  (2026-06-08T04:58:52Z)
• IGV-RRT: Prior-Real-Time Observation Fusion for Active Object Search in Changing Environments [10.748685496551067]
  We propose a probabilistic planning framework that combines uncertainty-aware scene priors with online target relevance estimates.<n>The framework contains a dual-layer semantic mapping module and a real-time planner.<n>The proposed method effectively mitigates the impact of object rearrangement, achieving higher search efficiency and success rates.
  arXiv  Detail & Related papers  (2026-03-23T12:14:26Z)
• EPRBench: A High-Quality Benchmark Dataset for Event Stream Based Visual Place Recognition [54.55914886780534]
  Event stream-based Visual Place Recognition (VPR) is an emerging research direction that offers a compelling solution to the instability of conventional visible-light cameras under challenging conditions such as low illumination, overexposure, and high-speed motion.<n>We introduce EPRBench, a high-quality benchmark specifically designed for event stream-based VPR.<n>EPRBench comprises 10K event sequences and 65K event frames, collected using both handheld and vehicle-mounted setups to comprehensively capture real-world challenges across diverse viewpoints, weather conditions, and lighting scenarios.
  arXiv  Detail & Related papers  (2026-02-13T13:25:05Z)
• Place Recognition Meet Multiple Modalitie: A Comprehensive Review, Current Challenges and Future Directions [2.4775350526606355]
  We review recent advancements in place recognition, emphasizing three methodological paradigms.<n>CNN-based approaches, Transformer-based frameworks, and cross-modal strategies are discussed.<n>We identify current research challenges and outline prospective directions, including domain adaptation, real-time performance, and lifelong learning, to inspire future advancements in this domain.
  arXiv  Detail & Related papers  (2025-05-20T08:16:37Z)
• REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observation [58.91579272882073]
  This paper introduces a novel benchmark dataset, called textbfREO-Instruct to unify regression and generation tasks specifically for the Earth Observation domain.<n>We develop textbfREO-VLM, a groundbreaking model that seamlessly integrates regression capabilities with traditional generative functions.
  arXiv  Detail & Related papers  (2024-12-21T11:17:15Z)
• Advancing Large Language Models for Spatiotemporal and Semantic Association Mining of Similar Environmental Events [10.15184389811683]
  This paper introduces a novel retrieval-retrieval framework leveraging Large Language Models (LLMs) We propose a Geo-Time Re-ranking (GTR-ranking) strategy that identifies multi-faceted criteria including spatial proximity, temporal association, semantic similarity, and category-instructed similarity to rank similartemporal events. We apply the proposed framework to a dataset of four thousand Local Environmental Observer (LEO) Network events, achieving performance in recommending similar events among multiple cutting-edge dense retrieval models.
  arXiv  Detail & Related papers  (2024-11-19T21:57:22Z)
• Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework [51.26566634946208]
  We introduce smileGeo, a novel visual geo-localization framework. By inter-agent communication, smileGeo integrates the inherent knowledge of these agents with additional retrieved information. Results show that our approach significantly outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-21T03:31:30Z)
• Temporal and Semantic Evaluation Metrics for Foundation Models in Post-Hoc Analysis of Robotic Sub-tasks [1.8124328823188356]
  We present an automated framework to decompose trajectory data into temporally bounded and natural language-based descriptive sub-tasks.<n>Our framework provides both time-based and language-based descriptions for lower-level sub-tasks that comprise full trajectories.<n>The metrics measure the temporal alignment and semantic fidelity of language descriptions between two sub-task decompositions.
  arXiv  Detail & Related papers  (2024-03-25T22:39:20Z)
• Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling [96.75821232222201]
  Existing research on multimodal relation extraction (MRE) faces two co-existing challenges, internal-information over-utilization and external-information under-exploitation. We propose a novel framework that simultaneously implements the idea of internal-information screening and external-information exploiting.
  arXiv  Detail & Related papers  (2023-05-19T14:56:57Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Unified Visual Relationship Detection with Vision and Language Models [89.77838890788638]
  This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. We propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model.
  arXiv  Detail & Related papers  (2023-03-16T00:06:28Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• Representation-Centric Survey of Skeletal Action Recognition and the ANUBIS Benchmark [43.00059447663327]
  3D skeleton-based human action recognition has emerged as a powerful alternative to traditional RGB and depth-based approaches.<n>Despite remarkable progress, current research remains fragmented across diverse input representations.<n>ANUBIS is a large-scale, challenging skeleton action dataset designed to address critical gaps in existing benchmarks.
  arXiv  Detail & Related papers  (2022-05-04T14:03:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.