SounDiT: Geo-Contextual Soundscape-to-Landscape Generation

• URL: http://arxiv.org/abs/2505.12734v1
• Date: Mon, 19 May 2025 05:47:13 GMT
• Title: SounDiT: Geo-Contextual Soundscape-to-Landscape Generation
• Authors: Junbo Wang, Haofeng Tan, Bowen Liao, Albert Jiang, Teng Fei, Qixing Huang, Zhengzhong Tu, Shan Ye, Yuhao Kang,
• Abstract summary: We present a novel problem-Geo-Contextual Soundscape-to-Landscape (GeoS2L) generation.<n>GeoS2L aims to synthesize geographically realistic landscape images from environmental soundscapes.
• Score: 28.099729084181092
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a novel and practically significant problem-Geo-Contextual Soundscape-to-Landscape (GeoS2L) generation-which aims to synthesize geographically realistic landscape images from environmental soundscapes. Prior audio-to-image generation methods typically rely on general-purpose datasets and overlook geographic and environmental contexts, resulting in unrealistic images that are misaligned with real-world environmental settings. To address this limitation, we introduce a novel geo-contextual computational framework that explicitly integrates geographic knowledge into multimodal generative modeling. We construct two large-scale geo-contextual multimodal datasets, SoundingSVI and SonicUrban, pairing diverse soundscapes with real-world landscape images. We propose SounDiT, a novel Diffusion Transformer (DiT)-based model that incorporates geo-contextual scene conditioning to synthesize geographically coherent landscape images. Furthermore, we propose a practically-informed geo-contextual evaluation framework, the Place Similarity Score (PSS), across element-, scene-, and human perception-levels to measure consistency between input soundscapes and generated landscape images. Extensive experiments demonstrate that SounDiT outperforms existing baselines in both visual fidelity and geographic settings. Our work not only establishes foundational benchmarks for GeoS2L generation but also highlights the importance of incorporating geographic domain knowledge in advancing multimodal generative models, opening new directions at the intersection of generative AI, geography, urban planning, and environmental sciences.

Related papers

• Cross-Modal Urban Sensing: Evaluating Sound-Vision Alignment Across Street-Level and Aerial Imagery [13.86994497464469]
  We employ a multimodal approach that integrates geo-referenced sound recordings with both street-level and remote sensing imagery.<n>We find that embedding-based models offer superior semantic alignment, while segmentation-based methods provide interpretable links between visual structure and acoustic ecology.
  arXiv  Detail & Related papers  (2025-06-03T20:56:37Z)
• Geolocation with Real Human Gameplay Data: A Large-Scale Dataset and Human-Like Reasoning Framework [59.42946541163632]
  We introduce a comprehensive geolocation framework with three key components.<n>GeoComp, a large-scale dataset; GeoCoT, a novel reasoning method; and GeoEval, an evaluation metric.<n>We demonstrate that GeoCoT significantly boosts geolocation accuracy by up to 25% while enhancing interpretability.
  arXiv  Detail & Related papers  (2025-02-19T14:21:25Z)
• Swarm Intelligence in Geo-Localization: A Multi-Agent Large Vision-Language Model Collaborative Framework [51.26566634946208]
  We introduce smileGeo, a novel visual geo-localization framework. By inter-agent communication, smileGeo integrates the inherent knowledge of these agents with additional retrieved information. Results show that our approach significantly outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-21T03:31:30Z)
• G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Models [40.69217368870192]
  We propose a novel framework for worldwide geolocalization based on Retrieval-Augmented Generation (RAG) G3 consists of three steps, i.e., Geo-alignment, Geo-diversification, and Geo-verification. Experiments on two well-established datasets verify the superiority of G3 compared to other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-05-23T15:37:06Z)
• MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation [24.193486441413803]
  We present MetaEarth, a generative foundation model that breaks the barrier by scaling image generation to a global level. In MetaEarth, we propose a resolution-guided self-cascading generative framework, which enables the generating of images at any region with a wide range of geographical resolutions. Our model opens up new possibilities for constructing generative world models by simulating Earth visuals from an innovative overhead perspective.
  arXiv  Detail & Related papers  (2024-05-22T12:07:47Z)
• GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization [61.10806364001535]
  Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. Existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. We propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations.
  arXiv  Detail & Related papers  (2023-09-27T20:54:56Z)
• GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark [56.08664336835741]
  We propose a GeoGraphic Language Understanding Evaluation benchmark, named GeoGLUE. We collect data from open-released geographic resources and introduce six natural language understanding tasks. We pro vide evaluation experiments and analysis of general baselines, indicating the effectiveness and significance of the GeoGLUE benchmark.
  arXiv  Detail & Related papers  (2023-05-11T03:21:56Z)
• GeoNet: Benchmarking Unsupervised Adaptation across Geographies [71.23141626803287]
  We study the problem of geographic robustness and make three main contributions. First, we introduce a large-scale dataset GeoNet for geographic adaptation. Second, we hypothesize that the major source of domain shifts arise from significant variations in scene context. Third, we conduct an extensive evaluation of several state-of-the-art unsupervised domain adaptation algorithms and architectures.
  arXiv  Detail & Related papers  (2023-03-27T17:59:34Z)
• Where We Are and What We're Looking At: Query Based Worldwide Image Geo-localization Using Hierarchies and Scenes [53.53712888703834]
  We introduce an end-to-end transformer-based architecture that exploits the relationship between different geographic levels. We achieve state of the art street level accuracy on 4 standard geo-localization datasets.
  arXiv  Detail & Related papers  (2023-03-07T21:47:58Z)
• PetroGAN: A novel GAN-based approach to generate realistic, label-free petrographic datasets [0.0]
  We develop a novel deep learning framework based on generative adversarial networks (GANs) to create the first realistic synthetic petrographic dataset. The training dataset consists of 10070 images of rock thin sections both in plane- and cross-polarized light. The algorithm trained for 264 GPU hours and reached a state-of-the-art Fr'echet Inception Distance (FID) score of 12.49 for petrographic images.
  arXiv  Detail & Related papers  (2022-04-07T01:55:53Z)
• Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching [102.39635336450262]
  We address the problem of ground-to-satellite image geo-localization by matching a query image captured at the ground level against a large-scale database with geotagged satellite images. Our new method is able to achieve the fine-grained location of a query image, up to pixel size precision of the satellite image.
  arXiv  Detail & Related papers  (2022-03-26T20:10:38Z)
• Visual and Object Geo-localization: A Comprehensive Survey [11.120155713865918]
  Geo-localization refers to the process of determining where on earth some entity' is located. This paper provides a comprehensive survey of geo-localization involving images, which involves either determining from where an image has been captured (Image geo-localization) or geo-locating objects within an image (Object geo-localization) We will provide an in-depth study, including a summary of popular algorithms, a description of proposed datasets, and an analysis of performance results to illustrate the current state of each field.
  arXiv  Detail & Related papers  (2021-12-30T20:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.