Learning Street View Representations with Spatiotemporal Contrast

• URL: http://arxiv.org/abs/2502.04638v1
• Date: Fri, 07 Feb 2025 03:47:54 GMT
• Title: Learning Street View Representations with Spatiotemporal Contrast
• Authors: Yong Li, Yingjing Huang, Gengchen Mai, Fan Zhang,
• Abstract summary: We propose an innovative self-supervised learning framework that leverages temporal and spatial attributes of street view imagery to learn image representations of the dynamic urban environment. Our approach significantly outperforms traditional supervised and unsupervised methods in tasks such as visual place recognition, socioeconomic estimation, and human-environment perception.
• Score: 7.005144428229216
• License:
• Abstract: Street view imagery is extensively utilized in representation learning for urban visual environments, supporting various sustainable development tasks such as environmental perception and socio-economic assessment. However, it is challenging for existing image representations to specifically encode the dynamic urban environment (such as pedestrians, vehicles, and vegetation), the built environment (including buildings, roads, and urban infrastructure), and the environmental ambiance (such as the cultural and socioeconomic atmosphere) depicted in street view imagery to address downstream tasks related to the city. In this work, we propose an innovative self-supervised learning framework that leverages temporal and spatial attributes of street view imagery to learn image representations of the dynamic urban environment for diverse downstream tasks. By employing street view images captured at the same location over time and spatially nearby views at the same time, we construct contrastive learning tasks designed to learn the temporal-invariant characteristics of the built environment and the spatial-invariant neighborhood ambiance. Our approach significantly outperforms traditional supervised and unsupervised methods in tasks such as visual place recognition, socioeconomic estimation, and human-environment perception. Moreover, we demonstrate the varying behaviors of image representations learned through different contrastive learning objectives across various downstream tasks. This study systematically discusses representation learning strategies for urban studies based on street view images, providing a benchmark that enhances the applicability of visual data in urban science. The code is available at https://github.com/yonglleee/UrbanSTCL.

Related papers

• When Does Perceptual Alignment Benefit Vision Representations? [76.32336818860965]
  We investigate how aligning vision model representations to human perceptual judgments impacts their usability. We find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks. Our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations.
  arXiv  Detail & Related papers  (2024-10-14T17:59:58Z)
• Visualizing Routes with AI-Discovered Street-View Patterns [4.153397474276339]
  We propose a solution of using semantic latent vectors for quantifying visual appearance features. We calculate image similarities among a large set of street-view images and then discover spatial imagery patterns. We present VivaRoutes, an interactive visualization prototype, to show how visualizations leveraged with these discovered patterns can help users effectively and interactively explore multiple routes.
  arXiv  Detail & Related papers  (2024-03-30T17:32:26Z)
• Incorporating simulated spatial context information improves the effectiveness of contrastive learning models [1.4179832037924995]
  We present a unique approach, termed Environmental Spatial Similarity (ESS), that complements existing contrastive learning methods. ESS allows remarkable proficiency in room classification and spatial prediction tasks, especially in unfamiliar environments. Potentially transformative applications span from robotics to space exploration.
  arXiv  Detail & Related papers  (2024-01-26T03:44:58Z)
• Knowledge-infused Contrastive Learning for Urban Imagery-based Socioeconomic Prediction [13.26632316765164]
  Urban imagery in web like satellite/street view images has emerged as an important source for socioeconomic prediction. We propose a Knowledge-infused Contrastive Learning model for urban imagery-based socioeconomic prediction. Our proposed KnowCL model can apply to both satellite and street imagery with both effectiveness and transferability achieved.
  arXiv  Detail & Related papers  (2023-02-25T14:53:17Z)
• Street-View Image Generation from a Bird's-Eye View Layout [95.36869800896335]
  Bird's-Eye View (BEV) Perception has received increasing attention in recent years. Data-driven simulation for autonomous driving has been a focal point of recent research. We propose BEVGen, a conditional generative model that synthesizes realistic and spatially consistent surrounding images.
  arXiv  Detail & Related papers  (2023-01-11T18:39:34Z)
• Urban Visual Intelligence: Studying Cities with AI and Street-level Imagery [12.351356101876616]
  This paper reviews the literature on the appearance and function of cities to illustrate how visual information has been used to understand them. A conceptual framework, Urban Visual Intelligence, is introduced to elaborate on how new image data sources and AI techniques are reshaping the way researchers perceive and measure cities.
  arXiv  Detail & Related papers  (2023-01-02T10:00:26Z)
• Mitigating Urban-Rural Disparities in Contrastive Representation Learning with Satellite Imagery [19.93324644519412]
  We consider the risk of urban-rural disparities in identification of land-cover features. We propose fair dense representation with contrastive learning (FairDCL) as a method for de-biasing the multi-level latent space of convolution neural network models. The obtained image representation mitigates downstream urban-rural prediction disparities and outperforms state-of-the-art baselines on real-world satellite images.
  arXiv  Detail & Related papers  (2022-11-16T04:59:46Z)
• A domain adaptive deep learning solution for scanpath prediction of paintings [66.46953851227454]
  This paper focuses on the eye-movement analysis of viewers during the visual experience of a certain number of paintings. We introduce a new approach to predicting human visual attention, which impacts several cognitive functions for humans. The proposed new architecture ingests images and returns scanpaths, a sequence of points featuring a high likelihood of catching viewers' attention.
  arXiv  Detail & Related papers  (2022-09-22T22:27:08Z)
• Compositional Scene Representation Learning via Reconstruction: A Survey [48.33349317481124]
  Compositional scene representation learning is a task that enables such abilities. Deep neural networks have been proven to be advantageous in representation learning. Learning via reconstruction is advantageous because it may utilize massive unlabeled data and avoid costly and laborious data annotation.
  arXiv  Detail & Related papers  (2022-02-15T02:14:05Z)
• Environment Predictive Coding for Embodied Agents [92.31905063609082]
  We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. Our experiments on the photorealistic 3D environments of Gibson and Matterport3D show that our method outperforms the state-of-the-art on challenging tasks with only a limited budget of experience.
  arXiv  Detail & Related papers  (2021-02-03T23:43:16Z)
• VisualEchoes: Spatial Image Representation Learning through Echolocation [97.23789910400387]
  Several animal species (e.g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation. We propose a novel interaction-based representation learning framework that learns useful visual features via echolocation. Our work opens a new path for representation learning for embodied agents, where supervision comes from interacting with the physical world.
  arXiv  Detail & Related papers  (2020-05-04T16:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.