An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps

• URL: http://arxiv.org/abs/2410.15780v1
• Date: Mon, 21 Oct 2024 08:45:26 GMT
• Title: An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps
• Authors: Ziyi Liu, Claudio Affolter, Sidi Wu, Yizi Chen, Lorenz Hurni,
• Abstract summary: Historical maps provide valuable information and knowledge about the past. As they often feature non-standard projections, hand-drawn styles, and artistic elements, it is challenging for non-experts to identify and interpret them. Existing image captioning methods have achieved remarkable success on natural images, their performance on maps is suboptimal as maps are underrepresented in their pre-training process. Despite the recent advance of GPT-4 in text recognition and map captioning, it still has a limited understanding of maps, as its performance wanes when texts in maps are missing or inaccurate. We propose a novel decision tree architecture to only generate captions relevant
• Score: 11.037615422309296
• License:
• Abstract: Historical maps provide valuable information and knowledge about the past. However, as they often feature non-standard projections, hand-drawn styles, and artistic elements, it is challenging for non-experts to identify and interpret them. While existing image captioning methods have achieved remarkable success on natural images, their performance on maps is suboptimal as maps are underrepresented in their pre-training process. Despite the recent advance of GPT-4 in text recognition and map captioning, it still has a limited understanding of maps, as its performance wanes when texts (e.g., titles and legends) in maps are missing or inaccurate. Besides, it is inefficient or even impractical to fine-tune the model with users' own datasets. To address these problems, we propose a novel and lightweight map-captioning counterpart. Specifically, we fine-tune the state-of-the-art vision-language model CLIP to generate captions relevant to historical maps and enrich the captions with GPT-3.5 to tell a brief story regarding where, what, when and why of a given map. We propose a novel decision tree architecture to only generate captions relevant to the specified map type. Our system shows invariance to text alterations in maps. The system can be easily adapted and extended to other map types and scaled to a larger map captioning system. The code is open-sourced at https://github.com/claudaff/automatic-map-storytelling.

Related papers

• Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models [15.454856838083511]
  Large Language Models (LLM) have emerged as a tool for robots to generate task plans using common sense reasoning. Recent works have shifted from explicit maps with fixed semantic classes to implicit open vocabulary maps. We propose an explicit text-based map that can represent thousands of semantic classes while easily integrating with LLMs.
  arXiv  Detail & Related papers  (2024-09-23T18:26:19Z)
• Neural Semantic Surface Maps [52.61017226479506]
  We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement.
  arXiv  Detail & Related papers  (2023-09-09T16:21:56Z)
• Sat2Cap: Mapping Fine-Grained Textual Descriptions from Satellite Images [12.356676398446215]
  We propose a weakly supervised approach for creating maps using free-form textual descriptions. We train a contrastive learning framework called Sat2Cap on a new large-scale dataset with 6.1M pairs of overhead and ground-level images.
  arXiv  Detail & Related papers  (2023-07-29T06:23:51Z)
• The mapKurator System: A Complete Pipeline for Extracting and Linking Text from Historical Maps [7.209761597734092]
  mapKurator is an end-to-end system integrating machine learning models with a comprehensive data processing pipeline. We deployed the mapKurator system and enabled the processing of over 60,000 maps and over 100 million text/place names in the David Rumsey Historical Map collection.
  arXiv  Detail & Related papers  (2023-06-29T16:05:40Z)
• SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding [57.108301842535894]
  We introduce SNAP, a deep network that learns rich neural 2D maps from ground-level and overhead images. We train our model to align neural maps estimated from different inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods.
  arXiv  Detail & Related papers  (2023-06-08T17:54:47Z)
• Long-term Visual Map Sparsification with Heterogeneous GNN [47.12309045366042]
  In this paper, we aim to overcome the environmental changes and reduce the map size at the same time by selecting points that are valuable to future localization. Inspired by the recent progress in Graph Neural Network(GNN), we propose the first work that models SfM maps as heterogeneous graphs and predicts 3D point importance scores with a GNN. Two novel supervisions are proposed: 1) a data-fitting term for selecting valuable points to future localization based on training queries; 2) a K-Cover term for selecting sparse points with full map coverage.
  arXiv  Detail & Related papers  (2022-03-29T01:46:12Z)
• Synthetic Map Generation to Provide Unlimited Training Data for Historical Map Text Detection [5.872532529455414]
  We propose a method to automatically generate an unlimited amount of annotated historical map images for training text detection models. We show that the state-of-the-art text detection models can benefit from the synthetic historical maps.
  arXiv  Detail & Related papers  (2021-12-12T00:27:03Z)
• An Automatic Approach for Generating Rich, Linked Geo-Metadata from Historical Map Images [6.962949867017594]
  This paper presents an end-to-end approach to address the real-world problem of finding and indexing historical map images. We have implemented the approach in a system called mapKurator.
  arXiv  Detail & Related papers  (2021-12-03T01:44:38Z)
• Semantic Image Alignment for Vehicle Localization [111.59616433224662]
  We present a novel approach to vehicle localization in dense semantic maps using semantic segmentation from a monocular camera. In contrast to existing visual localization approaches, the system does not require additional keypoint features, handcrafted localization landmark extractors or expensive LiDAR sensors.
  arXiv  Detail & Related papers  (2021-10-08T14:40:15Z)
• HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps [81.86923212296863]
  HD maps are maps with precise definitions of road lanes with rich semantics of the traffic rules. There are only a small amount of real-world road topologies and geometries, which significantly limits our ability to test out the self-driving stack. We propose HDMapGen, a hierarchical graph generation model capable of producing high-quality and diverse HD maps.
  arXiv  Detail & Related papers  (2021-06-28T17:59:30Z)
• Cross-Descriptor Visual Localization and Mapping [81.16435356103133]
  Visual localization and mapping is the key technology underlying the majority of Mixed Reality and robotics systems. We present three novel scenarios for localization and mapping which require the continuous update of feature representations. Our data-driven approach is agnostic to the feature descriptor type, has low computational requirements, and scales linearly with the number of description algorithms.
  arXiv  Detail & Related papers  (2020-12-02T18:19:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.