CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

• URL: http://arxiv.org/abs/2405.11039v3
• Date: Thu, 29 Aug 2024 16:57:38 GMT
• Title: CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl
• Authors: Ilya Ilyankou, Meihui Wang, Stefano Cavazzi, James Haworth,
• Abstract summary: The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. In this paper, we introduce an efficient pipeline to extract annotated user-generated tracks from GPX files found in CC. The resulting multimodal dataset includes 1,416 pairings of human-written descriptions and MultiLineString vector data from the 6 most recent CC releases.
• Score: 0.07499722271664144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content, and distilled for smaller, domain-specific datasets. However, to our knowledge, no research has been dedicated to using CC as a source of annotated geospatial data. In this paper, we introduce an efficient pipeline to extract annotated user-generated tracks from GPX files found in CC, and the resulting multimodal dataset with 1,416 pairings of human-written descriptions and MultiLineString vector data from the 6 most recent CC releases. The dataset can be used to study people's outdoor activity patterns, the way people talk about their outdoor experiences, as well as for developing trajectory generation or track annotation models, or for various other problems in place of synthetically generated routes. Our reproducible code is available on GitHub: https://github.com/ilyankou/cc-gpx

Related papers

• CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training [63.07024608399447]
  We propose an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting. We introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and ClimbMix, a compact yet powerful 400-billion-token dataset.
  arXiv  Detail & Related papers  (2025-04-17T17:58:13Z)
• SEART Data Hub: Streamlining Large-Scale Source Code Mining and Pre-Processing [13.717170962455526]
  We present the SEART Data Hub, a web application that allows to easily build and pre-process large-scale datasets featuring code mined from public GitHub repositories. Through a simple web interface, researchers can specify a set of mining criteria as well as specific pre-processing steps they want to perform. After submitting the request, the user will receive an email with a download link for the required dataset within a few hours.
  arXiv  Detail & Related papers  (2024-09-27T11:42:19Z)
• Quantifying Geospatial in the Common Crawl Corpus [0.07499722271664144]
  This paper investigates the prevalence of geospatial data in recent Common Crawl releases using Gemini 1.5, a powerful language model. We estimate that 18.7% of web documents in CC contain geospatial information such as coordinates and addresses.
  arXiv  Detail & Related papers  (2024-06-07T14:16:37Z)
• UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction [93.77809355002591]
  We introduce UniTraj, a comprehensive framework that unifies various datasets, models, and evaluation criteria. We conduct extensive experiments and find that model performance significantly drops when transferred to other datasets. We provide insights into dataset characteristics to explain these findings.
  arXiv  Detail & Related papers  (2024-03-22T10:36:50Z)
• Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora [104.16648246740543]
  We propose an efficient data collection method based on large language models. The method bootstraps seed information through a large language model and retrieves related data from public corpora. It not only collects knowledge-related data for specific domains but unearths the data with potential reasoning procedures.
  arXiv  Detail & Related papers  (2024-01-26T03:38:23Z)
• DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models [61.906934570771256]
  We present a generic dataset generation model that can produce diverse synthetic images and perception annotations. Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module.
  arXiv  Detail & Related papers  (2023-08-11T14:38:11Z)
• trajdata: A Unified Interface to Multiple Human Trajectory Datasets [32.93180256927027]
  We present trajdata, a unified interface to multiple human trajectory datasets. Trajdata provides a simple, uniform, and efficient representation and API for trajectory and map data.
  arXiv  Detail & Related papers  (2023-07-26T02:45:59Z)
• GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition [31.194474203667042]
  GeoDE is a geographically diverse dataset with 61,940 images from 40 classes and 6 world regions. We release the full dataset and code at https://geodiverse-data-collection.cs.princeton.edu/.
  arXiv  Detail & Related papers  (2023-01-05T18:21:50Z)
• AutoGeoLabel: Automated Label Generation for Geospatial Machine Learning [69.47585818994959]
  We evaluate a big data processing pipeline to auto-generate labels for remote sensing data. We utilize the big geo-data platform IBM PAIRS to dynamically generate such labels in dense urban areas.
  arXiv  Detail & Related papers  (2022-01-31T20:02:22Z)
• Datasets: A Community Library for Natural Language Processing [55.48866401721244]
  datasets is a community library for contemporary NLP. The library includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects.
  arXiv  Detail & Related papers  (2021-09-07T03:59:22Z)
• Sketch and Scale: Geo-distributed tSNE and UMAP [75.44887265789056]
  Running machine learning analytics over geographically distributed datasets is a rapidly arising problem. We introduce a novel framework: Sketch and Scale (SnS) It leverages a Count Sketch data structure to compress the data on the edge nodes, aggregates the reduced size sketches on the master node, and runs vanilla tSNE or UMAP on the summary. We show this technique to be fully parallel, scale linearly in time, logarithmically in memory, and communication, making it possible to analyze datasets with many millions, potentially billions of data points, spread across several data centers around the globe.
  arXiv  Detail & Related papers  (2020-11-11T22:32:21Z)
• A Large Dataset of Historical Japanese Documents with Complex Layouts [5.343406649012619]
  HJDataset is a large dataset of historical Japanese documents with complex layouts. It contains over 250,000 layout element annotations seven types. A semi-rule based method is developed to extract the layout elements, and the results are checked by human inspectors.
  arXiv  Detail & Related papers  (2020-04-18T18:38:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.