Pre-trained Transformer Uncovers Meaningful Patterns in Human Mobility Data

• URL: http://arxiv.org/abs/2406.04029v1
• Date: Thu, 6 Jun 2024 12:59:46 GMT
• Title: Pre-trained Transformer Uncovers Meaningful Patterns in Human Mobility Data
• Authors: Alameen Najjar,
• Abstract summary: We show that a transformer pre-trained on country-scale unlabeled human mobility data learns embeddings capable of developing a deep understanding of the target geography. We evaluate the performance of our pre-trained embeddings in encapsulating a broad spectrum of concepts related to human mobility.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We empirically demonstrate that a transformer pre-trained on country-scale unlabeled human mobility data learns embeddings capable, through fine-tuning, of developing a deep understanding of the target geography and its corresponding mobility patterns. Utilizing an adaptation framework, we evaluate the performance of our pre-trained embeddings in encapsulating a broad spectrum of concepts directly and indirectly related to human mobility. This includes basic notions, such as geographic location and distance, and extends to more complex constructs, such as administrative divisions and land cover. Our extensive empirical analysis reveals a substantial performance boost gained from pre-training, reaching up to 38% in tasks such as tree-cover regression. We attribute this result to the ability of the pre-training to uncover meaningful patterns hidden in the raw data, beneficial for modeling relevant high-level concepts. The pre-trained embeddings emerge as robust representations of regions and trajectories, potentially valuable for a wide range of downstream applications.

Related papers

• Features are fate: a theory of transfer learning in high-dimensional regression [23.840251319669907]
  We show that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch. For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance.
  arXiv  Detail & Related papers  (2024-10-10T17:58:26Z)
• Efficient Transferability Assessment for Selection of Pre-trained Detectors [63.21514888618542]
  This paper studies the efficient transferability assessment of pre-trained object detectors. We build up a detector transferability benchmark which contains a large and diverse zoo of pre-trained detectors. Experimental results demonstrate that our method outperforms other state-of-the-art approaches in assessing transferability.
  arXiv  Detail & Related papers  (2024-03-14T14:23:23Z)
• Which Model to Transfer? A Survey on Transferability Estimation [43.93666569439436]
  Transfer learning methods endeavor to leverage relevant knowledge from existing source pre-trained models or datasets to solve downstream target tasks. Model transferability estimation is an emerging and growing area of interest, aiming to propose a metric to quantify this suitability without training them individually. We present the first review of existing advances in this area and categorize them into two separate realms: source-free model transferability estimation and source-dependent model transferability estimation.
  arXiv  Detail & Related papers  (2024-02-23T09:47:27Z)
• SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations [76.45009891152178]
  Pretraining-finetuning approach can alleviate the labeling burden by fine-tuning a pre-trained backbone across various downstream datasets as well as tasks. We show, for the first time, that general representations learning can be achieved through the task of occupancy prediction. Our findings will facilitate the understanding of LiDAR points and pave the way for future advancements in LiDAR pre-training.
  arXiv  Detail & Related papers  (2023-09-19T11:13:01Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation [91.62129090006745]
  This paper studies the distribution shift problem from the perspective of pre-training and data augmentation. We provide the first comprehensive empirical study focusing on pre-training and data augmentation.
  arXiv  Detail & Related papers  (2022-05-25T13:04:53Z)
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data [96.23611272637943]
  We propose a self-supervised approach for pre-training deep neural networks in remote sensing. By exploiting the correspondence between geo-tagged audio recordings and remote sensing, this is done in a completely label-free manner. We show that our approach outperforms existing pre-training strategies for remote sensing imagery.
  arXiv  Detail & Related papers  (2021-08-02T07:50:50Z)
• Adversarial Training Helps Transfer Learning via Better Representations [17.497590668804055]
  Transfer learning aims to leverage models pre-trained on source data to efficiently adapt to target setting. Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains. We show that adversarial training in the source data generates provably better representations, so fine-tuning on top of this representation leads to a more accurate predictor of the target data.
  arXiv  Detail & Related papers  (2021-06-18T15:41:07Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.