CoLES: Contrastive Learning for Event Sequences with Self-Supervision
- URL: http://arxiv.org/abs/2002.08232v3
- Date: Fri, 22 Jul 2022 09:45:55 GMT
- Title: CoLES: Contrastive Learning for Event Sequences with Self-Supervision
- Authors: Dmitrii Babaev, Ivan Kireev, Nikita Ovsov, Mariya Ivanova, Gleb Gusev,
Ivan Nazarov, Alexander Tuzhilin
- Abstract summary: We address the problem of self-supervised learning on discrete event sequences generated by real-world users.
We propose a new method "CoLES", which adapts contrastive learning, previously used for audio and computer vision domains.
- Score: 63.3568071938238
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We address the problem of self-supervised learning on discrete event
sequences generated by real-world users. Self-supervised learning incorporates
complex information from the raw data in low-dimensional fixed-length vector
representations that could be easily applied in various downstream machine
learning tasks. In this paper, we propose a new method "CoLES", which adapts
contrastive learning, previously used for audio and computer vision domains, to
the discrete event sequences domain in a self-supervised setting. We deployed
CoLES embeddings based on sequences of transactions at the large European
financial services company. Usage of CoLES embeddings significantly improves
the performance of the pre-existing models on downstream tasks and produces
significant financial gains, measured in hundreds of millions of dollars
yearly. We also evaluated CoLES on several public event sequences datasets and
showed that CoLES representations consistently outperform other methods on
different downstream tasks.
Related papers
- Data-CUBE: Data Curriculum for Instruction-based Sentence Representation
Learning [85.66907881270785]
We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training.
In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk.
In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
arXiv Detail & Related papers (2024-01-07T18:12:20Z) - In-Domain Self-Supervised Learning Improves Remote Sensing Image Scene
Classification [5.323049242720532]
Self-supervised learning has emerged as a promising approach for remote sensing image classification.
We present a study of different self-supervised pre-training strategies and evaluate their effect across 14 downstream datasets.
arXiv Detail & Related papers (2023-07-04T10:57:52Z) - ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
We introduce Action-Aware Embodied Learning for Perception (ALP)
ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective.
We show that ALP outperforms existing baselines in several downstream perception tasks.
arXiv Detail & Related papers (2023-06-16T21:51:04Z) - Task Agnostic Representation Consolidation: a Self-supervised based
Continual Learning Approach [14.674494335647841]
We propose a two-stage training paradigm for CL that intertwines task-agnostic and task-specific learning.
We show that our training paradigm can be easily added to memory- or regularization-based approaches.
arXiv Detail & Related papers (2022-07-13T15:16:51Z) - Beyond Transfer Learning: Co-finetuning for Action Localisation [64.07196901012153]
We propose co-finetuning -- simultaneously training a single model on multiple upstream'' and downstream'' tasks.
We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data.
We also show how we can easily extend our approach to multiple upstream'' datasets to further improve performance.
arXiv Detail & Related papers (2022-07-08T10:25:47Z) - Beyond Just Vision: A Review on Self-Supervised Representation Learning
on Multimodal and Temporal Data [10.006890915441987]
Popularity of self-supervised learning is driven by the fact that traditional models typically require a huge amount of well-annotated data for training.
Self-supervised methods have been introduced to improve the efficiency of training data through discriminative pre-training of models.
We aim to provide the first comprehensive review of multimodal self-supervised learning methods for temporal data.
arXiv Detail & Related papers (2022-06-06T04:59:44Z) - Multi-Augmentation for Efficient Visual Representation Learning for
Self-supervised Pre-training [1.3733988835863333]
We propose Multi-Augmentations for Self-Supervised Learning (MA-SSRL), which fully searched for various augmentation policies to build the entire pipeline.
MA-SSRL successfully learns the invariant feature representation and presents an efficient, effective, and adaptable data augmentation pipeline for self-supervised pre-training.
arXiv Detail & Related papers (2022-05-24T04:18:39Z) - ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for
Semi-supervised Continual Learning [52.831894583501395]
Continual learning assumes the incoming data are fully labeled, which might not be applicable in real applications.
We propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN)
We show ORDisCo achieves significant performance improvement on various semi-supervised learning benchmark datasets for SSCL.
arXiv Detail & Related papers (2021-01-02T09:04:14Z) - Few-Shot Unsupervised Continual Learning through Meta-Examples [21.954394608030388]
We introduce a novel and complex setting involving unsupervised meta-continual learning with unbalanced tasks.
We exploit a meta-learning scheme that simultaneously alleviates catastrophic forgetting and favors the generalization to new tasks.
Experimental results on few-shot learning benchmarks show competitive performance even compared to the supervised case.
arXiv Detail & Related papers (2020-09-17T07:02:07Z) - Learning to Count in the Crowd from Limited Labeled Data [109.2954525909007]
We focus on reducing the annotation efforts by learning to count in the crowd from limited number of labeled samples.
Specifically, we propose a Gaussian Process-based iterative learning mechanism that involves estimation of pseudo-ground truth for the unlabeled data.
arXiv Detail & Related papers (2020-07-07T04:17:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.