CycleCL: Self-supervised Learning for Periodic Videos

• URL: http://arxiv.org/abs/2311.03402v2
• Date: Mon, 13 Nov 2023 13:09:49 GMT
• Title: CycleCL: Self-supervised Learning for Periodic Videos
• Authors: Matteo Destro, Michael Gygli
• Abstract summary: We propose CycleCL, a self-supervised learning method specifically designed to work with periodic data. We exploit the repetitions in videos to design a novel contrastive learning method based on a triplet loss. Our method uses pre-trained features to sample pairs of frames from approximately the same phase and negative pairs of frames from different phases.
• Score: 5.9647924003148365
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Analyzing periodic video sequences is a key topic in applications such as automatic production systems, remote sensing, medical applications, or physical training. An example is counting repetitions of a physical exercise. Due to the distinct characteristics of periodic data, self-supervised methods designed for standard image datasets do not capture changes relevant to the progression of the cycle and fail to ignore unrelated noise. They thus do not work well on periodic data. In this paper, we propose CycleCL, a self-supervised learning method specifically designed to work with periodic data. We start from the insight that a good visual representation for periodic data should be sensitive to the phase of a cycle, but be invariant to the exact repetition, i.e. it should generate identical representations for a specific phase throughout all repetitions. We exploit the repetitions in videos to design a novel contrastive learning method based on a triplet loss that optimizes for these desired properties. Our method uses pre-trained features to sample pairs of frames from approximately the same phase and negative pairs of frames from different phases. Then, we iterate between optimizing a feature encoder and resampling triplets, until convergence. By optimizing a model this way, we are able to learn features that have the mentioned desired properties. We evaluate CycleCL on an industrial and multiple human actions datasets, where it significantly outperforms previous video-based self-supervised learning methods on all tasks.

Related papers

• On the Regularization of Learnable Embeddings for Time Series Processing [18.069747511100132]
  We investigate methods to regularize the learning of local learnable embeddings for time series processing. We show that methods preventing the co-adaptation of local and global parameters are particularly effective in this context.
  arXiv  Detail & Related papers  (2024-10-18T17:30:20Z)
• Continual Learning for Multimodal Data Fusion of a Soft Gripper [1.0589208420411014]
  A model trained on one data modality often fails when tested with a different modality. We introduce a continual learning algorithm capable of incrementally learning different data modalities. We evaluate the algorithm's effectiveness on a challenging custom multimodal dataset.
  arXiv  Detail & Related papers  (2024-09-20T09:53:27Z)
• Motion Code: Robust Time series Classification and Forecasting via Sparse Variational Multi-Stochastic Processes Learning [3.2857981869020327]
  We introduce a novel framework that considers each time series as a sample realization of a continuous-time process. Such mathematical model explicitly captures the data dependence across several timestamps and detects the hidden time-dependent signals from noise. We then propose the abstract concept of the most informative timestamps to infer a sparse approximation of the individual dynamics based on their assigned vectors.
  arXiv  Detail & Related papers  (2024-02-21T19:10:08Z)
• Curricular and Cyclical Loss for Time Series Learning Strategy [17.725840333187577]
  We propose a novel Curricular and CyclicaL loss (CRUCIAL) to learn time series for the first time. CRUCIAL has two characteristics: It can arrange an easy-to-hard learning order and achieve an adaptive cycle. We prove that compared with monotonous size, cyclical size can reduce expected error.
  arXiv  Detail & Related papers  (2023-12-26T02:40:05Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• Imputing Missing Observations with Time Sliced Synthetic Minority Oversampling Technique [0.3973560285628012]
  We present a simple yet novel time series imputation technique with the goal of constructing an irregular time series that is uniform across every sample in a data set. We fix a grid defined by the midpoints of non-overlapping bins (dubbed "slices") of observation times and ensure that each sample has values for all of the features at that given time. This allows one to both impute fully missing observations to allow uniform time series classification across the entire data and, in special cases, to impute individually missing features.
  arXiv  Detail & Related papers  (2022-01-14T19:23:24Z)
• Learning summary features of time series for likelihood free inference [93.08098361687722]
  We present a data-driven strategy for automatically learning summary features from time series data. Our results indicate that learning summary features from data can compete and even outperform LFI methods based on hand-crafted values.
  arXiv  Detail & Related papers  (2020-12-04T19:21:37Z)
• Counting Out Time: Class Agnostic Video Repetition Counting in the Wild [82.26003709476848]
  We present an approach for estimating the period with which an action is repeated in a video. The crux of the approach lies in constraining the period prediction module to use temporal self-similarity. We train this model, called Repnet, with a synthetic dataset that is generated from a large unlabeled video collection.
  arXiv  Detail & Related papers  (2020-06-27T18:00:42Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning [116.24172387469994]
  We propose a higher-order LSTM model that can efficiently learn long-term correlations in the video sequence. This is accomplished through a novel tensor train module that performs prediction by combining convolutional features across time. Our results achieve state-of-the-art performance-art in a wide range of applications and datasets.
  arXiv  Detail & Related papers  (2020-02-21T05:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.