Aligned Contrastive Predictive Coding
- URL: http://arxiv.org/abs/2104.11946v1
- Date: Sat, 24 Apr 2021 13:07:22 GMT
- Title: Aligned Contrastive Predictive Coding
- Authors: Jan Chorowski, Grzegorz Ciesielski, Jaros{\l}aw Dzikowski, Adrian
{\L}ancucki, Ricard Marxer, Mateusz Opala, Piotr Pusz, Pawe{\l} Rychlikowski,
Micha{\l} Stypu{\l}kowski
- Abstract summary: We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations.
Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned.
- Score: 10.521845940927163
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We investigate the possibility of forcing a self-supervised model trained
using a contrastive predictive loss to extract slowly varying latent
representations. Rather than producing individual predictions for each of the
future representations, the model emits a sequence of predictions shorter than
that of the upcoming representations to which they will be aligned. In this
way, the prediction network solves a simpler task of predicting the next
symbols, but not their exact timing, while the encoding network is trained to
produce piece-wise constant latent codes. We evaluate the model on a speech
coding task and demonstrate that the proposed Aligned Contrastive Predictive
Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX
error rates, while being slightly faster to train due to the reduced number of
prediction heads.
Related papers
- Reinforced Decoder: Towards Training Recurrent Neural Networks for Time Series Forecasting [1.5213268724320657]
Recurrent neural network-based sequence-to-sequence models have been extensively applied for multi-step-ahead time series forecasting.
These models typically involve a decoder trained using either its previous forecasts or the actual observed values as the decoder inputs.
This study proposes a novel training approach called reinforced decoder, which introduces auxiliary models to generate alternative decoder inputs.
arXiv Detail & Related papers (2024-06-14T00:24:29Z) - TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction [61.295716741720284]
TokenUnify is a novel pretraining method that integrates random token prediction, next-token prediction, and next-all token prediction.
Cooperated with TokenUnify, we have assembled a large-scale electron microscopy (EM) image dataset with ultra-high resolution.
This dataset includes over 120 million annotated voxels, making it the largest neuron segmentation dataset to date.
arXiv Detail & Related papers (2024-05-27T05:45:51Z) - Conformal online model aggregation [29.43493007296859]
This paper proposes a new approach towards conformal model aggregation in online settings.
It is based on combining the prediction sets from several algorithms by voting, where weights on the models are adapted over time based on past performance.
arXiv Detail & Related papers (2024-03-22T15:40:06Z) - Non-autoregressive Sequence-to-Sequence Vision-Language Models [63.77614880533488]
We propose a parallel decoding sequence-to-sequence vision-language model that marginalizes over multiple inference paths in the decoder.
The model achieves performance on-par with its state-of-the-art autoregressive counterpart, but is faster at inference time.
arXiv Detail & Related papers (2024-03-04T17:34:59Z) - Explain, Adapt and Retrain: How to improve the accuracy of a PPM
classifier through different explanation styles [4.6281736192809575]
Recent papers have introduced a novel approach to explain why a Predictive Process Monitoring model for outcome-oriented predictions provides wrong predictions.
We show how to exploit the explanations to identify the most common features that induce a predictor to make mistakes in a semi-automated way.
arXiv Detail & Related papers (2023-03-27T06:37:55Z) - Debiased Fine-Tuning for Vision-language Models by Prompt Regularization [50.41984119504716]
We present a new paradigm for fine-tuning large-scale vision pre-trained models on downstream task, dubbed Prompt Regularization (ProReg)
ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning.
We show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
arXiv Detail & Related papers (2023-01-29T11:53:55Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Zero-Episode Few-Shot Contrastive Predictive Coding: Solving
intelligence tests without prior training [0.0]
We argue that finding a predictive latent variable and using it to evaluate the consistency of a future image enables data-efficient predictions.
We show that a one-dimensional Markov Contrastive Predictive Coding model solves sequence completion intelligence tests efficiently.
arXiv Detail & Related papers (2022-05-04T07:46:03Z) - Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
We show that pre-trained language models with few examples show strong prediction bias across labels.
Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features.
These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
arXiv Detail & Related papers (2022-04-17T15:55:18Z) - Hybrid Predictive Coding: Inferring, Fast and Slow [62.997667081978825]
We propose a hybrid predictive coding network that combines both iterative and amortized inference in a principled manner.
We demonstrate that our model is inherently sensitive to its uncertainty and adaptively balances balances to obtain accurate beliefs using minimum computational expense.
arXiv Detail & Related papers (2022-04-05T12:52:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.