Variational Connectionist Temporal Classification for Order-Preserving Sequence Modeling

• URL: http://arxiv.org/abs/2309.11983v3
• Date: Fri, 15 Dec 2023 00:35:30 GMT
• Title: Variational Connectionist Temporal Classification for Order-Preserving Sequence Modeling
• Abstract summary: We derive loss functions that can be used to train more generalizable sequence models that preserve order. We show that both loss functions allow direct optimization of the variational lower bound for the model log-likelihood.
• Score: 11.312044841380649
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Connectionist temporal classification (CTC) is commonly adopted for sequence modeling tasks like speech recognition, where it is necessary to preserve order between the input and target sequences. However, CTC is only applied to deterministic sequence models, where the latent space is discontinuous and sparse, which in turn makes them less capable of handling data variability when compared to variational models. In this paper, we integrate CTC with a variational model and derive loss functions that can be used to train more generalizable sequence models that preserve order. Specifically, we derive two versions of the novel variational CTC based on two reasonable assumptions, the first being that the variational latent variables at each time step are conditionally independent; and the second being that these latent variables are Markovian. We show that both loss functions allow direct optimization of the variational lower bound for the model log-likelihood, and present computationally tractable forms for implementing them.

Related papers

• Guarantees on Dynamical System Distinguishability for LLM Token Generation [3.6209977238182276]
  We formalize the classification task as a binary hypothesis test between two linear DSs.<n>We show that the total variation distance between the stationary marginal distributions of the two DSs can be arbitrarily small.<n>We also characterize cross-embedding generalization by introducing an approximate intertwining condition between embedding models.
  arXiv  Detail & Related papers  (2026-07-25T04:10:49Z)
• Permutation-Equivariant 2D State Space Models: Theory and Canonical Architecture for Multivariate Time Series [16.396187082494958]
  We formalize a violation of the permutation symmetry principle and require state-space dynamics to be permutation-equivariant along the variable axis.<n>We introduce the Variable-Invariant Two-Dimensional State Space Model (VI 2D SSM), which realizes the canonical equivariant form via permutation-invariant aggregation.
  arXiv  Detail & Related papers  (2026-03-07T04:10:20Z)
• Sequential-Parallel Duality in Prefix Scannable Models [68.39855814099997]
  Recent developments have given rise to various models, such as Gated Linear Attention (GLA) and Mamba.<n>This raises a natural question: can we characterize the full class of neural sequence models that support near-constant-time parallel evaluation and linear-time, constant-space sequential inference?
  arXiv  Detail & Related papers  (2025-06-12T17:32:02Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Efficient Interpretable Nonlinear Modeling for Multiple Time Series [5.448070998907116]
  This paper proposes an efficient nonlinear modeling approach for multiple time series. It incorporates nonlinear interactions among different time-series variables. Experimental results show that the proposed algorithm improves the identification of the support of the VAR coefficients in a parsimonious manner.
  arXiv  Detail & Related papers  (2023-09-29T11:42:59Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)
• An Interpretable and Efficient Infinite-Order Vector Autoregressive Model for High-Dimensional Time Series [1.4939176102916187]
  This paper proposes a novel sparse infinite-order VAR model for high-dimensional time series. The temporal and cross-sectional structures of the VARMA-type dynamics captured by this model can be interpreted separately. Greater statistical efficiency and interpretability can be achieved with little loss of temporal information.
  arXiv  Detail & Related papers  (2022-09-02T17:14:24Z)
• Dynamically-Scaled Deep Canonical Correlation Analysis [77.34726150561087]
  Canonical Correlation Analysis (CCA) is a method for feature extraction of two views by finding maximally correlated linear projections of them. We introduce a novel dynamic scaling method for training an input-dependent canonical correlation model.
  arXiv  Detail & Related papers  (2022-03-23T12:52:49Z)
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder [59.69303945834122]
  We present a Smoothness-Inducing Sequential Variational Auto-Encoder (SISVAE) model for robust estimation and anomaly detection of time series. Our model parameterizes mean and variance for each time-stamp with flexible neural networks. We show the effectiveness of our model on both synthetic datasets and public real-world benchmarks.
  arXiv  Detail & Related papers  (2021-02-02T06:15:15Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)
• Differentiable Segmentation of Sequences [2.1485350418225244]
  We build on advances in learning continuous warping functions and propose a novel family of warping functions based on the two-sided power (TSP) distribution. Our formulation includes the important class of segmented generalized linear models as a special case. We use our approach to model the spread of COVID-19 with Poisson regression, apply it on a change point detection task, and learn classification models with concept drift.
  arXiv  Detail & Related papers  (2020-06-23T15:51:48Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.