Stochastic Clock Attention for Aligning Continuous and Ordered Sequences

• URL: http://arxiv.org/abs/2509.14678v1
• Date: Thu, 18 Sep 2025 07:18:34 GMT
• Title: Stochastic Clock Attention for Aligning Continuous and Ordered Sequences
• Abstract summary: We formulate an attention mechanism for continuous and ordered sequences that functions as an alignment model.<n>In a Transformer text-to-speech testbed, this construction produces more stable alignments and improved robustness to global time-scaling.
• Score: 1.2418532541734193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We formulate an attention mechanism for continuous and ordered sequences that explicitly functions as an alignment model, which serves as the core of many sequence-to-sequence tasks. Standard scaled dot-product attention relies on positional encodings and masks but does not enforce continuity or monotonicity, which are crucial for frame-synchronous targets. We propose learned nonnegative \emph{clocks} to source and target and model attention as the meeting probability of these clocks; a path-integral derivation yields a closed-form, Gaussian-like scoring rule with an intrinsic bias toward causal, smooth, near-diagonal alignments, without external positional regularizers. The framework supports two complementary regimes: normalized clocks for parallel decoding when a global length is available, and unnormalized clocks for autoregressive decoding -- both nearly-parameter-free, drop-in replacements. In a Transformer text-to-speech testbed, this construction produces more stable alignments and improved robustness to global time-scaling while matching or improving accuracy over scaled dot-product baselines. We hypothesize applicability to other continuous targets, including video and temporal signal modeling.

Related papers

• Liquid Gated Attention [12.636376869841826]
  Liquid Gated Attention (LGA) is a solver-free parallel temporal operator.<n>Building on LGA, we instantiate LFormer, a modular backbone for continuous-time representation learning.
  arXiv  Detail & Related papers  (2026-08-31T12:30:09Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• DeCoDrift: Stabilizing Decoder Coupling in Closed-Loop Foundation Segmentation [0.431493144418712]
  We show that feedback loop can induce a previously overlooked failure mode, decoder coupling drift, in which the mask decoder's cross-attention progressively loses alignment with the target object, causing errors to accumulate across iterations.<n>We introduce DeCoDrift, a training-free inference-time stabilization framework that constrains prompt updates and preserves decoder coupling across iterations.
  arXiv  Detail & Related papers  (2026-05-25T11:41:21Z)
• Beyond Similarity: Temporal Operator Attention for Time Series Analysis [5.084091872064936]
  A persistent paradox in time-series forecasting is that structurally simple and linear models often outperform high-capacity Transformers.<n>We propose $textbf Operator Attention (TOA)$, a framework that augments attention with explicit learnable sequence-space operators.<n>To make dense $N times operators practical, we introduce a high-variance dropout mechanism that stabilizes training and prevents trivial memorization.
  arXiv  Detail & Related papers  (2026-05-11T22:13:53Z)
• TIE: Time Interval Encoding for Video Generation over Events [50.66585165263848]
  Director-style prompting, robotic action prediction, and interactive video agents demand temporal grounding over concurrent events.<n>Modern video generators represent time as discrete points through point-wise positional encodings.<n>Time Interval TIE is a principled, plug-and-play interval-aware generalization.
  arXiv  Detail & Related papers  (2026-05-11T13:23:14Z)
• From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation [54.68405211129937]
  Exo-Seqto-Ego video generation aims to synthesize a first-person view from a synchronized third-person and corresponding camera poses.<n> synchronized exo-ego data inherently introduces substantial-temporal discontinuities, violating the smooth-motion assumptions of standard video generation benchmarks.<n>We propose Syn2Ex-Forcing, a sequential sequence model that interpolates between the source and target videos to form a single continuous signal.
  arXiv  Detail & Related papers  (2026-04-15T12:32:25Z)
• FuXi-Linear: Unleashing the Power of Linear Attention in Long-term Time-aware Sequential Recommendation [86.55349738440087]
  FuXi-Linear is a linear-complexity model designed for efficient long-sequence recommendation.<n>Our approach introduces two key components: (1) a Temporal Retention Channel that independently computes periodic attention weights using temporal data, preventing crosstalk between temporal and semantic signals; and (2) a Linear Positional Channel that integrates positional information through learnable kernels within linear complexity.
  arXiv  Detail & Related papers  (2026-02-27T04:38:28Z)
• TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs [8.818252253980985]
  TempoNet is a reinforcement learning scheduler that pairs a permutation-invariant Transformer with a deep Q-approximation.<n>A latency-aware sparse attention stack with blockwise top-k selection and locality-sensitive chunking enables global reasoning over unordered task sets.
  arXiv  Detail & Related papers  (2026-02-20T09:56:23Z)
• Selective Synchronization Attention [0.0]
  Transformer architecture has become the foundation of modern deep learning, but its core self-attention mechanism lacks grounding in biological neural computation.<n>We propose Selective Synchronization Attention (SSA), a novel attention mechanism that replaces the standard dotproduct self-attention with a closed-form operator.<n>SSA provides three key advantages: (i) natural sparsity from the phase-locking threshold, whereby tokens with frequencies automatically receive zero attention weight without explicit masking; (ii) positional-semantic encoding through the natural frequency spectrum, eliminating the need for separate positional encodings; and (iii) a single-pass, closed-
  arXiv  Detail & Related papers  (2026-02-16T03:58:12Z)
• Oscillators Are All You Need: Irregular Time Series Modelling via Damped Harmonic Oscillators with Closed-Form Solutions [3.8401108926879495]
  Transformers excel at time series modelling through attention mechanisms that capture long-term temporal patterns.<n>They assume uniform time intervals and therefore struggle with irregular time series.<n>We show how to handle irregular time series by modelling hidden states as continuously evolving trajectories.
  arXiv  Detail & Related papers  (2026-02-12T16:27:09Z)
• Forecasting as Rendering: A 2D Gaussian Splatting Framework for Time Series Forecasting [79.37674445572462]
  Time series forecasting (TSF) remains a challenging problem due to the intricate entanglement of intraperiod-fluctuations and interperiod-trends.<n>Treating reshaped tensors as static images results in a topological mismatch.<n> relying on uniform fixed-size representations allocates modeling capacity inefficiently.<n>TimeGS is a novel framework that fundamentally shifts the forecasting paradigm from regression to 2D generative rendering.
  arXiv  Detail & Related papers  (2026-02-10T14:13:36Z)
• LEFT: Learnable Fusion of Tri-view Tokens for Unsupervised Time Series Anomaly Detection [53.191369031661885]
  Unsupervised time series anomaly detection aims to build a model for identifying abnormal timestamps without assuming the availability of annotations.<n>We present Learnable Fusion of Tri-view Tokens (LEFT), a unified unsupervised TSAD framework that models anomalies as inconsistencies across complementary representations.<n>Experiments on real-world benchmarks show that LEFT yields the best detection accuracy against SOTA baselines, while achieving a 5x reduction on FLOPs and 8x speed-up for training.
  arXiv  Detail & Related papers  (2026-02-09T13:33:49Z)
• Adaptive Path Integral Diffusion: AdaPID [0.0]
  We develop a path-wise schedule for selection gramework for Harmonic PID with a time-varying stiffness.<n>We introduce schedule-sensitive Quality-of-Sampling (QoS) diagnostics.
  arXiv  Detail & Related papers  (2025-12-05T04:57:00Z)
• PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs [57.790910044227935]
  Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames.<n>We present Phase Aggregated Smoothing (PAS), a training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs.<n>Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling.
  arXiv  Detail & Related papers  (2025-11-14T05:56:47Z)
• Higher-order Linear Attention [59.92962330635185]
  quadratic cost of scaled dot-product attention is a central obstacle to scaling autoregressive language models to long contexts.<n>We introduce Higher-order Linear Attention (HLA), a causal, streaming mechanism that realizes higher interactions via compact prefix sufficient statistics.
  arXiv  Detail & Related papers  (2025-10-31T07:54:37Z)
• Timer-XL: Long-Context Transformers for Unified Time Series Forecasting [67.83502953961505]
  We present Timer-XL, a causal Transformer for unified time series forecasting.<n>Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-10-07T07:27:39Z)
• Oscillatory State-Space Models [61.923849241099184]
  We propose Lineary State-Space models (LinOSS) for efficiently learning on long sequences.<n>A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model.<n>We show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions.
  arXiv  Detail & Related papers  (2024-10-04T22:00:13Z)
• Continuous-Time Linear Positional Embedding for Irregular Time Series Forecasting [14.038703353884033]
  We propose CTLPE, a method learning a continuous linear function for encoding temporal information. CTLPE outperforms existing techniques across various irregularly-sampled time series datasets.
  arXiv  Detail & Related papers  (2024-09-30T08:46:18Z)
• Latte: Latent Attention for Linear Time Transformers [11.524573224123905]
  We propose a probabilistic framework for attention. Our method can be seamlessly integrated as a drop-in replacement for the standard attention mechanism. The resulting Latte Transformer'' achieves performance comparable to standard attention and other state-of-the-art models.
  arXiv  Detail & Related papers  (2024-02-27T13:54:48Z)
• Imputing Missing Observations with Time Sliced Synthetic Minority Oversampling Technique [0.3973560285628012]
  We present a simple yet novel time series imputation technique with the goal of constructing an irregular time series that is uniform across every sample in a data set. We fix a grid defined by the midpoints of non-overlapping bins (dubbed "slices") of observation times and ensure that each sample has values for all of the features at that given time. This allows one to both impute fully missing observations to allow uniform time series classification across the entire data and, in special cases, to impute individually missing features.
  arXiv  Detail & Related papers  (2022-01-14T19:23:24Z)
• Spectral embedding for dynamic networks with stability guarantees [7.575860121202432]
  We consider the problem of embedding a dynamic network vector to obtain time-evolving representations of each node. These properties are defined formally within a generic dynamic latent position model. We show how this model can be recast as a multilayer random dot product graph.
  arXiv  Detail & Related papers  (2021-06-02T16:43:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.