Learning the Koopman Operator using Attention Free Transformers

• URL: http://arxiv.org/abs/2606.23957v1
• Date: Mon, 22 Jun 2026 21:36:55 GMT
• Title: Learning the Koopman Operator using Attention Free Transformers
• Abstract summary: We introduce two complementary components that make Koopman predictors more robust.<n>First, we add an attention-free latent memory (AFT) block that aggregates a short window of past latents to produce a corrected latent before each Koopman update.<n>Second, we propose dynamic re-encoding: lightweight, online change-point triggers that detect latent drift and project predictions back onto the autoencoder manifold.
• Score: 0.3848364262836075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning Koopman operators with autoencoders enables linear prediction in a latent space, but long-horizon rollouts often drift off the learned manifold, leading to phase and amplitude errors on systems with switching, continuous spectra, or strong transients. We introduce two complementary components that make Koopman predictors more robust. First, we add an attention-free latent memory (AFT) block that aggregates a short window of past latents to produce a corrected latent before each Koopman update. Unlike multi-head attention, AFT operates in linear time and adds only $\approx$30k parameters ($3d^2 + T^2$, fewer than matched multi-head attention), yet captures the local temporal context needed to suppress error divergence. Second, we propose dynamic re-encoding: lightweight, online change-point triggers (EWMA, CUSUM, and sequential two-sample tests) that detect latent drift and project predictions back onto the autoencoder manifold. Across three benchmark systems -- Duffing oscillator, Repressilator, IRMA -- our model consistently reduces error accumulation compared to a Koopman autoencoder and matched-capacity multi-head attention. We also compare against GRU and Transformer autoencoders, evaluated both from initial conditions and with a 50-step context, and find that Koopman+AFT (with optional re-encoding) attains markedly lower long-horizon error while maintaining lower inference latency. We report improvements over horizons up to 1000 steps, together with ablations over trigger policies. The result is a fast, compact predictor that stays on the learned manifold over long horizons.

Related papers

• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• Precision Tracked Transformer via Kalman Filtering, Kriging and Process Noise [11.579218102536762]
  The Transformer is the foundational building block of modern AI, yet offers no principled handling of emphuncertainty<n>We show this is a case of our emphBayesian Filtering Transformer (BFT)<n>BFT replaces any Transformer layer with negligible overhead.
  arXiv  Detail & Related papers  (2026-05-12T22:22:15Z)
• CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling [46.16066322190728]
  We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture.<n>Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors.<n>To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism.
  arXiv  Detail & Related papers  (2026-04-05T20:13:22Z)
• KoopmanFlow: Spectrally Decoupled Generative Control Policy via Koopman Structural Bias [31.486395926173838]
  Generative Control Policies (GCPs) show immense promise in robotic manipulation but struggle to simultaneously model stable global motions and high-frequency local corrections.<n>We introduce KoopmanFlow, a parameter-efficient generative policy guided by a Koopman-inspired structural inductive bias.
  arXiv  Detail & Related papers  (2026-03-14T06:23:07Z)
• Surprised by Attention: Predictable Query Dynamics for Time Series Anomaly Detection [1.7751300245073598]
  AxonAD is an unsupervised detector that treats multi-head attention query evolution as a short horizon predictable process.<n>A tail-aggregated query mismatch score measures cosine deviation between predicted and target queries on recent timesteps.<n>On in-vehicle telemetry with interval annotations, AxonAD improves ranking quality and temporal localization over strong baselines.
  arXiv  Detail & Related papers  (2026-03-13T11:40:51Z)
• PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs [57.790910044227935]
  Video LLMs suffer from temporal inconsistency: small shifts in frame timing can flip attention and suppress relevant frames.<n>We present Phase Aggregated Smoothing (PAS), a training-free mechanism that applies small opposed phase offsets across heads and then aggregates their outputs.<n>Our analysis shows that the RoPE rotated logit can be approximated as a content dot product scaled by a time kernel; smoothing this kernel yields Lipschitz stability of attention to small temporal shifts; multi phase averaging attenuates high frequency ripples while preserving per-head spectra under Nyquist-valid sampling.
  arXiv  Detail & Related papers  (2025-11-14T05:56:47Z)
• Pi-Transformer: A Physics-informed Attention Mechanism for Time Series Anomaly Detection [3.8673630752805437]
  We present Pi-Transformer, a physics-informed transformer with two attention pathways.<n>The model combines an alignment-weighted reconstruction signal with a mismatch signal that highlights timing and phase disruptions.<n>Pi-Transformer achieves state-of-the-art or highly competitive F1, with particular strength on timing and phase-breaking anomalies.
  arXiv  Detail & Related papers  (2025-09-24T10:47:48Z)
• Diffusion Language Models Know the Answer Before Decoding [56.96815863705218]
  Diffusion language models (DLMs) have emerged as an alternative to autoregressive approaches.<n>Our work highlights and leverage an overlooked property of DLMs early answer convergence.<n>We introduce Prophet, a training-free fast decoding paradigm that enables early commit decoding.
  arXiv  Detail & Related papers  (2025-08-27T15:40:25Z)
• CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting [50.23240107430597]
  We design a special Transformer, i.e., Channel Aligned Robust Blend Transformer (CARD for short), that addresses key shortcomings of CI type Transformer in time series forecasting. First, CARD introduces a channel-aligned attention structure that allows it to capture both temporal correlations among signals. Second, in order to efficiently utilize the multi-scale knowledge, we design a token blend module to generate tokens with different resolutions. Third, we introduce a robust loss function for time series forecasting to alleviate the potential overfitting issue.
  arXiv  Detail & Related papers  (2023-05-20T05:16:31Z)
• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition [62.83832841523525]
  We propose a fast and accurate parallel transformer, termed Paraformer. It accurately predicts the number of output tokens and extract hidden variables. It can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
  arXiv  Detail & Related papers  (2022-06-16T17:24:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.