FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning

• URL: http://arxiv.org/abs/2605.04421v1
• Date: Wed, 06 May 2026 02:27:25 GMT
• Title: FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning
• Abstract summary: Continuous-time (CT) Transformers improve irregular and long-range modeling over CT-RNNs by exploiting inputs or outputs embeddings with continuous dynamics.<n>We propose FLUID, a CT Transformer that incorporates continuous dynamics directly into the attention computation by replacing it with Liquid Attention Network (LAN)<n>We evaluate FLUID on a broad set of learning tasks, including (i) irregular time-series, (ii) long-range modeling, (iii) lane-keeping control of autonomous vehicles, and (iv) learning physical dynamics under a scarce data regime.
• Score: 2.312232949770907
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continuous-time (CT) Transformers improve irregular and long-range modeling over CT-RNNs by exploiting inputs or outputs embeddings with continuous dynamics. However, the core scaled-dot-product-attention (SDPA) mechanism remains inherently discrete. We propose FLUID (Flexible Unified Information Dynamics), a CT Transformer that incorporates continuous dynamics directly into the attention computation by replacing it with Liquid Attention Network (LAN). LAN reinterprets attention logits as continuous dynamical system and reformulates them as the solution to a linear ODE modulated by input-dependent nonlinear recurrent gates. Theoretically, we establish stability guarantees for LAN dynamics and show that it serves as an interpolating middle ground between SDPA and CT-RNNs, recovering each as special case under well-defined parameterization of its gating functions. LAN also introduces an explicit attention-sink gate to eliminate disproportionate attention mass on uninformative nodes. FLUID replaces standard residual connections with input-dependent Liquid Hyper-Connections to adaptively regulate interlayer information flow. Empirically, we evaluate FLUID on a broad set of learning tasks, including (i) irregular time-series, (ii) long-range modeling, (iii) lane-keeping control of autonomous vehicles, and (iv) learning physical dynamics under a scarce data regime. Across all the tasks, FLUID consistently matches or outperforms CT baselines, achieving improvements of up to 47% in certain scenarios and enhancing generalization under distributional shifts. Additionally, FLUID demonstrates superior noise robustness and a self-correcting inductive bias in autonomous vehicle control. We also provide a detailed analysis of key hyperparameters to guide tuning and show that FLUID occupies an intermediate position among competing approaches in terms of runtime and memory efficiency.

Related papers

• LargeMonitor: Monitoring Online Task-Free Continual Learning via Large Pretrained Models [5.257217773590746]
  Online task-free continual learning (TFCL) requires intelligent agents to sequentially accumulate knowledge from an unbounded, non-stationary data stream.<n>Existing online TFCL paradigms rely on parameter-efficient prompt tuning or dynamic structure expansion driven by training-coupled optimization dynamics.<n>We propose LargeMonitor, a framework that leverages large pretrained foundation models to autonomously orchestrate task-free continuous adaptation.
  arXiv  Detail & Related papers  (2026-06-08T12:41:15Z)
• U-STS-LLM A Unified Spatio-Temporal Steered Large Language Model for Traffic Prediction and Imputation [6.194596351746244]
  We present U-STS-LLM, a unified framework built on a-temporal-ly steered LLM.<n>Our core innovation is a Dynamic S-Lo Concurrent Attention Bias Generator that synthesizes a persistent functional graph with transient nodal states.<n>Experiments on real-world cellular datasets demonstrate that U-STS-LLM establishes new state-of-the-art performance in both long-horizon forecasting and high-missing-rate imputation.
  arXiv  Detail & Related papers  (2026-05-12T08:16:22Z)
• Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective [10.517508846742682]
  Decentralized federated learning (FL) is a promising approach for training machine learning models on edge systems where no central server exists.<n>FedF-ADMM exploits the convexity of loss functionals within function space to derive alternating direction method multipliers.<n>Experiments show FedF-ADMM achieves faster and more stable convergence than existing decentralized FL methods.
  arXiv  Detail & Related papers  (2026-05-10T06:11:46Z)
• Drift-Aware Online Dynamic Learning for Nonstationary Multivariate Time Series: Application to Sintering Quality Prediction [8.77587871192859]
  A Drift-Aware Multi-Scale Dynamic Learning (DA-MSDL) framework is proposed to maintain robust multi-temporal predictive performance.<n>The proposed framework provides an effective online dynamic learning paradigm for quality monitoring in nonstationary environments.
  arXiv  Detail & Related papers  (2026-04-10T14:30:57Z)
• Extended Hybrid Timed Petri Nets with Semi-Supervised Anomaly Detection for Switched Systems, Modelling and Fault Detection [0.0]
  This paper proposes a unified fault detection framework for hybrid dynamical systems.<n>It integrates an Extended Timed Continuous Petri Net model with semi-supervised anomaly detection.<n>Results demonstrate high detection accuracy, fast convergence, and robust performance.
  arXiv  Detail & Related papers  (2026-04-05T10:54:15Z)
• Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Driving [22.3805998088591]
  DACER-F is a flow matching algorithm for generative policies in autonomous driving systems.<n>It achieves a score of 775.8 on the humanoid-stand task and surpassing prior methods.
  arXiv  Detail & Related papers  (2026-03-03T05:35:53Z)
• Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA [61.12136997430116]
  Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration.<n> directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) catastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (ii) textitinefficient communication and convergence during model aggregation process,
  arXiv  Detail & Related papers  (2026-02-24T02:45:32Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• FAIM: Frequency-Aware Interactive Mamba for Time Series Classification [87.84511960413715]
  Time series classification (TSC) is crucial in numerous real-world applications, such as environmental monitoring, medical diagnosis, and posture recognition.<n>We propose FAIM, a lightweight Frequency-Aware Interactive Mamba model.<n>We show that FAIM consistently outperforms existing state-of-the-art (SOTA) methods, achieving a superior trade-off between accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-11-26T08:36:33Z)
• Simulating Distribution Dynamics: Liquid Temporal Feature Evolution for Single-Domain Generalized Object Detection [58.25418970608328]
  Single-Domain Generalized Object Detection (Single-DGOD) aims to transfer a detector trained on one source domain to multiple unknown domains.<n>Existing methods for Single-DGOD typically rely on discrete data augmentation or static perturbation methods to expand data diversity.<n>We propose a new method, which simulates the progressive evolution of features from the source domain to simulated latent distributions.
  arXiv  Detail & Related papers  (2025-11-13T03:10:39Z)
• Optimal Control Meets Flow Matching: A Principled Route to Multi-Subject Fidelity [35.95129874095729]
  Text-to-image (T2I) models excel on single-entity prompts but struggle with multi-subject descriptions.<n>We introduce the first theoretical framework with principled optimizable objective for steering sampling dynamics toward multi-subject fidelity.
  arXiv  Detail & Related papers  (2025-10-02T17:59:58Z)
• Vaccinating Federated Learning for Robust Modulation Classification in Distributed Wireless Networks [0.0]
  We propose FedVaccine, a novel AMC model aimed at improving generalizability across signals with varying noise levels. FedVaccine overcomes the limitations of existing FL-based AMC models' linear aggregation by employing a split-learning strategy. These findings highlight FedVaccine's potential to enhance the reliability and performance of AMC systems in practical wireless network environments.
  arXiv  Detail & Related papers  (2024-10-16T17:48:47Z)
• Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control [63.310780486820796]
  We show how a parameterization of recurrent connectivity influences robustness in closed-loop settings. We find that closed-form continuous-time neural networks (CfCs) with fewer parameters can outperform their full-rank, fully-connected counterparts.
  arXiv  Detail & Related papers  (2023-10-05T21:44:18Z)
• Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution [57.71199089609161]
  Long-term time-series forecasting (LTTF) has become a pressing demand in many applications, such as wind power supply planning. Transformer models have been adopted to deliver high prediction capacity because of the high computational self-attention mechanism. We propose an efficient Transformerbased model, named Conformer, which differentiates itself from existing methods for LTTF in three aspects.
  arXiv  Detail & Related papers  (2023-01-05T13:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.