Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training

• URL: http://arxiv.org/abs/2606.12240v1
• Date: Wed, 10 Jun 2026 15:43:20 GMT
• Title: Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training
• Abstract summary: Liquid Neural Networks (LNNs) address some of these limitations through continuous-time dynamics.<n>We propose a Multi-Rate Mixture-of-Experts (MR-MoE) framework built on top of Liquid Neural Networks.<n>In the proposed architecture, multiple LNN-based experts operate at distinct time scales, enabling the model to explicitly separate fast-changing dynamics from slow-evolving temporal trends.
• Score: 5.248012610915348
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multivariate time-series data often exhibit complex temporal dependencies, irregular sampling, and heterogeneous dynamics across multiple time scales, making accurate sequence modeling particularly challenging. Traditional recurrent neural networks (RNNs), such as Long Short-Term Memory (LSTM) networks, operate in discrete time and may struggle to effectively capture continuous and irregular temporal behaviors. Liquid Neural Networks (LNNs) address some of these limitations through continuous-time dynamics, but standard LNN architectures typically rely on a single dynamical system, limiting their ability to model heterogeneous temporal patterns. To address these challenges, we propose a Multi-Rate Mixture-of-Experts (MR-MoE) framework built on top of Liquid Neural Networks. In the proposed architecture, multiple LNN-based experts operate at distinct time scales, enabling the model to explicitly separate fast-changing dynamics from slow-evolving temporal trends. A gating network further enables adaptive expert specialization based on input conditions. In addition, we incorporate both feature-level and temporal attention mechanisms to improve robustness, interpretability, and long-range dependency modeling. Feature-level attention suppresses noisy or irrelevant variables, while temporal attention selectively focuses on informative historical states. We evaluate the proposed framework on a complex multivariate time-series prediction task and compare it against strong baselines, including LSTM, monolithic LNN, and standard MoE models. Experimental results demonstrate that the proposed MR-MoE framework consistently achieves improved AUROC and AUPRC performance while maintaining favorable computational efficiency. These results highlight the effectiveness of combining continuous-time dynamics, multi-scale expert decomposition, and adaptive attention mechanisms for time-series modeling.

Related papers

• Multi-Timescale Conductance Spiking Networks: A Sparse, Gradient-Trainable Framework with Rich Firing Dynamics for Enhanced Temporal Processing [0.20240089219927163]
  Spiking neural networks (SNNs) promise low-power event-driven computation for temporally rich tasks.<n>We introduce multi-timescale conductance spiking networks, a gradient-trainable framework in which neural dynamics emerge from shaping the current-voltage curve.<n>We derive a discrete-time formulation of these differentiable dynamics, enabling direct backpropagation through time without surrogate-gradient approximations.
  arXiv  Detail & Related papers  (2026-05-12T09:22:45Z)
• NeuroSSM: Multiscale Differential State-Space Modeling for Context-Aware fMRI Analysis [4.753690672619091]
  We propose NeuroSSM, a selective state-space architecture designed for end-to-end analysis of raw BOLD signals in fMRI time series.<n>NeuroSSM addresses the above limitations through two complementary design components.<n> Experiments on clinical and non-clinical datasets demonstrate that NeuroSSM achieves competitive performance and efficiency against state-of-the-art fMRI analysis methods.
  arXiv  Detail & Related papers  (2026-01-03T16:35:45Z)
• TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition [59.99922360648663]
  TSkel-Mamba is a hybrid Transformer-Mamba framework that effectively captures both spatial and temporal dynamics.<n>The MTI module employs multi-scale Cycle operators to capture cross-channel temporal interactions, a critical factor in action recognition.
  arXiv  Detail & Related papers  (2025-12-12T11:55:16Z)
• MSTN: Fast and Efficient Multivariate Time Series Model [0.0]
  We introduce the Multi-scale Temporal Network (MSTN), a novel deep learning architecture founded on a hierarchical multi-scale and sequence modeling principle.<n>MSTN integrates a convolutional encoder that constructs a hierarchical feature pyramid for local patterns and a sequence modeling component for long-range temporal dependencies.<n>Extensive evaluations across time-series long-horizon forecasting, imputation, classification and generalizability study demonstrate that MSTN achieves competitive state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2025-11-25T18:09:42Z)
• TS-LIF: A Temporal Segment Spiking Neuron Network for Time Series Forecasting [27.91825785119938]
  Spiking Neural Networks (SNNs) offer a promising, biologically inspired approach for processing data for time series forecasting.<n>We introduce the Temporal Leaky Segment Integrate-and-Fire model, featuring a dual-compartment architecture.<n> Experimental results show that TS-LIF outperforms traditional SNNs in time series forecasting.
  arXiv  Detail & Related papers  (2025-03-07T03:06:21Z)
• Trajectory Flow Matching with Applications to Clinical Time Series Modeling [77.58277281319253]
  Trajectory Flow Matching (TFM) trains a Neural SDE in a simulation-free manner, bypassing backpropagation through the dynamics.<n>We demonstrate improved performance on three clinical time series datasets in terms of absolute performance and uncertainty prediction.
  arXiv  Detail & Related papers  (2024-10-28T15:54:50Z)
• PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations [49.80959046861793]
  We present PDETime, a novel LMTF model inspired by the principles of Neural PDE solvers. Our experimentation across seven diversetemporal real-world LMTF datasets reveals that PDETime adapts effectively to the intrinsic nature of the data.
  arXiv  Detail & Related papers  (2024-02-25T17:39:44Z)
• Contextualizing MLP-Mixers Spatiotemporally for Urban Data Forecast at Scale [54.15522908057831]
  We propose an adapted version of the computationally-Mixer for STTD forecast at scale. Our results surprisingly show that this simple-yeteffective solution can rival SOTA baselines when tested on several traffic benchmarks. Our findings contribute to the exploration of simple-yet-effective models for real-world STTD forecasting.
  arXiv  Detail & Related papers  (2023-07-04T05:19:19Z)
• Gated Recurrent Neural Networks with Weighted Time-Delay Feedback [55.596897987498174]
  We present a novel approach to modeling long-term dependencies in sequential data by introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism.<n>Our proposed model, named $tau$-GRU, is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs)
  arXiv  Detail & Related papers  (2022-12-01T02:26:34Z)
• Deep Explicit Duration Switching Models for Time Series [84.33678003781908]
  We propose a flexible model that is capable of identifying both state- and time-dependent switching dynamics. State-dependent switching is enabled by a recurrent state-to-switch connection. An explicit duration count variable is used to improve the time-dependent switching behavior.
  arXiv  Detail & Related papers  (2021-10-26T17:35:21Z)
• Dynamic Gaussian Mixture based Deep Generative Model For Robust Forecasting on Sparse Multivariate Time Series [43.86737761236125]
  We propose a novel generative model, which tracks the transition of latent clusters, instead of isolated feature representations. It is characterized by a newly designed dynamic Gaussian mixture distribution, which captures the dynamics of clustering structures. A structured inference network is also designed for enabling inductive analysis.
  arXiv  Detail & Related papers  (2021-03-03T04:10:07Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.