On the Role of Inductive Bias in Time-Series Pretraining: A Case Study in Learning Generalizable Representations for Clinical Time Series

• URL: http://arxiv.org/abs/2605.26194v2
• Date: Wed, 27 May 2026 19:20:57 GMT
• Title: On the Role of Inductive Bias in Time-Series Pretraining: A Case Study in Learning Generalizable Representations for Clinical Time Series
• Abstract summary: We study the question of which inductive biases should the pretraining objective impose so that representations transfer across task types and subjects.<n>We find that dynamics-centric mixtures produce the most balanced transfer.<n>Overall, combining local reconstruction with temporal continuity, and adding in-context conditioning when access is realistic, yields robust subject-generalizing representations.
• Score: 3.5353167506352445
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Clinical time-series learning is routinely constrained by small, heterogeneous cohorts and protocol drift, while its downstream use spans both classification (e.g., pathology diagnosis) and regression (e.g., temporal forecasting). These constraints make foundation-model pretraining appealing, but raises an important question of which inductive biases should the pretraining objective impose so that representations transfer across task types and subjects. We study this question in pathological gait analysis for spinal cord injury (SCI) via PathoFM, an encoder-centric transformer pretrained on multivariate gait windows with three complementary objectives: Local Completion (reconstruct contiguous masked spans to enforce local structure), Temporal Continuity (predict a masked mid-horizon continuation from an observed prefix to enforce smoothness and causal consistency), and Unsupervised In-Context Dynamics (support-query reconstruction conditioned on subject exemplar windows via attention). Empirically comparing objective families (grouping/contrastive, dynamics-based, and generative reconstruction), we find that dynamics-centric mixtures produce the most balanced transfer: grouping objectives favor discriminative margins but can degrade magnitude fidelity needed for continuous targets, whereas reconstruction-only objectives preserve waveform structure but may underperform on classification. Overall, combining local reconstruction with temporal continuity, and adding in-context conditioning when exemplar access is realistic, yields robust subject-generalizing representations.

Related papers

• Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory [6.1311571881384745]
  We show that divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs.<n>Our findings identify divisive normalization as a critical computational mechanism for learning high-fidelity continuous representations.
  arXiv  Detail & Related papers  (2026-08-03T09:20:05Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Continual Segmentation under Joint Nonstationarity [8.522964568257455]
  Evolving data streams induce joint nonstationarity in continual semantic segmentation.<n>This setting reflects practical structured prediction systems, yet remains largely unexplored in prior continual learning work.<n>We formalize continual segmentation under coupled class, domain, and label shifts and investigate learning in heterogeneous dense prediction environments.
  arXiv  Detail & Related papers  (2026-05-19T22:20:00Z)
• AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Models [1.7823291802496628]
  We introduce AURORA, Adaptive Uncertainty Representations through Orthogonalized observational alignment.<n>AURORA decomposes semantic subspaces into contextual factors learns consistency objectives within each subspace.<n>It consistently outperforms reconstruction and substantially improves contextual disentanglement, neighborhood purity, and under institutional distribution shift.
  arXiv  Detail & Related papers  (2026-05-18T02:32:50Z)
• Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation [91.20850436220267]
  We introduce the Referring-Aware Visuomotor Policy (ReV)<n>ReV incorporates sparse referring points provided by a human or a high-level reasoning planner.<n>It is trained only by applying targeted perturbations to expert demonstrations.
  arXiv  Detail & Related papers  (2026-04-07T07:41:11Z)
• Smooth embeddings in contracting recurrent networks driven by regular dynamics: A synthesis for neural representation [45.88028371034407]
  Recent empirical work has documented topology-preserving latent organization in trained recurrent models.<n>Recent theoretical results in reservoir computing establish conditions under which the synchronization map is an embedding.<n>Our contribution is an integrated framework that assembles generalized synchronization and embedding guarantees for contracting reservoirs.
  arXiv  Detail & Related papers  (2026-01-26T23:10:39Z)
• A Backpropagation-Free Feedback-Hebbian Network for Continual Learning Dynamics [0.0]
  We show that a compact feedback pathway trained with local plasticity can support regeneration and continual-learning--relevant dynamics.<n>Results show that a compact feedback pathway trained with local plasticity can support regeneration and continual-learning--relevant dynamics.
  arXiv  Detail & Related papers  (2026-01-11T03:25:38Z)
• Limits of Generative Pre-Training in Structured EMR Trajectories with Irregular Sampling [0.7537475180985093]
  Foundation models refer to architectures trained on vast datasets using autoregressive pre-training to capture intricate patterns and motifs.<n>We trained two autoregressive models -- a sequence-to-sequence LSTM and a reduced Transformer -- on longitudinal ART for HIV and Acute Hypotension datasets.<n> Controlled irregularity was added during training via random inter-visit gaps, while test sequences stayed complete.<n>Both reproduced feature distributions but failed to preserve cross-feature structure.
  arXiv  Detail & Related papers  (2025-10-27T00:04:17Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Semantic Self-adaptation: Enhancing Generalization with a Single Sample [45.111358665370524]
  We propose a self-adaptive approach for semantic segmentation. It fine-tunes the parameters of convolutional layers to the input image using consistency regularization. Our empirical study suggests that self-adaptation may complement the established practice of model regularization at training time.
  arXiv  Detail & Related papers  (2022-08-10T12:29:01Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.