Next-Token Prediction Learns Generalisable Representations of Sleep Physiology

• URL: http://arxiv.org/abs/2606.09605v1
• Date: Mon, 08 Jun 2026 15:13:57 GMT
• Title: Next-Token Prediction Learns Generalisable Representations of Sleep Physiology
• Abstract summary: We develop Hypnos, a sleep foundation model trained using eight different sensing modalities.<n>We tokenize each modality into streams of discrete tokens using residual vector quantization.<n>We train a large auto-regressive RQ-Transformer to jointly predict the next token across all modalities in parallel.
• Score: 3.80147466733132
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Foundation models offer a promising route to compress multi-modal physiological signals into compact representations of human health, with broad applications across sleep medicine, cardiology, neurology and other healthcare domains. Existing models have typically been trained with masked-reconstruction or contrastive objectives. However, masked reconstruction may be poorly suited to the stochastic nature of these signals, while contrastive approaches rely on positive-pair definitions despite the semantic invariances of physiological signals being poorly understood. In this work, we show that next-token prediction is a simple and scalable alternative. We develop Hypnos, a multi-modal sleep foundation model trained using eight different sensing modalities (e.g. EEG, ECG, respiratory signals) drawn from over 20,000 overnight polysomnography recordings. We tokenize each modality into streams of discrete tokens using residual vector quantization, then train a large auto-regressive RQ-Transformer to jointly predict the next token across all modalities in parallel. After training, Hypnos can be applied to continuous streams of sensor data from any subset of supported modalities, generating embeddings for downstream tasks. Across a range of benchmarks, Hypnos significantly outperforms existing foundation models. In sleep stage classification, we match the performance of strong supervised baselines on held-out test sets whilst using \(100\times\) less labelled data. Hypnos even generalises to daytime physiology, surpassing a dedicated ECG foundation model at detecting atrial fibrillation. Our results demonstrate that next-token prediction is a strong self-supervised objective for representation learning from multi-modal physiological signals.

Related papers

• Deep Learning Approaches for Sleep Apnea Classification from Polysomnographic EEG Signals [0.0]
  Sleep apnea diagnosis via polysomnography remains resource intensive and relies on manual data analysis and scoring.<n>Recent work has demonstrated that central nervous system effects can be detected through electroencephalogram (EEG) signals.<n>We present a comprehensive comparison of deep learning architectures and feature representations for automated sleep apnea detection from multichannel EEG.
  arXiv  Detail & Related papers  (2026-07-16T21:51:20Z)
• Pretraining on Sleep Data Improves non-Sleep Biosignal Tasks [5.729701207077404]
  Sleep foundation models have recently demonstrated strong performance on in-domain polysomnography tasks.<n>We investigate whether sleep biosignals can serve as an effective pretraining distribution for learning representations that transfer beyond sleep.
  arXiv  Detail & Related papers  (2026-05-04T11:51:54Z)
• Deep Learning for Metabolic Rate Estimation from Biosignals: A Comparative Study of Architectures and Signal Selection [7.2462186877798755]
  Energy expenditure estimation aims to infer human metabolic rate from physiological signals such as heart rate, respiration, or accelerometer data.<n>In this work, we systematically evaluate the role of neural architecture from that of signal choice.<n>Our results show that minute ventilation is the most predictive individual signal, with a transformer model achieving the lowest root mean square error (RMSE) of 0.87 W/kg.
  arXiv  Detail & Related papers  (2025-11-12T12:44:27Z)
• NAP: Attention-Based Late Fusion for Automatic Sleep Staging [0.0]
  NAP is an attention-based model which learns to combine multiple prediction streams using a tri-axial attention mechanism that captures temporal, spatial, and predictor-level dependencies.<n> NAP consistently outperforms individual predictors and simple ensembles, achieving state-of-the-art zero-shot generalization across multiple datasets.
  arXiv  Detail & Related papers  (2025-11-05T14:15:58Z)
• NeuroRVQ: Multi-Scale EEG Tokenization for Generative Large Brainwave Models [66.91449452840318]
  We introduce NeuroRVQ, a scalable Large Brainwave Model (LBM) centered on a codebook-based tokenizer.<n>Our tokenizer integrates: (i) multi-scale feature extraction modules that capture the full frequency neural spectrum; (ii) hierarchical residual vector quantization (RVQ) codebooks for high-resolution encoding; and, (iii) an EEG signal phase- and amplitude-aware loss function for efficient training.<n>Our empirical results demonstrate that NeuroRVQ achieves lower reconstruction error and outperforms existing LBMs on a variety of downstream tasks.
  arXiv  Detail & Related papers  (2025-10-15T01:26:52Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities [9.785262633953794]
  Physio Omni is a foundation model for multimodal physiological signal analysis.<n>It trains a decoupled multimodal tokenizer, enabling masked signal pre-training.<n>It achieves state-of-the-art performance while maintaining strong robustness to missing modalities.
  arXiv  Detail & Related papers  (2025-04-28T09:00:04Z)
• TokenUnify: Scaling Up Autoregressive Pretraining for Neuron Segmentation [65.65530016765615]
  We propose a hierarchical predictive coding framework that captures multi-scale dependencies through three complementary learning objectives.<n> TokenUnify integrates random token prediction, next-token prediction, and next-all token prediction to create a comprehensive representational space.<n>We also introduce a large-scale EM dataset with 1.2 billion annotated voxels, offering ideal long-sequence visual data with spatial continuity.
  arXiv  Detail & Related papers  (2024-05-27T05:45:51Z)
• Continuous time recurrent neural networks: overview and application to forecasting blood glucose in the intensive care unit [56.801856519460465]
  Continuous time autoregressive recurrent neural networks (CTRNNs) are a deep learning model that account for irregular observations. We demonstrate the application of these models to probabilistic forecasting of blood glucose in a critical care setting.
  arXiv  Detail & Related papers  (2023-04-14T09:39:06Z)
• Ensemble of Convolution Neural Networks on Heterogeneous Signals for Sleep Stage Scoring [63.30661835412352]
  This paper explores and compares the convenience of using additional signals apart from electroencephalograms. The best overall model, an ensemble of Depth-wise Separational Convolutional Neural Networks, has achieved an accuracy of 86.06%.
  arXiv  Detail & Related papers  (2021-07-23T06:37:38Z)
• Convolutional Neural Networks for Sleep Stage Scoring on a Two-Channel EEG Signal [63.18666008322476]
  Sleep problems are one of the major diseases all over the world. Basic tool used by specialists is the Polysomnogram, which is a collection of different signals recorded during sleep. Specialists have to score the different signals according to one of the standard guidelines.
  arXiv  Detail & Related papers  (2021-03-30T09:59:56Z)
• MSED: a multi-modal sleep event detection model for clinical sleep analysis [62.997667081978825]
  We designed a single deep neural network architecture to jointly detect sleep events in a polysomnogram. The performance of the model was quantified by F1, precision, and recall scores, and by correlating index values to clinical values.
  arXiv  Detail & Related papers  (2021-01-07T13:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.