Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

• URL: http://arxiv.org/abs/2605.31580v1
• Date: Fri, 29 May 2026 17:48:30 GMT
• Title: Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings
• Abstract summary: We introduce CHARM, which incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order.<n>ChaRM is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings.<n>Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel identifiers for cross-dataset generalization.
• Score: 0.43748379918040853
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Transformer-based architectures have advanced sequence modeling in language and vision, yet general-purpose representation learning for heterogeneous multivariate time series remains underexplored. We introduce CHARM (Channel-Aware Representation Model), which incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order. CHARM is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings; latent-space prediction encourages robustness to sensor noise while description-aware gating provides interpretability through learned inter-channel relationships. Across anomaly detection, classification, and short- and long-term forecasting, the learned embeddings achieve strong performance using only a linear probe. Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel identifiers for cross-dataset generalization.

Related papers

• ChorusTIC: Training-Free Multivariate Time Series Classification via Chorus In-Context Learning [52.72143039958434]
  We introduce ChorusTIC, a classification-native foundation model for in-context classification across heterogeneous channel configurations.<n>ChorusTIC maps variable channel configurations into a fixed-width representation independent of the original channel count.<n>It calibrates feature axes using context-derived distributions and predicts query labels through leakage-protected in-context learning.
  arXiv  Detail & Related papers  (2026-08-25T03:45:24Z)
• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing [8.306039957986394]
  We extend transcoder-based circuit tracing to multimodal diffusion transformers.<n>We find mechanisms underlying attribute binding and cross-stream semantic propagation.<n>Our results provide causal explanations for systematic generation errors.
  arXiv  Detail & Related papers  (2026-06-14T13:02:44Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Learning Transferable Sensor Models via Language-Informed Pretraining [4.764303199378882]
  textbfSLIP is an open-source framework for learning language-aligned representations that generalize across diverse sensor setups.<n>SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning.<n>Across 11 datasets, SLIP demonstrates superior performance in zero-shot transfer, signal captioning, and question answering.
  arXiv  Detail & Related papers  (2026-03-12T13:59:13Z)
• Adapformer: Adaptive Channel Management for Multivariate Time Series Forecasting [49.40321003932633]
  Adapformer is an advanced Transformer-based framework that merges the benefits of CI and CD methodologies through effective channel management.<n>Adapformer achieves superior performance over existing models, enhancing both predictive accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-11-18T16:24:05Z)
• SEED: A Structural Encoder for Embedding-Driven Decoding in Time Series Prediction with LLMs [3.036179638516407]
  We introduce SEED, a structural encoder for embedding-driven decoding, which integrates four stages: a token-aware encoder for patch extraction, a projection module that aligns patches with language model embeddings, and a semantic reprogramming mechanism that maps patches to task-aware prototypes.<n>This modular architecture decouples representation learning from inference, enabling efficient alignment between numerical patterns and semantic reasoning.
  arXiv  Detail & Related papers  (2025-06-25T06:40:14Z)
• FaCTR: Factorized Channel-Temporal Representation Transformers for Efficient Time Series Forecasting [0.0]
  Time series data is characterized by low per-timestep information density and complex dependencies across channels.<n>We propose a lightweight Transformer with an explicitly structural design FaCTR.<n>Despite its compact design, FaCTR state-the-art performance on eleven public forecasting benchmarks.
  arXiv  Detail & Related papers  (2025-06-05T21:17:53Z)
• Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction [20.1863553357121]
  Current deep learning architectures for remote sensing are fundamentally rigid.<n>We introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling.<n> STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands.<n>It unifies various dense prediction tasks and diverse semantic class predictions.
  arXiv  Detail & Related papers  (2025-05-18T07:39:17Z)
• ALFEE: Adaptive Large Foundation Model for EEG Representation [17.166788472910806]
  We propose the Adaptive Large Foundation model for EEG signal representation(ALFEE) framework.<n>ALFEE is a novel hybrid transformer architecture with two learning stages for robust EEG representation learning.<n>After 25,000 hours of pretraining, extensive experimental results on six downstream EEG tasks demonstrate the superior performance of ALFEE over existing models.
  arXiv  Detail & Related papers  (2025-05-07T13:32:31Z)
• Sentinel: Multi-Patch Transformer with Temporal and Channel Attention for Time Series Forecasting [48.52101281458809]
  Transformer-based time series forecasting has recently gained strong interest due to the ability of transformers to model sequential data.<n>We propose Sentinel, a transformer-based architecture composed of an encoder able to extract contextual information from the channel dimension.<n>We introduce a multi-patch attention mechanism, which leverages the patching process to structure the input sequence in a way that can be naturally integrated into the transformer architecture.
  arXiv  Detail & Related papers  (2025-03-22T06:01:50Z)
• Latent Diffusion Model-Enabled Low-Latency Semantic Communication in the Presence of Semantic Ambiguities and Wireless Channel Noises [18.539501941328393]
  This paper develops a latent diffusion model-enabled SemCom system to handle outliers in source data.<n>A lightweight single-layer latent space transformation adapter completes one-shot learning at the transmitter.<n>An end-to-end consistency distillation strategy is used to distill the diffusion models trained in latent space.
  arXiv  Detail & Related papers  (2024-06-09T23:39:31Z)
• CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting [50.23240107430597]
  We design a special Transformer, i.e., Channel Aligned Robust Blend Transformer (CARD for short), that addresses key shortcomings of CI type Transformer in time series forecasting. First, CARD introduces a channel-aligned attention structure that allows it to capture both temporal correlations among signals. Second, in order to efficiently utilize the multi-scale knowledge, we design a token blend module to generate tokens with different resolutions. Third, we introduce a robust loss function for time series forecasting to alleviate the potential overfitting issue.
  arXiv  Detail & Related papers  (2023-05-20T05:16:31Z)
• CCVS: Context-aware Controllable Video Synthesis [95.22008742695772]
  presentation introduces a self-supervised learning approach to the synthesis of new video clips from old ones. It conditions the synthesis process on contextual information for temporal continuity and ancillary information for fine control.
  arXiv  Detail & Related papers  (2021-07-16T17:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.