Learning Transferable Sensor Models via Language-Informed Pretraining

• URL: http://arxiv.org/abs/2603.11950v1
• Date: Thu, 12 Mar 2026 13:59:13 GMT
• Title: Learning Transferable Sensor Models via Language-Informed Pretraining
• Abstract summary: textbfSLIP is an open-source framework for learning language-aligned representations that generalize across diverse sensor setups.<n>SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning.<n>Across 11 datasets, SLIP demonstrates superior performance in zero-shot transfer, signal captioning, and question answering.
• Score: 4.764303199378882
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern sensing systems generate large volumes of unlabeled multivariate time-series data. This abundance of unlabeled data makes self-supervised learning (SSL) a natural approach for learning transferable representations. However, most existing approaches are optimized for reconstruction or forecasting objectives and often fail to capture the semantic structure required for downstream classification and reasoning tasks. While recent sensor-language alignment methods improve semantic generalization through captioning and zero-shot transfer, they are limited to fixed sensor configurations, such as predefined channel sets, signal lengths, or temporal resolutions, which hinders cross-domain applicability. To address these gaps, we introduce \textbf{SLIP} (\textbf{S}ensor \textbf{L}anguage-\textbf{I}nformed \textbf{P}retraining), an open-source framework for learning language-aligned representations that generalize across diverse sensor setups. SLIP integrates contrastive alignment with sensor-conditioned captioning, facilitating both discriminative understanding and generative reasoning. By repurposing a pretrained decoder-only language model via cross-attention and introducing an elegant, flexible patch-embedder, SLIP supports different temporal resolutions and variable-length input at inference time without additional retraining. Across 11 datasets, SLIP demonstrates superior performance in zero-shot transfer, signal captioning, and question answering. It achieves a 77.14% average linear-probing accuracy, a 5.93% relative improvement over strong baselines, and reaches 64.83% accuracy in sensor-based question answering.

Related papers

• Hierarchical Adaptive Feature Refinement Network for VHR Remote Sensing Image Segmentation [15.24999553337715]
  We present HAFR-Net, a progressive refinement framework that adaptively organizes and refines hierarchical representations.<n>Under a matched Swin-B training and single-scale inference protocol, HAFR-Net attains 84.12%, 87.86%, 55.17%, and 67.70% mIoU.
  arXiv  Detail & Related papers  (2026-08-16T09:26:58Z)
• Self-Supervised Representation-Guided Generative Dataset Distillation [55.27114962330541]
  We propose self-supervised representation-guided generative dataset distillation (SRG)<n>SRG translates the SSL geometry into diffusion guidance.<n>It consistently outperforms evaluated generative baselines across multiple datasets and IPC settings.
  arXiv  Detail & Related papers  (2026-08-04T06:51:45Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings [0.43748379918040853]
  We introduce CHARM, which incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order.<n>ChaRM is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings.<n>Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel identifiers for cross-dataset generalization.
  arXiv  Detail & Related papers  (2026-05-29T17:48:30Z)
• Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning [57.91658393667469]
  Interpretable Prompt Learning is a hybrid framework that alternates between discrete semantic token selection and continuous prompt optimization.<n>Our framework is plug-and-play, allowing seamless integration with existing prompt learning methods.
  arXiv  Detail & Related papers  (2026-05-06T02:38:59Z)
• SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation [52.752934028506274]
  We present SignDPO, a novel multi-level Direct Preference optimisation framework.<n>We show that SignDPO consistently outperforms state-of-the-art gloss-free methods.<n>Our results suggest that multi-level preference alignment is a powerful paradigm for bridging the gap between high-entropy skeletal trajectories and discrete linguistic semantics.
  arXiv  Detail & Related papers  (2026-04-20T09:59:18Z)
• Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining [59.2578488860426]
  Heterogeneous multi-modal remote sensing object detection aims to accurately detect objects from diverse sensors.<n>Existing approaches largely adopt a late alignment paradigm, in which modality alignment and task-specific optimization are entangled during downstream fine-tuning.<n>We propose BabelRS, a unified language-pivoted pretraining framework that explicitly decouples modality alignment from downstream task learning.
  arXiv  Detail & Related papers  (2026-03-02T11:38:12Z)
• Steering Vision-Language Pre-trained Models for Incremental Face Presentation Attack Detection [62.89126207012712]
  Face Presentation Attack Detection (PAD) demands incremental learning to combat spoofing tactics and domains.<n>Privacy regulations forbid retaining past data, necessitating rehearsal-free learning (RF-IL)
  arXiv  Detail & Related papers  (2025-12-22T04:30:11Z)
• Few-Shot Remote Sensing Image Scene Classification with CLIP and Prompt Learning [0.9558392439655014]
  We explore prompt learning as a lightweight and efficient adaptation strategy for few-shot remote sensing image scene classification.<n>We benchmark these prompt-learning methods against two standard baselines: zero-shot CLIP with hand-crafted prompts and a linear probe trained on frozen CLIP features.<n>Our findings underscore prompt learning as a scalable and efficient solution for bridging the domain gap in satellite and aerial imagery.
  arXiv  Detail & Related papers  (2025-10-28T11:39:22Z)
• AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition [0.0]
  Continuously recognizing sign gestures and converting them to glosses plays a key role in bridging the gap between hearing and hearing-impaired communities.<n>We propose AutoSign, an autoregressive decoder-only transformer that directly translates pose sequences to natural language text.<n>By eliminating the multi-stage pipeline, AutoSign achieves substantial improvements on the Isharah-1000 dataset.
  arXiv  Detail & Related papers  (2025-07-26T07:28:33Z)
• VoxelOpt: Voxel-Adaptive Message Passing for Discrete Optimization in Deformable Abdominal CT Registration [15.78340001680369]
  We propose VoxelOpt, a discrete optimization-based deformable image registration framework.<n>It combines the strengths of learning-based and iterative methods to achieve a better balance between registration accuracy and runtime.<n>In abdominal CT registration, these changes allow VoxelOpt to outperform leading iterative in both efficiency and accuracy, while matching state-of-the-art learning-based methods trained with label supervision.
  arXiv  Detail & Related papers  (2025-06-24T19:44:04Z)
• StgcDiff: Spatial-Temporal Graph Condition Diffusion for Sign Language Transition Generation [33.695308849489784]
  We propose StgcDiff, a graph-based conditional diffusion framework that generates smooth transitions between discrete signs.<n>Specifically, we train an encoder-decoder architecture to learn a structure-aware representation of spatial-temporal skeleton.<n>We design the Sign-GCN module as the key component in our framework, which effectively models the spatial-temporal features.
  arXiv  Detail & Related papers  (2025-06-16T07:09:51Z)
• Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion [52.315729095824906]
  MLLM Semantic-Corrected Ping-Pong-Ahead Diffusion (PPAD) is a novel framework that introduces a Multimodal Large Language Model (MLLM) as a semantic observer during inference.<n>It performs real-time analysis on intermediate generations, identifies latent semantic inconsistencies, and translates feedback into controllable signals that actively guide the remaining denoising steps.<n>Extensive experiments demonstrate PPAD's significant improvements.
  arXiv  Detail & Related papers  (2025-05-26T14:42:35Z)
• Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction [20.1863553357121]
  Current deep learning architectures for remote sensing are fundamentally rigid.<n>We introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling.<n> STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands.<n>It unifies various dense prediction tasks and diverse semantic class predictions.
  arXiv  Detail & Related papers  (2025-05-18T07:39:17Z)
• LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models [67.19124099815645]
  We propose a novel Language-Aware Soft Prompting (LASP) learning method to alleviate base class overfitting. LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available. LASP matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets.
  arXiv  Detail & Related papers  (2022-10-03T17:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.