OctoSense: Self-Supervised Learning for Multimodal Robot Perception

• URL: http://arxiv.org/abs/2606.27317v1
• Date: Thu, 25 Jun 2026 17:30:49 GMT
• Title: OctoSense: Self-Supervised Learning for Multimodal Robot Perception
• Abstract summary: We present OctoSense, an open-source sensor platform with stereo and event cameras, LiDAR, a thermal camera, an inertial measurement unit, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot)<n>The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day.<n>We demonstrate multi-modal self-supervised learning using such real-world data, where sensors have different representations, frequencies, latencies and noise.
• Score: 50.20346079739425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.

Related papers

• SensorLM: Learning the Language of Wearable Sensors [50.95988682423808]
  We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language.<n>We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data.<n>This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people.
  arXiv  Detail & Related papers  (2025-06-10T17:13:09Z)
• Multi-modal Multi-platform Person Re-Identification: Benchmark and Method [58.59888754340054]
  MP-ReID is a novel dataset designed specifically for multi-modality and multi-platform ReID.<n>This benchmark compiles data from 1,930 identities across diverse modalities, including RGB, infrared, and thermal imaging.<n>We introduce Uni-Prompt ReID, a framework with specific-designed prompts, tailored for cross-modality and cross-platform scenarios.
  arXiv  Detail & Related papers  (2025-03-21T12:27:49Z)
• OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving [6.426389871517664]
  High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions.<n>We present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data.<n>The dataset comprises 1501 clips, each approximately 30-s long, totaling more than 450K synchronized frames and more than 5.85 million synchronized sensor data points.
  arXiv  Detail & Related papers  (2024-12-14T08:08:40Z)
• RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments [62.5830455357187]
  We setup an egocentric multi-sensor data collection platform based on 3 main types of sensors (Camera, LiDAR and Fisheye)<n>A large-scale multimodal dataset is constructed, named RoboSense, to facilitate egocentric robot perception.
  arXiv  Detail & Related papers  (2024-08-28T03:17:40Z)
• UnLoc: A Universal Localization Method for Autonomous Vehicles using LiDAR, Radar and/or Camera Input [51.150605800173366]
  UnLoc is a novel unified neural modeling approach for localization with multi-sensor input in all weather conditions. Our method is extensively evaluated on Oxford Radar RobotCar, ApolloSouthBay and Perth-WA datasets.
  arXiv  Detail & Related papers  (2023-07-03T04:10:55Z)
• SUPS: A Simulated Underground Parking Scenario Dataset for Autonomous Driving [41.221988979184665]
  SUPS is a simulated dataset for underground automatic parking. It supports multiple tasks with multiple sensors and multiple semantic labels aligned with successive images. We also evaluate the state-of-the-art SLAM algorithms and perception models on our dataset.
  arXiv  Detail & Related papers  (2023-02-25T02:59:12Z)
• The LuViRA Dataset: Synchronized Vision, Radio, and Audio Sensors for Indoor Localization [41.58739817444644]
  The dataset includes color images, corresponding depth maps, inertial measurement unit (IMU) readings, channel response between a 5G massive multiple-input and multiple-output (MIMO) testbed and user equipment. We synchronize these sensors to ensure that all data is recorded simultaneously. The main aim of this dataset is to enable research on sensor fusion with the most commonly used sensors for localization tasks.
  arXiv  Detail & Related papers  (2023-02-10T15:12:40Z)
• SELMA: SEmantic Large-scale Multimodal Acquisitions in Variable Weather, Daytime and Viewpoints [36.57734409668748]
  We introduce SELMA, a novel dataset for semantic segmentation. It contains more than 30K unique waypoints acquired from 24 different sensors including RGB, depth, semantic cameras and LiDARs. It is based on CARLA, an open-source simulator for generating synthetic data in autonomous driving scenarios.
  arXiv  Detail & Related papers  (2022-04-20T21:22:56Z)
• TUM-VIE: The TUM Stereo Visual-Inertial Event Dataset [50.8779574716494]
  Event cameras are bio-inspired vision sensors which measure per pixel brightness changes. They offer numerous benefits over traditional, frame-based cameras, including low latency, high dynamic range, high temporal resolution and low power consumption. To foster the development of 3D perception and navigation algorithms with event cameras, we present the TUM-VIE dataset.
  arXiv  Detail & Related papers  (2021-08-16T19:53:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.