123D: Unifying Multi-Modal Autonomous Driving Data at Scale

• URL: http://arxiv.org/abs/2605.08084v1
• Date: Fri, 08 May 2026 17:59:53 GMT
• Title: 123D: Unifying Multi-Modal Autonomous Driving Data at Scale
• Abstract summary: We present 123D, an open-source framework that unifies such multi-modal driving data through a single API.<n>We consolidate eight real-world driving datasets spanning 3,300 hours and 90,000 kilometers, together with a synthetic dataset with collection scripts, and provide tools for data analysis and visualization.
• Score: 58.4010470217018
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each dataset adopts different 2D and 3D modalities, such as cameras, lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple datasets. We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, we store each modality as an independent timestamped event stream with no prescribed rate, enabling synchronous or asynchronous access across arbitrary datasets. Using 123D, we consolidate eight real-world driving datasets spanning 3,300 hours and 90,000 kilometers, together with a synthetic dataset with configurable collection scripts, and provide tools for data analysis and visualization. We conduct a systematic study comparing annotation statistics and assessing each dataset's pose and calibration accuracy. Further, we showcase two applications 123D enables: cross-dataset 3D object detection transfer and reinforcement learning for planning, and offer recommendations for future directions. Code and documentation are available at https://github.com/kesai-labs/py123d.

Related papers

• ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K [83.97657657382376]
  We present ManiTwin, a pipeline for generating data-generation-ready digital object twins.<n>We construct ManiTwin-100K, a dataset containing 100K high-quality annotated 3D assets.<n>Experiments demonstrate that ManiTwin provides an efficient asset synthesis and annotation workflow.
  arXiv  Detail & Related papers  (2026-03-17T17:59:49Z)
• DrivIng: A Large-Scale Multimodal Driving Dataset with Full Digital Twin Integration [35.44401354820601]
  We present DrivIng, a large-scale multimodal dataset with a complete digital twin of a 18 km route.<n>Our dataset provides continuous recordings from six RGB cameras, one LiDAR, and high-precision ADMA-based localization, captured across day, dusk, and night.<n>DrivIng enables a 1-to-1 transfer of real traffic into simulation, preserving agent interactions while enabling realistic and flexible scenario testing.
  arXiv  Detail & Related papers  (2026-01-21T18:41:05Z)
• SCaRL- A Synthetic Multi-Modal Dataset for Autonomous Driving [0.0]
  We present a novel synthetically generated multi-modal dataset, SCaRL, to enable the training and validation of autonomous driving solutions. SCaRL is a large dataset based on the CARLA Simulator, which provides data for diverse, dynamic scenarios and traffic conditions.
  arXiv  Detail & Related papers  (2024-05-27T10:31:26Z)
• Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting [64.7364925689825]
  Argoverse 2 (AV2) is a collection of three datasets for perception and forecasting research in the self-driving domain. The Lidar dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. The Motion Forecasting dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene.
  arXiv  Detail & Related papers  (2023-01-02T00:36:22Z)
• IDD-3D: Indian Driving Dataset for 3D Unstructured Road Scenes [79.18349050238413]
  Preparation and training of deploy-able deep learning architectures require the models to be suited to different traffic scenarios. An unstructured and complex driving layout found in several developing countries such as India poses a challenge to these models. We build a new dataset, IDD-3D, which consists of multi-modal data from multiple cameras and LiDAR sensors with 12k annotated driving LiDAR frames.
  arXiv  Detail & Related papers  (2022-10-23T23:03:17Z)
• SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving [94.11868795445798]
  We release a Large-Scale Object Detection benchmark for Autonomous driving, named as SODA10M, containing 10 million unlabeled images and 20K images labeled with 6 representative object categories. To improve diversity, the images are collected every ten seconds per frame within 32 different cities under different weather conditions, periods and location scenes. We provide extensive experiments and deep analyses of existing supervised state-of-the-art detection models, popular self-supervised and semi-supervised approaches, and some insights about how to develop future models.
  arXiv  Detail & Related papers  (2021-06-21T13:55:57Z)
• One Million Scenes for Autonomous Driving: ONCE Dataset [91.94189514073354]
  We introduce the ONCE dataset for 3D object detection in the autonomous driving scenario. The data is selected from 144 driving hours, which is 20x longer than the largest 3D autonomous driving dataset available. We reproduce and evaluate a variety of self-supervised and semi-supervised methods on the ONCE dataset.
  arXiv  Detail & Related papers  (2021-06-21T12:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.