Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving

• URL: http://arxiv.org/abs/2605.22809v2
• Date: Fri, 22 May 2026 21:31:07 GMT
• Title: Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
• Abstract summary: Robust training and validation of Autonomous Driving Systems (ADS) require massive, diverse datasets.<n>In-the-wild data from sources like dashcams offers immense scale and diversity, capturing critical long-tail scenarios and novel environments.<n>We propose Sensor2Sensor, a novel generative modeling paradigm that translates in-the-wild dashcam videos into a high-fidelity, multi-modal sensor suite.
• Score: 35.07458382590446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robust training and validation of Autonomous Driving Systems (ADS) require massive, diverse datasets. Proprietary data collected by Autonomous Vehicle (AV) fleets, while high-fidelity, are limited in scale, diversity of sensor configurations, as well as geographic and long-tail-behavioral coverage. In contrast, in-the-wild data from sources like dashcams offers immense scale and diversity, capturing critical long-tail scenarios and novel environments. However, this unstructured, in-the-wild video data is incompatible with ADS expecting structured, multi-modal sensor inputs for validation and training. To bridge this data gap, we propose Sensor2Sensor, a novel generative modeling paradigm that translates in-the-wild monocular dashcam videos into a high-fidelity, multi-modal sensor suite (AV logs) comprising multi-view camera images and LiDAR point clouds. A core challenge is the lack of paired training data. We address this by converting real AV logs into dashcam-style videos via 4D Gaussian Splatting (4DGS) reconstruction and novel-view rendering. Sensor2Sensor then utilizes a diffusion architecture to perform the generative conversion. We perform comprehensive quantitative evaluations on the fidelity and realism of the generated sensor data. We demonstrate Sensor2Sensor's practical utility by converting challenging in-the-wild internet and dashcam footage into realistic, multi-modal data formats, further unlocking vast external data sources for AV development.

Related papers

• iVISION-2DCD: A Long-Term Change Detection Dataset for Large-Scale Outdoor Construction Monitoring [50.60358463714119]
  We present iVISION-2DCD, a large-scale synthetically generated dataset from dense LiDAR point clouds.<n>Our dataset formally defines the problem of viewpoint-robust 2DCD at construction sites.<n> Benchmark evaluations using state-of-the-art 2DCD algorithms demonstrate that iVISION-2DCD poses novel research challenges.
  arXiv  Detail & Related papers  (2026-07-03T18:29:13Z)
• Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis [53.67632483195509]
  We introduce StreetNVS, a video diffusion framework that conditions on all three signals through a Reference-Enhanced Camera Attention module.<n>We show capabilities of synthesizing coherent videos along extreme out-of-trajectory paths such as elevation, lane-shift, pullback, and rotation.
  arXiv  Detail & Related papers  (2026-06-01T02:37:56Z)
• REFNet++: Multi-Task Efficient Fusion of Camera and Radar Sensor Data in Bird's-Eye Polar View [13.463025176847703]
  We address the challenge of multimodal sensor fusion by aligning radar and camera data in a unified domain.<n>We employ a variational encoder-decoder architecture that learns the transformation of front-view camera data into the Bird's-Eye View (BEV) polar domain.<n>This alignment ensures that both modalities are represented in a compatible domain, robust and efficient sensor fusion.
  arXiv  Detail & Related papers  (2026-05-12T09:13:02Z)
• OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving [58.693329943871355]
  We propose OminiGen, which generates aligned multimodal sensor data in a unified framework.<n>Our approach leverages a shared Birdu 2019s Eye View (BEV) space to unify multimodal features.<n>UAE achieves multimodal sensor decoding through volume rendering, enabling accurate and flexible reconstruction.
  arXiv  Detail & Related papers  (2025-12-16T09:18:15Z)
• Neural Rendering for Sensor Adaptation in 3D Object Detection [3.10688583550805]
  We investigate the impact of the cross-sensor domain gap on state-of-the-art 3D object detectors.<n>We show that model architectures based on a dense Bird's Eye View (BEV) representation with backward projection, such as BEVFormer, are the most robust against varying sensor configurations.<n>We propose a novel data-driven sensor adaptation pipeline based on neural rendering, which can transform entire datasets to match different camera sensor setups.
  arXiv  Detail & Related papers  (2025-08-18T07:53:45Z)
• Adaptive Domain Learning for Cross-domain Image Denoising [57.4030317607274]
  We present a novel adaptive domain learning scheme for cross-domain image denoising. We use existing data from different sensors (source domain) plus a small amount of data from the new sensor (target domain) The ADL training scheme automatically removes the data in the source domain that are harmful to fine-tuning a model for the target domain. Also, we introduce a modulation module to adopt sensor-specific information (sensor type and ISO) to understand input data for image denoising.
  arXiv  Detail & Related papers  (2024-11-03T08:08:26Z)
• Multi-Modal 3D Object Detection by Box Matching [109.43430123791684]
  We propose a novel Fusion network by Box Matching (FBMNet) for multi-modal 3D detection. With the learned assignments between 3D and 2D object proposals, the fusion for detection can be effectively performed by combing their ROI features.
  arXiv  Detail & Related papers  (2023-05-12T18:08:51Z)
• HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detection [0.0]
  We propose HRFuser, a modular architecture for multi-modal 2D object detection. It fuses multiple sensors in a multi-resolution fashion and scales to an arbitrary number of input modalities. We demonstrate via experiments on nuScenes and the adverse conditions DENSE datasets that our model effectively leverages complementary features from additional modalities.
  arXiv  Detail & Related papers  (2022-06-30T09:40:05Z)
• Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles [3.7543422202019427]
  Generalized Sensor Fusion (GSF) is designed in such a way that both sensor inputs and target tasks are modular and modifiable. This enables AV system designers to easily experiment with different sensor configurations and methods and opens up the ability to deploy on heterogeneous fleets.
  arXiv  Detail & Related papers  (2021-12-01T01:43:15Z)
• VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles [131.2240621036954]
  We present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras. We demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle.
  arXiv  Detail & Related papers  (2021-11-23T18:58:10Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)
• High-Precision Digital Traffic Recording with Multi-LiDAR Infrastructure Sensor Setups [0.0]
  We investigate the impact of fused LiDAR point clouds compared to single LiDAR point clouds. The evaluation of the extracted trajectories shows that a fused infrastructure approach significantly increases the tracking results and reaches accuracies within a few centimeters.
  arXiv  Detail & Related papers  (2020-06-22T10:57:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.