High-Fidelity Digital Twins for Bridging the Sim2Real Gap in LiDAR-Based ITS Perception

• URL: http://arxiv.org/abs/2509.02904v1
• Date: Wed, 03 Sep 2025 00:12:58 GMT
• Title: High-Fidelity Digital Twins for Bridging the Sim2Real Gap in LiDAR-Based ITS Perception
• Authors: Muhammad Shahbaz, Shaurya Agarwal,
• Abstract summary: This paper proposes a high-fidelity digital twin (HiFi DT) framework that incorporates real-world background geometry, lane-level road topology, and sensor-specific specifications and placement.<n>Experiments show that the DT-trained model outperforms the equivalent model trained on real data by 4.8%.
• Score: 3.1508266388327324
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Sim2Real domain transfer offers a cost-effective and scalable approach for developing LiDAR-based perception (e.g., object detection, tracking, segmentation) in Intelligent Transportation Systems (ITS). However, perception models trained in simulation often under perform on real-world data due to distributional shifts. To address this Sim2Real gap, this paper proposes a high-fidelity digital twin (HiFi DT) framework that incorporates real-world background geometry, lane-level road topology, and sensor-specific specifications and placement. We formalize the domain adaptation challenge underlying Sim2Real learning and present a systematic method for constructing simulation environments that yield in-domain synthetic data. An off-the-shelf 3D object detector is trained on HiFi DT-generated synthetic data and evaluated on real data. Our experiments show that the DT-trained model outperforms the equivalent model trained on real data by 4.8%. To understand this gain, we quantify distributional alignment between synthetic and real data using multiple metrics, including Chamfer Distance (CD), Maximum Mean Discrepancy (MMD), Earth Mover's Distance (EMD), and Fr'echet Distance (FD), at both raw-input and latent-feature levels. Results demonstrate that HiFi DTs substantially reduce domain shift and improve generalization across diverse evaluation scenarios. These findings underscore the significant role of digital twins in enabling reliable, simulation-based LiDAR perception for real-world ITS applications.

Related papers

• RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data [25.53943767088309]
  We introduce RealPDEBench, the first benchmark for scientific Machine Learning (ML) that integrates real-world measurements with paired numerical simulations.<n>RealPDEBench consists of five datasets, three tasks, eight metrics, and ten baselines.<n> Experiments reveal significant discrepancies between simulated and real-world data, while showing that pretraining with simulated data consistently improves both accuracy and convergence.
  arXiv  Detail & Related papers  (2026-01-05T06:49:13Z)
• UrbanTwin: Synthetic LiDAR Datasets (LUMPI, V2X-Real-IC, and TUMTraf-I) [3.1508266388327324]
  UrbanTwin datasets are high-fidelity, realistic replicas of three public roadside lidar datasets.<n>Each UrbanTwin dataset contains 10K frames corresponding to one of the public datasets.
  arXiv  Detail & Related papers  (2025-09-08T15:06:02Z)
• PercepTwin: Modeling High-Fidelity Digital Twins for Sim2Real LiDAR-based Perception for Intelligent Transportation Systems [3.1508266388327324]
  This paper introduces a rigorous methodology for creating large-scale, high-quality synthetic datasets using High-Fidelity Digital Twins (HiFi DTs)<n>The proposed workflow outlines the steps, tools, and best practices for digitally replicating real-world environments.
  arXiv  Detail & Related papers  (2025-09-03T00:12:15Z)
• How to Bridge the Sim-to-Real Gap in Digital Twin-Aided Telecommunication Networks [30.858857240474077]
  Training effective artificial intelligence models for telecommunications is challenging due to the scarcity of deployment-specific data.<n>Real data collection is expensive, and available datasets often fail to capture the unique operational conditions and contextual variability of the network environment.<n>Digital twinning provides a potential solution to this problem, as simulators tailored to the current network deployment can generate site-specific data to augment the available training datasets.
  arXiv  Detail & Related papers  (2025-07-09T17:27:51Z)
• Topology-Aware Modeling for Unsupervised Simulation-to-Reality Point Cloud Recognition [63.55828203989405]
  We introduce a novel Topology-Aware Modeling (TAM) framework for Sim2Real UDA on object point clouds.<n>Our approach mitigates the domain gap by leveraging global spatial topology, characterized by low-level, high-frequency 3D structures.<n>We propose an advanced self-training strategy that combines cross-domain contrastive learning with self-training.
  arXiv  Detail & Related papers  (2025-06-26T11:53:59Z)
• How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection [0.0]
  Event cameras are well-suited for real-time object detection at traffic intersections.<n>The development of robust event-based detection models is hindered by the limited availability of annotated real-world datasets.<n>This study offers the first quantifiable analysis of the sim-to-real gap in event-based object detection using CARLAs DVS.
  arXiv  Detail & Related papers  (2025-06-16T17:27:43Z)
• JiSAM: Alleviate Labeling Burden and Corner Case Problems in Autonomous Driving via Minimal Real-World Data [49.2298619289506]
  We propose a plug-and-play method called JiSAM, shorthand for Jittering augmentation, domain-aware backbone and memory-based Sectorized AlignMent.<n>In extensive experiments conducted on the famous AD dataset NuScenes, we demonstrate that, with SOTA 3D object detector, JiSAM is able to utilize the simulation data and only labels on 2.5% available real data to achieve comparable performance to models trained on all real data.
  arXiv  Detail & Related papers  (2025-03-11T13:35:39Z)
• Synth It Like KITTI: Synthetic Data Generation for Object Detection in Driving Scenarios [3.30184292168618]
  We propose a dataset generation pipeline based on the CARLA simulator for 3D object detection on LiDAR point clouds.<n>We are able to train an object detector on the synthetic data and demonstrate strong generalization capabilities to the KITTI dataset.
  arXiv  Detail & Related papers  (2025-02-20T22:27:42Z)
• Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer [0.0]
  We evaluate the performance of pre-trained vision encoders to address the Sim2Real gap.<n>We show that manipulation-pretrained encoders consistently achieve higher Action Scores.
  arXiv  Detail & Related papers  (2025-01-26T00:27:04Z)
• Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing [2.194575078433007]
  Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic
  arXiv  Detail & Related papers  (2024-04-29T10:37:38Z)
• RaSim: A Range-aware High-fidelity RGB-D Data Simulation Pipeline for Real-world Applications [55.24463002889]
  We focus on depth data synthesis and develop a range-aware RGB-D data simulation pipeline (RaSim) In particular, high-fidelity depth data is generated by imitating the imaging principle of real-world sensors. RaSim can be directly applied to real-world scenarios without any finetuning and excel at downstream RGB-D perception tasks.
  arXiv  Detail & Related papers  (2024-04-05T08:52:32Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• ePointDA: An End-to-End Simulation-to-Real Domain Adaptation Framework for LiDAR Point Cloud Segmentation [111.56730703473411]
  Training deep neural networks (DNNs) on LiDAR data requires large-scale point-wise annotations. Simulation-to-real domain adaptation (SRDA) trains a DNN using unlimited synthetic data with automatically generated labels. ePointDA consists of three modules: self-supervised dropout noise rendering, statistics-invariant and spatially-adaptive feature alignment, and transferable segmentation learning.
  arXiv  Detail & Related papers  (2020-09-07T23:46:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.