When the City Teaches the Car: Label-Free 3D Perception from Infrastructure

• URL: http://arxiv.org/abs/2603.16742v1
• Date: Tue, 17 Mar 2026 16:21:06 GMT
• Title: When the City Teaches the Car: Label-Free 3D Perception from Infrastructure
• Abstract summary: Building robust 3D perception for self-driving still relies heavily on large-scale data collection and manual annotation.<n>Modern cities are increasingly instrumented with roadside units (RSUs), static sensors deployed along roads and at intersections to monitor traffic.<n>We propose infrastructure-taught, label-free 3D perception, a paradigm in which RSUs act as stationary, unsupervised teachers for ego vehicles.
• Score: 47.72611382185263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building robust 3D perception for self-driving still relies heavily on large-scale data collection and manual annotation, yet this paradigm becomes impractical as deployment expands across diverse cities and regions. Meanwhile, modern cities are increasingly instrumented with roadside units (RSUs), static sensors deployed along roads and at intersections to monitor traffic. This raises a natural question: can the city itself help train the vehicle? We propose infrastructure-taught, label-free 3D perception, a paradigm in which RSUs act as stationary, unsupervised teachers for ego vehicles. Leveraging their fixed viewpoints and repeated observations, RSUs learn local 3D detectors from unlabeled data and broadcast predictions to passing vehicles, which are aggregated as pseudo-label supervision for training a standalone ego detector. The resulting model requires no infrastructure or communication at test time. We instantiate this idea as a fully label-free three-stage pipeline and conduct a concept-and-feasibility study in a CARLA-based multi-agent environment. With CenterPoint, our pipeline achieves 82.3% AP for detecting vehicles, compared to a fully supervised ego upper bound of 94.4%. We further systematically analyze each stage, evaluate its scalability, and demonstrate complementarity with existing ego-centric label-free methods. Together, these results suggest that city infrastructure itself can potentially provide a scalable supervisory signal for autonomous vehicles, positioning infrastructure-taught learning as a promising orthogonal paradigm for reducing annotation cost in 3D perception.

Related papers

• Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting [13.375894985533838]
  Accurately predicting pedestrian motion is crucial for safe and reliable autonomous driving in complex urban environments.<n>We present a 3D vehicle-conditioned pedestrian pose forecasting framework that explicitly incorporates surrounding vehicle information.
  arXiv  Detail & Related papers  (2026-02-09T17:58:53Z)
• Inverse++: Vision-Centric 3D Semantic Occupancy Prediction Assisted with 3D Object Detection [11.33083039877258]
  3D semantic occupancy prediction aims to forecast detailed geometric and semantic information of the surrounding environment for autonomous vehicles.<n>We introduce an additional 3D supervision signal by incorporating an additional 3D object detection auxiliary branch.<n>Our approach attains state-of-the-art results, achieving an IoU score of 31.73% and a mIoU score of 20.91%.
  arXiv  Detail & Related papers  (2025-04-07T05:08:22Z)
• Unsupervised Adaptation from Repeated Traversals for Autonomous Driving [54.59577283226982]
  Self-driving cars must generalize to the end-user's environment to operate reliably. One potential solution is to leverage unlabeled data collected from the end-users' environments. There is no reliable signal in the target domain to supervise the adaptation process. We show that this simple additional assumption is sufficient to obtain a potent signal that allows us to perform iterative self-training of 3D object detectors on the target domain.
  arXiv  Detail & Related papers  (2023-03-27T15:07:55Z)
• HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving [95.42203932627102]
  3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. Our method efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages.
  arXiv  Detail & Related papers  (2022-12-15T11:15:14Z)
• Exploring Contextual Representation and Multi-Modality for End-to-End Autonomous Driving [58.879758550901364]
  Recent perception systems enhance spatial understanding with sensor fusion but often lack full environmental context. We introduce a framework that integrates three cameras to emulate the human field of view, coupled with top-down bird-eye-view semantic data to enhance contextual representation. Our method achieves displacement error by 0.67m in open-loop settings, surpassing current methods by 6.9% on the nuScenes dataset.
  arXiv  Detail & Related papers  (2022-10-13T05:56:20Z)
• Real-Time And Robust 3D Object Detection with Roadside LiDARs [20.10416681832639]
  We design a 3D object detection model that can detect traffic participants in roadside LiDARs in real-time. Our model uses an existing 3D detector as a baseline and improves its accuracy. We make a significant contribution with our LiDAR-based 3D detector that can be used for smart city applications.
  arXiv  Detail & Related papers  (2022-07-11T21:33:42Z)
• ONCE-3DLanes: Building Monocular 3D Lane Detection [41.46466150783367]
  We present ONCE-3DLanes, a real-world autonomous driving dataset with lane layout annotation in 3D space. By exploiting the explicit relationship between point clouds and image pixels, a dataset annotation pipeline is designed to automatically generate high-quality 3D lane locations.
  arXiv  Detail & Related papers  (2022-04-30T16:35:25Z)
• Urban Traffic Surveillance (UTS): A fully probabilistic 3D tracking approach based on 2D detections [11.34426502082293]
  Urban Traffic Surveillance (UTS) is a surveillance system based on a monocular and calibrated video camera. UTS tracks the vehicles using a 3D bounding box representation and a physically reasonable 3D motion model.
  arXiv  Detail & Related papers  (2021-05-31T14:29:02Z)
• End-to-end Interpretable Neural Motion Planner [78.69295676456085]
  We propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios. We design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America.
  arXiv  Detail & Related papers  (2021-01-17T14:16:12Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.