Pictura: Perspective-View Self-Play at Scale for Driving

• URL: http://arxiv.org/abs/2607.26005v1
• Date: Tue, 28 Jul 2026 17:20:39 GMT
• Title: Pictura: Perspective-View Self-Play at Scale for Driving
• Abstract summary: We introduce Pictura, a GPU-accelerated driving simulator that renders each agent's egocentric view at every step.<n>Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100.<n>It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations.
• Score: 80.56728858364484
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-play in simulation produces robust driving policies at scale. Demonstrations of such behavior have been made using privileged vectorized observations such as exact poses and velocities, even for occluded agents. This assumes that perception is solved and introduces a representation gap with the partial observation of a deployed agent driving from the perspective view of egocentric cameras. A common fix, distilling the privileged policy into a camera-input student, leaves the student imitating decisions its own view cannot justify. Instead, we establish perspective-view self-play as a practical training regime. We introduce Pictura, a GPU-accelerated multi-agent driving simulator that renders each agent's egocentric view at every step, mitigating the representation gap at its source. Pictura sustains up to 500K agent-steps/s (2M images/s) on a single H100. Using Pictura, we train Alberti by self-play with plain PPO. It is the first large-scale driving self-play policy trained directly from perspective images, without privileged observations. Training spans 50B agent steps for ~35M km of driving. It approaches the driving performance of its privileged vectorized counterpart, and transfers zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura, where it outperforms privileged vectorized agents. Project page: https://valeoai.github.io/Pictura/

Related papers

• TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale [18.077282979391423]
  TerraZero is a procedural driving simulator and self-play training stack.<n>A C engine runs simulation on the CPU and policy inference on the GPU over a zero-copy path.<n>Every reported policy trains from scratch by reinforcement learning alone on a compute-efficient self-play recipe.
  arXiv  Detail & Related papers  (2026-07-14T17:59:02Z)
• Scaling Self-Play for End-to-End Driving [40.15606566638922]
  Gigapixel is a high- throughput batched driving simulator with perspective rendering.<n>We train pixel-based policies in self-play via on-policy distillation from a privileged RL teacher.<n>We transfer the self-play trained policies to real-world sensor data through lightweight perception adaptation.
  arXiv  Detail & Related papers  (2026-06-17T22:39:11Z)
• Robust Autonomy Emerges from Self-Play [63.44745854476206]
  We show that robust and naturalistic driving emerges entirely from self-play in simulation at unprecedented scale.<n>This is enabled by Gigaflow, a batched simulator that can synthesize and train on 42 years of subjective driving experience per hour.
  arXiv  Detail & Related papers  (2025-02-05T16:41:05Z)
• HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving [48.84595398410479]
  HUGSIM is a closed-loop, photo-realistic, and real-time simulator for evaluating autonomous driving algorithms.<n>We tackle challenges of novel view synthesis in closed-loop scenarios, including viewpoint extrapolation and 360-degree vehicle rendering.<n> HUGSIM offers a comprehensive benchmark across more than 70 sequences from KITTI-360, nuScenes, and PandaSet, along with over 400 varying scenarios.
  arXiv  Detail & Related papers  (2024-12-02T17:07:59Z)
• Linking vision and motion for self-supervised object-centric perception [16.821130222597155]
  Object-centric representations enable autonomous driving algorithms to reason about interactions between many independent agents and scene features. Traditionally these representations have been obtained via supervised learning, but this decouples perception from the downstream driving task and could harm generalization. We adapt a self-supervised object-centric vision model to perform object decomposition using only RGB video and the pose of the vehicle as inputs.
  arXiv  Detail & Related papers  (2023-07-14T04:21:05Z)
• Policy Pre-training for End-to-end Autonomous Driving via Self-supervised Geometric Modeling [96.31941517446859]
  We propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
  arXiv  Detail & Related papers  (2023-01-03T08:52:49Z)
• Estimation of Appearance and Occupancy Information in Birds Eye View from Surround Monocular Images [2.69840007334476]
  Birds-eye View (BEV) expresses the location of different traffic participants in the ego vehicle frame from a top-down view. We propose a novel representation that captures various traffic participants appearance and occupancy information from an array of monocular cameras covering 360 deg field of view (FOV) We use a learned image embedding of all camera images to generate a BEV of the scene at any instant that captures both appearance and occupancy of the scene.
  arXiv  Detail & Related papers  (2022-11-08T20:57:56Z)
• Self-Supervised Steering Angle Prediction for Vehicle Control Using Visual Odometry [55.11913183006984]
  We show how a model can be trained to control a vehicle's trajectory using camera poses estimated through visual odometry methods. We propose a scalable framework that leverages trajectory information from several different runs using a camera setup placed at the front of a car.
  arXiv  Detail & Related papers  (2021-03-20T16:29:01Z)
• Learning by Cheating [72.9701333689606]
  We show that this challenging learning problem can be simplified by decomposing it into two stages. We use the presented approach to train a vision-based autonomous driving system that substantially outperforms the state of the art. Our approach achieves, for the first time, 100% success rate on all tasks in the original CARLA benchmark, sets a new record on the NoCrash benchmark, and reduces the frequency of infractions by an order of magnitude compared to the prior state of the art.
  arXiv  Detail & Related papers  (2019-12-27T18:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.