TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations

• URL: http://arxiv.org/abs/2606.17386v1
• Date: Tue, 16 Jun 2026 00:45:01 GMT
• Title: TerraTransfer: Learning End-to-End Driving Policies Without Expert Demonstrations
• Abstract summary: We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss.<n>On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.
• Score: 21.33643567755425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all stages: collecting and labeling millions of driving frames is costly, and closed-loop RL on images is bottlenecked by the per-step cost of photorealistic rendering plus a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: millions of rollout steps per second, and a state distribution naturally rich in collisions, near-misses, and recoveries that no driving log contains. Our approach exploits this asymmetry by decoupling learning to drive from learning to see. We pretrain a single policy by self-play, then align its latent space with a pretrained vision backbone, through the action KL divergence and a batch-relational low-rank structural loss. The action target comes from the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. On photorealistic 3D Gaussian splatting closed-loop scenarios, the resulting end-to-end policy matches or exceeds prior end-to-end methods.

Related papers

• PAVE: Predictive Alignment and Value-Guided Evolution for World-Action Policies [17.87859739145508]
  Direct vision-language-action policies generate continuous robot actions efficiently, but standard behavior cloning leaves two complementary gaps.<n>We introduce method, a direct world-action policy that combines outcome-agnostic predictive learning with outcome-aware policy improvement.
  arXiv  Detail & Related papers  (2026-08-31T07:36:11Z)
• Scaling Self-Play for End-to-End Driving [40.15606566638922]
  Gigapixel is a high- throughput batched driving simulator with perspective rendering.<n>We train pixel-based policies in self-play via on-policy distillation from a privileged RL teacher.<n>We transfer the self-play trained policies to real-world sensor data through lightweight perception adaptation.
  arXiv  Detail & Related papers  (2026-06-17T22:39:11Z)
• SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation [73.2142090645987]
  SC3-Eval is a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator.<n>We demonstrate SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison.
  arXiv  Detail & Related papers  (2026-06-17T02:15:46Z)
• Perfect Demo Makes Poor Teacher: Learning Robust Alignment from Critical Motion Segments [53.85970147743299]
  Expert demonstrations are widely assumed to be the gold standard for robot imitation learning.<n>Yet for fine-grained manipulation such as insertion, stacking, and alignment, we uncover a counterintuitive failure mode: fluent demonstrations can be poor teachers.<n>We introduce STAIR, a compact dynamic feature that bridges the vision-language model and the action expert, distilling the short-horizon motion already recorded in each trajectory into dense, motion-aware supervision.
  arXiv  Detail & Related papers  (2026-06-14T04:15:29Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving [19.081114003415863]
  We propose a unified understanding-and-generation framework to enable meaningful exploration and provide dense supervision.<n>We augment trajectory prediction with future RGB and depth image generation as dense world modeling objectives.<n>We incorporate this exploration signal into a safety-gated reward and optimize the policy.
  arXiv  Detail & Related papers  (2026-04-03T04:14:13Z)
• Optimization-Guided Diffusion for Interactive Scene Generation [52.23368750264419]
  We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling.<n>We show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes.<n>Our approach can also generate $5times$ more near-collision frames with a time-to-collision under three seconds.
  arXiv  Detail & Related papers  (2025-12-08T15:56:18Z)
• XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis [84.23233209017192]
  This paper presents a synthetic dataset for novel driving view synthesis evaluation.<n>It includes testing images captured by deviating from the training trajectory by $1-4$ meters.<n>We establish the first realistic benchmark for evaluating existing NVS approaches under front-only and multicamera settings.
  arXiv  Detail & Related papers  (2024-06-26T14:00:21Z)
• Enhancing End-to-End Autonomous Driving with Latent World Model [78.22157677787239]
  We propose a novel self-supervised learning approach using the LAtent World model (LAW) for end-to-end driving.<n> LAW predicts future scene features based on current features and ego trajectories.<n>This self-supervised task can be seamlessly integrated into perception-free and perception-based frameworks.
  arXiv  Detail & Related papers  (2024-06-12T17:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.