CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction

• URL: http://arxiv.org/abs/2609.00242v1
• Date: Mon, 31 Aug 2026 18:44:30 GMT
• Title: CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction
• Abstract summary: Long-tail autonomous driving failures are often framed as rare-object recognition errors.<n>We formalize this problem as decision-level driving affordance prediction, where a model maps a front-view image, ego-motion history, and navigation command to a structured longitudinal-lateral meta-action.<n>To improve deployable small VLMs, we propose KPA, a knowledge-preserving adaptation framework that combines structured perception-to-decision prompting, SLERP-based expert merging, and RegMoE, a regime-aware LoRA mixture-of-experts module.
• Score: 27.20821192335003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-tail autonomous driving failures are often framed as rare-object recognition errors. We argue that this view is incomplete: the decision-critical question is not only whether a model recognizes an unusual object, but whether it infers how that object changes the ego vehicle's feasible high-level actions. We formalize this problem as decision-level driving affordance prediction, where a model maps a front-view image, ego-motion history, and navigation command to a structured longitudinal--lateral meta-action. To evaluate this capability, we introduce CoLT-Drive, a 3,536-sample counterfactual long-tail benchmark that inserts rare objects into otherwise fixed driving scenes and measures whether models predict acceptable action pairs. To improve deployable small VLMs, we propose KPA, a knowledge-preserving adaptation framework that combines structured perception-to-decision prompting, SLERP-based expert merging, and RegMoE, a regime-aware LoRA mixture-of-experts module. KPA preserves the pretrained model's open-world knowledge while allocating lightweight adaptation capacity to different driving decision regimes. Experiments on an in-domain driving split and CoLT-Drive show that KPA achieves 60.8\% pair accuracy on CoLT-Drive, outperforming the pretrained Qwen3-VL-2B baseline (50.3\%) and LoRA SFT (32.4\%) while maintaining competitive in-domain accuracy. Our benchmark and code are available at https://huggingface.co/datasets/tangzx2024/CoLT-Drive and https://github.com/tangzhengxu/CoLT-Drive.

Related papers

• DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving [39.60553851234952]
  Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving.<n>We propose DriveVLA-M0, a retrieval-augmented VLA with failure-aware latent memory.<n>We show that DriveVLA-M0 scales effectively with additional memory, enabling training-free performance gains through memory expansion.
  arXiv  Detail & Related papers  (2026-08-11T03:01:46Z)
• PerceptDrive: Perception Prior World-Action Modeling with Adaptive Expert Routing for End-to-End Autonomous Driving [20.924755586440362]
  PerceptDrive is a perception prior world-action modeling framework with adaptive expert routing.<n>PerceptDrive feeds teacher-distilled priors from a frozen, driving-adapted provider.<n>A prior-retention objective anchors each branch to its prior.
  arXiv  Detail & Related papers  (2026-07-22T14:09:15Z)
• CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving [19.474428775260034]
  We propose CLEAR, a framework that combines ultra-fast generative planning with deep semantic reasoning.<n> CLEAR employs Drive-JEPA as the visual encoder and replaces the multi-step denoising chain with a single-step conditional drift in a VAE latent space.<n>On the NAVSIM v1 benchmark, CLEAR achieves a state-of-the-art PDMS of 93.7.
  arXiv  Detail & Related papers  (2026-06-04T14:32:10Z)
• Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving [54.31800246594724]
  We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units.<n>We show that Fast-dDrive redefines the speed-accuracy frontier for driving agents.
  arXiv  Detail & Related papers  (2026-05-22T02:31:32Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving [48.461267171124945]
  We propose UniDriveVLA, a Unified Driving Vision-Language-Action model for autonomous driving.<n>It comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention.<n>It achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive.
  arXiv  Detail & Related papers  (2026-04-02T15:48:45Z)
• CoReVLA: A Dual-Stage End-to-End Autonomous Driving Framework for Long-Tail Scenarios via Collect-and-Refine [73.74077186298523]
  CoReVLA is a continual learning framework for autonomous driving.<n>It improves the performance in long-tail scenarios through a dual-stage process of data Collection and behavior Refinement.<n>CoReVLA achieves a Driving Score (DS) of 72.18 and a Success Rate (SR) of 50%, outperforming state-of-the-art methods by 7.96 DS and 15% SR under long-tail, safety-critical scenarios.
  arXiv  Detail & Related papers  (2025-09-19T13:25:56Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• Towards Interactive and Learnable Cooperative Driving Automation: a Large Language Model-Driven Decision-Making Framework [87.7482313774741]
  Connected Autonomous Vehicles (CAVs) have begun to open road testing around the world, but their safety and efficiency performance in complex scenarios is still not satisfactory.<n>This paper proposes CoDrivingLLM, an interactive and learnable LLM-driven cooperative driving framework.
  arXiv  Detail & Related papers  (2024-09-19T14:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.