TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Driving

• URL: http://arxiv.org/abs/2607.04812v1
• Date: Mon, 06 Jul 2026 08:47:56 GMT
• Title: TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Driving
• Abstract summary: Birds-Eye View (BEV) end-to-end instance prediction has emerged as a robust paradigm for autonomous driving perception.<n>Current state-of-the-art approaches rely predominantly on geometric supervision, such as occupancy regression and optical flow.<n>We introduce Text-Guided Representation for Instance Prediction (TGRIP), a novel framework that bridges this gap by injecting rich semantic priors into the instance prediction loop.
• Score: 1.5327485092552822
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Bird's-Eye View (BEV) end-to-end instance prediction has emerged as a robust paradigm for autonomous driving perception, effectively mitigating the error propagation inherent in traditional modular pipelines. However, current state-of-the-art approaches rely predominantly on geometric supervision, such as occupancy regression and optical flow, effectively treating scene agents as generic moving obstacles. This absence of explicit semantic awareness imposes limitations on the capacity of the model to solve ambiguities in complex scenarios, particularly those where object-specific behavior is essential for accurate forecasting (e.g. overtaking, intersections). In this paper, we introduce Text-Guided Representation for Instance Prediction (TGRIP), a novel framework that bridges this gap by injecting rich semantic priors into the instance prediction loop. The proposed teacher-student pipeline employs Vision-Language Foundation Models to generate dense, semantic-enhanced BEV maps from multi-camera images. These maps serve as auxiliary supervision during training, guiding the network to learn spatio-temporal representations that are not only geometrically consistent but also semantically discriminative. To the best of our knowledge, this represents the first attempt to unify semantic guidance with the temporal task of future instance prediction. The experimental results demonstrate that TGRIP surpasses existing state-of-the-art models in nuScenes, validating the hypothesis that semantic enrichment is a fundamental element for robust, end-to-end motion prediction. Code is available on https://github.com/miguelag99/TGRIP.

Related papers

• ThinkingVLA: Interleaved Vision and Language Reasoning for Robotic Manipulation [71.69632267962993]
  We argue that manipulation planning naturally decomposes into prediction, anticipating the next visual state, and inverse dynamics.<n>We propose textbfThinkingVLA, a generative model that realizes this decomposition within a unified Mixture-of-Transformers architecture.<n>Experiments on simulation and real-world benchmarks demonstrate that ThinkingVLA consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T13:45:17Z)
• FLaRA: Predicting Future Latent Representations for Accident Anticipation [15.994549120609358]
  Anticipating traffic accidents from dashcam videos is a critical challenge in intelligent transportation systems.<n>We propose FLaRA, a novel predictive architecture that shifts this paradigm by forecasting future latent representations for accident anticipation.<n>Our approach achieves state-of-the-art performance while maintaining realistic early warning capabilities.
  arXiv  Detail & Related papers  (2026-06-12T12:16:45Z)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation [61.18260993245354]
  Chain-of-Thought (CoT) reasoning is a powerful driver of trajectory prediction in VLA-based autonomous driving.<n>We present OneVL, a unified VLA and World Model framework that routes reasoning through compact latent tokens supervised by dual auxiliary decoders.<n>OneVL becomes the first latent CoT method to surpass explicit CoT, delivering state-of-the-art accuracy at answer-only latency.
  arXiv  Detail & Related papers  (2026-04-20T16:37:22Z)
• Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction [12.661995278049764]
  We present Re2Pix, a hierarchical video prediction framework.<n>It decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis.<n>A key challenge arises from the train-test mismatch between ground-truth representations available during training and predicted ones used at inference.
  arXiv  Detail & Related papers  (2026-04-13T16:42:46Z)
• ForecastOcc: Vision-based Semantic Occupancy Forecasting [16.699381591572163]
  We present ForecastOcc, the first framework for vision-based semantic occupancy forecasting that predicts future occupancy states and semantic categories.<n>Our framework yields semantic occupancy forecasts for multiple horizons directly from past camera images, without relying on externally estimated maps.
  arXiv  Detail & Related papers  (2026-02-08T15:16:06Z)
• Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations [53.91818843831925]
  We propose NExT-Vid, a novel autoregressive visual generative pretraining framework.<n>We introduce a context-isolated autoregressive predictor to decouple semantic representation from target decoding.<n>Through context-isolated flow-matching pretraining, our approach achieves strong representations.
  arXiv  Detail & Related papers  (2025-12-24T07:07:08Z)
• GASP: Unifying Geometric and Semantic Self-Supervised Pre-training for Autonomous Driving [12.889523014369884]
  We propose a geometric and semantic self-supervised pre-training method, GASP, that learns a unified representation by predicting, at any queried future point in spacetime.<n>By modeling geometric and semantic 4D occupancy fields instead of raw sensor measurements, the model learns a structured, general representation of the environment and its evolution through time.
  arXiv  Detail & Related papers  (2025-03-19T20:00:27Z)
• Flow-guided Motion Prediction with Semantics and Dynamic Occupancy Grid Maps [5.9803668726235575]
  Occupancy Grid Maps (OGMs) are commonly employed for scene prediction. Recent studies have successfully combined OGMs with deep learning methods to predict the evolution of scene. We propose a novel multi-task framework that leverages dynamic OGMs and semantic information to predict both future vehicle semantic grids and the future flow of the scene.
  arXiv  Detail & Related papers  (2024-07-22T14:42:34Z)
• SEPT: Towards Efficient Scene Representation Learning for Motion Prediction [19.111948522155004]
  This paper presents SEPT, a modeling framework that leverages self-supervised learning to develop powerful models for complex traffic scenes. experiments demonstrate that SEPT, without elaborate architectural design or feature engineering, achieves state-of-the-art performance on the Argoverse 1 and Argoverse 2 motion forecasting benchmarks.
  arXiv  Detail & Related papers  (2023-09-26T21:56:03Z)
• Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation [66.86987509942607]
  We evaluate how such a paradigm should be done in imitation learning. We consider a setting where the pretraining corpus consists of multitask demonstrations. We argue that inverse dynamics modeling is well-suited to this setting.
  arXiv  Detail & Related papers  (2023-05-26T14:40:46Z)
• Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction [60.60223171143206]
  Trajectory prediction is a crucial undertaking in understanding entity movement or human behavior from observed sequences. Current methods often assume that the observed sequences are complete while ignoring the potential for missing values. This paper presents a unified framework, the Graph-based Conditional Variational Recurrent Neural Network (GC-VRNN), which can perform trajectory imputation and prediction simultaneously.
  arXiv  Detail & Related papers  (2023-03-28T14:27:27Z)
• Self-Supervision by Prediction for Object Discovery in Videos [62.87145010885044]
  In this paper, we use the prediction task as self-supervision and build a novel object-centric model for image sequence representation. Our framework can be trained without the help of any manual annotation or pretrained network. Initial experiments confirm that the proposed pipeline is a promising step towards object-centric video prediction.
  arXiv  Detail & Related papers  (2021-03-09T19:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.