Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024

• URL: http://arxiv.org/abs/2412.07247v1
• Date: Tue, 10 Dec 2024 07:13:39 GMT
• Title: Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024
• Abstract summary: This report describes the methods we employed for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge.<n>We utilized a powerful open-source multimodal model, InternVL-1.5, and conducted a full-fledged fine-tuning on the competition dataset, DriveLM-nuScenes.<n>Our single model achieved a score of 0.6002 on the final leadboard.
• Score: 23.193095382776725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This technical report describes the methods we employed for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. We utilized a powerful open-source multimodal model, InternVL-1.5, and conducted a full-parameter fine-tuning on the competition dataset, DriveLM-nuScenes. To effectively handle the multi-view images of nuScenes and seamlessly inherit InternVL's outstanding multimodal understanding capabilities, we formatted and concatenated the multi-view images in a specific manner. This ensured that the final model could meet the specific requirements of the competition task while leveraging InternVL's powerful image understanding capabilities. Meanwhile, we designed a simple automatic annotation strategy that converts the center points of objects in DriveLM-nuScenes into corresponding bounding boxes. As a result, our single model achieved a score of 0.6002 on the final leadboard.

Related papers

• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion [39.303609347179695]
  This study proposes MMDrive, a vision-language model framework that extends traditional image understanding to a generalized 3D scene understanding framework.<n> MMDrive incorporates three complementary modalities, including occupancy maps, LiDAR point clouds, and textual scene descriptions.<n> MMDrive achieves significant performance gains over existing vision-language models for autonomous driving, with a BLEU-4 score of 54.56 and METEOR of 41.78 on DriveLM, and an accuracy score of 62.7% on NuScenes-QA.
  arXiv  Detail & Related papers  (2025-12-15T10:37:59Z)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation [98.47375190901447]
  We present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation.<n>WeAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images.<n>WeAVEBench is a human-annotated benchmark with 100 tasks based on 480 images.
  arXiv  Detail & Related papers  (2025-11-14T16:02:38Z)
• The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge [10.577222594790333]
  This report outlines our approach using vision language model systems for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge.<n>We have exclusively utilized the DriveLM-nuScenes dataset for training our models.<n>We have integrated depth information from open-source depth estimation models to enrich the training and inference processes.
  arXiv  Detail & Related papers  (2025-09-14T03:37:17Z)
• 2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model [21.811872482011534]
  We show that combining end-to-end architectural design and knowledgeable VLMs yield impressive performance on the driving tasks.<n>It is worth noting that our method only uses a single camera and is the best camera-only solution across the leaderboard.
  arXiv  Detail & Related papers  (2025-09-02T17:52:29Z)
• LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving [58.535516533697425]
  Large vision-language models (VLMs) have shown promising capabilities in scene understanding.<n>We propose a novel vision-language framework tailored for autonomous driving, called LMAD.<n>Our framework emulates modern end-to-end driving paradigms by incorporating comprehensive scene understanding and a task-specialized structure with VLMs.
  arXiv  Detail & Related papers  (2025-08-17T15:42:54Z)
• TS-VLM: Text-Guided SoftSort Pooling for Vision-Language Models in Multi-View Driving Reasoning [1.4087249809872509]
  Vision-Language Models (VLMs) have shown potential to enhance scene perception, reasoning, and decision-making.<n>Existing models suffer from computational overhead and inefficient integration of multi-view sensor data.<n>This paper introduces a lightweight VLM called TS-VLM, which incorporates a novel Text-Guided SoftSort Pooling (TGSSP) module.
  arXiv  Detail & Related papers  (2025-05-19T03:37:15Z)
• Precise Drive with VLM: First Prize Solution for PRCV 2024 Drive LM challenge [8.941623670652389]
  This report outlines the methodologies we applied for the PRCV Challenge. It focuses on cognition and decision-making in driving scenarios. Our model achieved a score of 0.6064, securing the first prize on the competition's final results.
  arXiv  Detail & Related papers  (2024-11-05T11:00:55Z)
• TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning [54.033346088090674]
  We introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process.
  arXiv  Detail & Related papers  (2024-10-14T13:35:47Z)
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites [114.22835695929682]
  InternVL 1.5 is an open-source multimodal large language model (MLLM) It bridges the capability gap between open-source and proprietary commercial models in multimodal understanding.
  arXiv  Detail & Related papers  (2024-04-25T17:59:19Z)
• Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving [0.0]
  We develop an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations.
  arXiv  Detail & Related papers  (2024-03-28T21:18:33Z)
• Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model [83.85856356798531]
  VistaLLM is a visual system that addresses coarse- and fine-grained vision-language tasks. It employs a gradient-aware adaptive sampling technique to represent binary segmentation masks as sequences. We also introduce a novel task, AttCoSeg, which boosts the model's reasoning and grounding capability over multiple input images.
  arXiv  Detail & Related papers  (2023-12-19T18:53:01Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• Video Task Decathlon: Unifying Image and Video Tasks in Autonomous Driving [85.62076860189116]
  Video Task Decathlon (VTD) includes ten representative image and video tasks spanning classification, segmentation, localization, and association of objects and pixels. We develop our unified network, VTDNet, that uses a single structure and a single set of weights for all ten tasks.
  arXiv  Detail & Related papers  (2023-09-08T16:33:27Z)
• 1st Place in ICCV 2023 Workshop Challenge Track 1 on Resource Efficient Deep Learning for Computer Vision: Budgeted Model Training Challenge [15.213786895534225]
  We describe a resource-aware backbone search framework composed of profile and instantiation phases. We employ multi-resolution ensembles to boost inference accuracy on limited resources. Based on our approach, we win first place in International Conference on Computer Vision (ICCV) 2023 Workshop Challenge Track 1 on Resource Efficient Deep Learning for Computer Vision (RCV)
  arXiv  Detail & Related papers  (2023-08-09T05:38:18Z)
• Q-YOLOP: Quantization-aware You Only Look Once for Panoptic Driving Perception [6.3709120604927945]
  We present an efficient and quantization-aware panoptic driving perception model (Q- YOLOP) for object detection, drivable area segmentation, and lane line segmentation. The proposed model achieves state-of-the-art performance with an mAP@0.5 of 0.622 for object detection and an mIoU of 0.612 for segmentation.
  arXiv  Detail & Related papers  (2023-07-10T13:02:46Z)
• Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving [100.3848723827869]
  We present an effective multi-task framework, VE-Prompt, which introduces visual exemplars via task-specific prompting. Specifically, we generate visual exemplars based on bounding boxes and color-based markers, which provide accurate visual appearances of target categories. We bridge transformer-based encoders and convolutional layers for efficient and accurate unified perception in autonomous driving.
  arXiv  Detail & Related papers  (2023-03-03T08:54:06Z)
• CERBERUS: Simple and Effective All-In-One Automotive Perception Model with Multi Task Learning [4.622165486890318]
  In-vehicle embedded computing platforms cannot cope with the computational effort required to run a heavy model for each individual task. We present CERBERUS, a lightweight model that leverages a multitask-learning approach to enable the execution of multiple perception tasks at the cost of a single inference.
  arXiv  Detail & Related papers  (2022-10-03T08:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.