End-to-End 3D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration

• URL: http://arxiv.org/abs/2512.21831v1
• Date: Fri, 26 Dec 2025 02:20:22 GMT
• Title: End-to-End 3D Spatiotemporal Perception with Multimodal Fusion and V2X Collaboration
• Authors: Zhenwei Yang, Yibo Ai, Weidong Zhang,
• Abstract summary: X-2V2X is a multi-modal fused end-to-end framework for v2x collaboration.<n>It unifies multi-view multimodal sensing within shared representation.<n>X-V2X achieves robust and temporally stable perception in complex traffic scenarios.
• Score: 7.235781104512231
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-view cooperative perception and multimodal fusion are essential for reliable 3D spatiotemporal understanding in autonomous driving, especially under occlusions, limited viewpoints, and communication delays in V2X scenarios. This paper proposes XET-V2X, a multi-modal fused end-to-end tracking framework for v2x collaboration that unifies multi-view multimodal sensing within a shared spatiotemporal representation. To efficiently align heterogeneous viewpoints and modalities, XET-V2X introduces a dual-layer spatial cross-attention module based on multi-scale deformable attention. Multi-view image features are first aggregated to enhance semantic consistency, followed by point cloud fusion guided by the updated spatial queries, enabling effective cross-modal interaction while reducing computational overhead. Experiments on the real-world V2X-Seq-SPD dataset and the simulated V2X-Sim-V2V and V2X-Sim-V2I benchmarks demonstrate consistent improvements in detection and tracking performance under varying communication delays. Both quantitative results and qualitative visualizations indicate that XET-V2X achieves robust and temporally stable perception in complex traffic scenarios.

Related papers

• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• X-ReID: Multi-granularity Information Interaction for Video-Based Visible-Infrared Person Re-Identification [79.37768038337971]
  We propose a novel cross-modality feature learning framework named X-ReID for VVI-ReID.<n> Specifically, we first propose a Cross-modality Prototype Collaboration (CPC)<n>Then, a Multi-granularity Information Interaction (MII) is designed, incorporating short-term interactions from adjacent frames, long-term cross-frame information fusion, and cross-modality feature alignment.
  arXiv  Detail & Related papers  (2025-11-22T07:57:15Z)
• DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer [56.98400572837792]
  DiVE produces high-fidelity, temporally coherent, and cross-view consistent multi-view videos.<n>These innovations collectively achieve a 2.62x speedup with minimal quality degradation.
  arXiv  Detail & Related papers  (2025-04-28T09:20:50Z)
• LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving [88.85002707211777]
  LargeAD is a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets.<n>Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds.<n>This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data.
  arXiv  Detail & Related papers  (2025-01-07T18:59:59Z)
• V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction [44.40410127660706]
  Vehicle-to-everything (V2X) technologies offer the limitations of constrained observability in single-vehicle systems.<n>We focus on one-step and multi-step communication strategies (when to transmit) as well as examine their integration with three fusion strategies.<n>Our framework outperforms state-of-the-art methods in both perception and prediction tasks.
  arXiv  Detail & Related papers  (2024-12-02T18:55:34Z)
• CooPre: Cooperative Pretraining for V2X Cooperative Perception [47.00472259100765]
  CooPre is a self-supervised learning framwork for V2X cooperative perception.<n>We develop a V2X bird-eye-view (BEV) guided masking strategy which effectively allows the model to pay attention to 3D features across heterogeneous V2X agents.<n>CooPre achieves a 4% mAP improvement on V2X-Real dataset and surpasses baseline performance using only 50% of the training data.
  arXiv  Detail & Related papers  (2024-08-20T23:39:26Z)
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer [58.71845618090022]
  We build a holistic attention model, namely V2X-ViT, to fuse information across on-road agents. V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention. To validate our approach, we create a large-scale V2X perception dataset.
  arXiv  Detail & Related papers  (2022-03-20T20:18:25Z)
• V2X-Sim: A Virtual Collaborative Perception Dataset for Autonomous Driving [26.961213523096948]
  Vehicle-to-everything (V2X) denotes the collaboration between a vehicle and any entity in its surrounding. We present the V2X-Sim dataset, the first public large-scale collaborative perception dataset in autonomous driving.
  arXiv  Detail & Related papers  (2022-02-17T05:14:02Z)
• Full-Duplex Strategy for Video Object Segmentation [141.43983376262815]
  Full- Strategy Network (FSNet) is a novel framework for video object segmentation (VOS) Our FSNet performs the crossmodal feature-passing (i.e., transmission and receiving) simultaneously before fusion decoding stage. We show that our FSNet outperforms other state-of-the-arts for both the VOS and video salient object detection tasks.
  arXiv  Detail & Related papers  (2021-08-06T14:50:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.