Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction

• URL: http://arxiv.org/abs/2607.12293v1
• Date: Tue, 14 Jul 2026 03:05:09 GMT
• Title: Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction
• Abstract summary: ADAPT (Adaptive Domain-Aware Pedestrian Crossing Transformer) is a multimodal framework for predicting pedestrian crossing intention.<n> ADAPT processes four aligned visual modalities, including RGB images, local depth maps, global semantic maps, and global depth maps.<n>Experiments on the JAAD and PIE benchmark datasets demonstrate that ADAPT consistently outperforms existing state-of-the-art methods.
• Score: 4.8915598762204135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting pedestrian crossing intention is a safety-critical task for autonomous driving, yet existing approaches often rely on single-modal inputs or dense multimodal fusion strategies that inadequately capture complementary visual and kinematic information while introducing redundant inter-modal interactions. We propose ADAPT (Adaptive Domain-Aware Pedestrian Crossing Transformer), a multimodal framework that jointly models local and global visual context together with temporal motion dynamics for accurate pedestrian crossing intention prediction. ADAPT processes four spatially aligned visual modalities, including RGB images, local depth maps, global semantic maps, and global depth maps, together with ego-vehicle speed, pedestrian bounding boxes, and skeleton pose information through five specialized modules: a weight-shared Swin Transformer V2 backbone for visual feature extraction, a Cross-Modality Guided Attention module for hierarchical visual fusion, a Mamba-based Motion Feature Encoding module for efficient temporal modeling, a Sparse Cross-Modal Attention module that selectively preserves the most informative inter-modal interactions, and a Vision Transformer-based Temporal Feature Fusion module for sequence-level prediction. Extensive experiments on the JAAD and PIE benchmark datasets demonstrate that ADAPT consistently outperforms existing state-of-the-art methods while maintaining low computational complexity. On JAAD, the proposed method achieves an AUC of 0.73 on JAADbeh and 0.85 on JAADall, while on PIE it achieves an accuracy of 0.92 and an AUC of 0.90. Furthermore, ADAPT performs inference in only 17.23 ms per sample, offering an effective balance between predictive accuracy and real-time deployment efficiency for intelligent transportation and autonomous driving applications.

Related papers

• StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models [84.17128030384099]
  StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
  arXiv  Detail & Related papers  (2026-08-26T17:33:19Z)
• BMCR: Adaptive Backbone Module Composition via Reinforcement Learning for Remote Sensing Object Detection [4.303688954031907]
  We propose Backbone Module Composition via Reinforcement Learning (BMCR) for object detection detectors.<n> BMCR assembles input-adaptive inference paths from reusable modules decomposed from off-the-shelf CNN and ViT backbones.<n>We show that BMCR achieves 79.31%, 73.41% and 71.86% mAP, respectively, surpassing strong static and dynamic baselines by up to 2.5 points.
  arXiv  Detail & Related papers  (2026-06-04T02:04:15Z)
• DM$^3$T: Harmonizing Modalities via Diffusion for Multi-Object Tracking [10.270441242480482]
  This paper proposes DM$3$T, a novel framework that reformulates multimodal fusion as an iterative feature alignment process.<n>Our approach performs iterative cross-modal harmonization through a proposed Cross-Modal Diffusion Fusion (C-MDF) module.<n>To further improve tracking robustness, we design a Hierarchical Tracker that adaptively handles confidence estimation.
  arXiv  Detail & Related papers  (2025-11-28T06:02:58Z)
• ACIT: Attention-Guided Cross-Modal Interaction Transformer for Pedestrian Crossing Intention Prediction [3.878105750489656]
  This paper proposes an attention-guided cross-modal interaction Transformer (ACIT) for pedestrian crossing intention prediction.<n>ACIT leverages six visual and motion modalities, which are grouped into three interaction pairs.<n>ACIT outperforms state-of-the-art methods, achieving accuracy rates of 70% and 89% on the JAADbeh and JAADall datasets.
  arXiv  Detail & Related papers  (2025-11-25T07:41:11Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• Trajeglish: Traffic Modeling as Next-Token Prediction [67.28197954427638]
  A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs. We apply tools from discrete sequence modeling to model how vehicles, pedestrians and cyclists interact in driving scenarios. Our model tops the Sim Agents Benchmark, surpassing prior work along the realism meta metric by 3.3% and along the interaction metric by 9.9%.
  arXiv  Detail & Related papers  (2023-12-07T18:53:27Z)
• Ret3D: Rethinking Object Relations for Efficient 3D Object Detection in Driving Scenes [82.4186966781934]
  We introduce a simple, efficient, and effective two-stage detector, termed as Ret3D. At the core of Ret3D is the utilization of novel intra-frame and inter-frame relation modules. With negligible extra overhead, Ret3D achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-18T03:48:58Z)
• Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking [59.79252390626194]
  We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects. The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
  arXiv  Detail & Related papers  (2022-05-31T01:19:18Z)
• Multi-Modal Fusion Transformer for End-to-End Autonomous Driving [59.60483620730437]
  We propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.
  arXiv  Detail & Related papers  (2021-04-19T11:48:13Z)
• Shared Cross-Modal Trajectory Prediction for Autonomous Driving [24.07872495811019]
  We propose a Cross-Modal Embedding framework that aims to benefit from the use of multiple input modalities. An extensive evaluation is conducted to show the efficacy of the proposed framework using two benchmark driving datasets.
  arXiv  Detail & Related papers  (2020-11-15T07:18:50Z)
• Shared Cross-Modal Trajectory Prediction for Autonomous Driving [24.07872495811019]
  We propose a Cross-Modal Embedding framework that aims to benefit from the use of multiple input modalities. An extensive evaluation is conducted to show the efficacy of the proposed framework using two benchmark driving datasets.
  arXiv  Detail & Related papers  (2020-04-01T02:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.