HAVT-IVD: Heterogeneity-Aware Cross-Modal Network for Audio-Visual Surveillance: Idling Vehicles Detection With Multichannel Audio and Multiscale Visual Cues

• URL: http://arxiv.org/abs/2504.16102v2
• Date: Thu, 23 Oct 2025 20:30:54 GMT
• Title: HAVT-IVD: Heterogeneity-Aware Cross-Modal Network for Audio-Visual Surveillance: Idling Vehicles Detection With Multichannel Audio and Multiscale Visual Cues
• Abstract summary: Idling vehicle detection (IVD) uses surveillance video and multichannel audio to localize and classify vehicles in pick-up zones.<n>IVD faces three challenges: (i) modality heterogeneity between visual cues and audio patterns; (ii) large box scale variation requiring multi-resolution detection; and (iii) training instability due to coupled detection heads.<n>We propose HAVT-IVD, a heterogeneous-aware network with a visual feature pyramid and decoupled heads.
• Score: 1.0705399532413615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Idling vehicle detection (IVD) uses surveillance video and multichannel audio to localize and classify vehicles in the last frame as moving, idling, or engine-off in pick-up zones. IVD faces three challenges: (i) modality heterogeneity between visual cues and audio patterns; (ii) large box scale variation requiring multi-resolution detection; and (iii) training instability due to coupled detection heads. The previous end-to-end (E2E) model with simple CBAM-based bi-modal attention fails to handle these issues and often misses vehicles. We propose HAVT-IVD, a heterogeneity-aware network with a visual feature pyramid and decoupled heads. Experiments show HAVT-IVD improves mAP by 7.66 over the disjoint baseline and 9.42 over the E2E baseline.

Related papers

• V2XCrafter: Learning to Generate Driving Scene Across Agents [52.22336670088219]
  Collaborative driving systems leverage vehicle-to-everything (V2X) communication for collaborative perception to enhance driving safety.<n>We propose V2XCrafter, the first framework for generating controllable and realistic collaborative driving scene across agents' camera views.<n>Experiments have shown that V2XCrafter can generate high-fidelity and controllable street views with consistency across agents, thereby effectively enhancing the downstream collaborative 3D object detection tasks.
  arXiv  Detail & Related papers  (2026-05-28T07:03:51Z)
• Radar-Camera BEV Multi-Task Learning with Cross-Task Attention Bridge for Joint 3D Detection and Segmentation [0.6187780920448871]
  We propose textbfCTAB (Cross-Task Attention Bridge), a module that exchanges features between detection and segmentation branches.<n>On nuScenes, CTAB improves segmentation on 7 classes over the joint multi-task baseline at essentially neutral detection.<n>On a 4-class subset (drivable area, pedestrian crossing, walkway, vehicle), our joint multi-task model reaches comparable mIoU on 4 classes while simultaneously providing 3D detection.
  arXiv  Detail & Related papers  (2026-04-14T16:00:53Z)
• DDAVS: Disentangled Audio Semantics and Delayed Bidirectional Alignment for Audio-Visual Segmentation [33.30812530784719]
  Audio-Visual aims to localize objects at the pixel level by jointly leveraging auditory and visual information.<n>Existing methods often suffer from multi-source entanglement and audio-visual misalignment.<n>We propose Disentangled Audio Semantics and Delayed Bidirectional Alignment framework.
  arXiv  Detail & Related papers  (2025-12-23T07:21:21Z)
• A Tri-Modal Dataset and a Baseline System for Tracking Unmanned Aerial Vehicles [74.8162337823142]
  MM-UAV is the first large-scale benchmark for Multi-Modal UAV Tracking.<n>The dataset spans over 30 challenging scenarios, with 1,321 synchronised multi-modal sequences, and more than 2.8 million annotated frames.<n>Accompanying the dataset, we provide a novel multi-modal multi-UAV tracking framework.
  arXiv  Detail & Related papers  (2025-11-23T08:42:17Z)
• Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos [58.156141601478794]
  Multi-object tracking (UAVT) aims to track multiple objects while maintaining consistent identities across frames of a given video.<n>Existing methods typically model motion cues and appearance separately, overlooking their interplay and resulting in suboptimal tracking performance.<n>We propose AMOT, which exploits appearance and motion cues through two key components: an Appearance-Motion Consistency (AMC) matrix and a Motion-aware Track Continuation (MTC) module.
  arXiv  Detail & Related papers  (2025-08-03T12:06:47Z)
• Towards Intelligent Transportation with Pedestrians and Vehicles In-the-Loop: A Surveillance Video-Assisted Federated Digital Twin Framework [62.47416496137193]
  We propose a surveillance video assisted federated digital twin (SV-FDT) framework to empower ITSs with pedestrians and vehicles in-the-loop.<n>The architecture consists of three layers: (i) the end layer, which collects traffic surveillance videos from multiple sources; (ii) the edge layer, responsible for semantic segmentation-based visual understanding, twin agent-based interaction modeling, and local digital twin system (LDTS) creation in local regions; and (iii) the cloud layer, which integrates LDTSs across different regions to construct a global DT model in realtime.
  arXiv  Detail & Related papers  (2025-03-06T07:36:06Z)
• Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies [2.8517252798391177]
  Idling vehicle detection can be helpful in monitoring and reducing unnecessary idling. We introduce an end-to-end joint audio-visual IVD task. Unlike feature co-occurrence task such as audio-visual vehicle tracking, our IVD task addresses complementary features.
  arXiv  Detail & Related papers  (2024-10-28T16:13:44Z)
• EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving [64.58258341591929]
  Auditory Referring Multi-Object Tracking (AR-MOT) is a challenging problem in autonomous driving. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. We establish the first set of large-scale AR-MOT benchmarks.
  arXiv  Detail & Related papers  (2024-02-28T12:50:16Z)
• Vanishing-Point-Guided Video Semantic Segmentation of Driving Scenes [70.08318779492944]
  We are the first to harness vanishing point (VP) priors for more effective segmentation. Our novel, efficient network for VSS, named VPSeg, incorporates two modules that utilize exactly this pair of static and dynamic VP priors.
  arXiv  Detail & Related papers  (2024-01-27T01:01:58Z)
• Text-Driven Traffic Anomaly Detection with Temporal High-Frequency Modeling in Driving Videos [22.16190711818432]
  We introduce TTHF, a novel single-stage method aligning video clips with text prompts, offering a new perspective on traffic anomaly detection. Unlike previous approaches, the supervised signal of our method is derived from languages rather than one-hot vectors, providing a more comprehensive representation. It is shown that our proposed TTHF achieves promising performance, outperforming state-of-the-art competitors by +5.4% AUC on the DoTA dataset.
  arXiv  Detail & Related papers  (2024-01-07T15:47:19Z)
• Exploring Driving Behavior for Autonomous Vehicles Based on Gramian Angular Field Vision Transformer [12.398902878803034]
  This paper presents the Gramian Angular Field Vision Transformer (GAF-ViT) model, designed to analyze driving behavior. The proposed-ViT model consists of three key components: Transformer Module, Channel Attention Module, and Multi-Channel ViT Module.
  arXiv  Detail & Related papers  (2023-10-21T04:24:30Z)
• Leveraging Foundation models for Unsupervised Audio-Visual Segmentation [49.94366155560371]
  Audio-Visual (AVS) aims to precisely outline audible objects in a visual scene at the pixel level. Existing AVS methods require fine-grained annotations of audio-mask pairs in supervised learning fashion. We introduce unsupervised audio-visual segmentation with no need for task-specific data annotations and model training.
  arXiv  Detail & Related papers  (2023-09-13T05:05:47Z)
• SVDM: Single-View Diffusion Model for Pseudo-Stereo 3D Object Detection [0.0]
  A recently proposed framework for monocular 3D detection based on Pseudo-Stereo has received considerable attention in the community. In this work, we propose an end-to-end, efficient pseudo-stereo 3D detection framework by introducing a Single-View Diffusion Model. SVDM allows the entire pseudo-stereo 3D detection pipeline to be trained end-to-end and can benefit from the training of stereo detectors.
  arXiv  Detail & Related papers  (2023-07-05T13:10:37Z)
• M$^2$DAR: Multi-View Multi-Scale Driver Action Recognition with Vision Transformer [5.082919518353888]
  We present a multi-view, multi-scale framework for naturalistic driving action recognition and localization in untrimmed videos. Our system features a weight-sharing, multi-scale Transformer-based action recognition network that learns robust hierarchical representations.
  arXiv  Detail & Related papers  (2023-05-13T02:38:15Z)
• CROVIA: Seeing Drone Scenes from Car Perspective via Cross-View Adaptation [20.476683921252867]
  We propose a novel Cross-View Adaptation (CROVIA) approach to adapt the knowledge learned from on-road vehicle views to UAV views. First, a novel geometry-based constraint to cross-view adaptation is introduced based on the geometry correlation between views. Second, cross-view correlations from image space are effectively transferred to segmentation space without any requirement of paired on-road and UAV view data.
  arXiv  Detail & Related papers  (2023-04-14T15:20:40Z)
• CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers [36.838065731893735]
  CoBEVT is the first generic multi-agent perception framework that can cooperatively generate BEV map predictions. CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation.
  arXiv  Detail & Related papers  (2022-07-05T17:59:28Z)
• V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer [58.71845618090022]
  We build a holistic attention model, namely V2X-ViT, to fuse information across on-road agents. V2X-ViT consists of alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention. To validate our approach, we create a large-scale V2X perception dataset.
  arXiv  Detail & Related papers  (2022-03-20T20:18:25Z)
• Full-Duplex Strategy for Video Object Segmentation [141.43983376262815]
  Full- Strategy Network (FSNet) is a novel framework for video object segmentation (VOS) Our FSNet performs the crossmodal feature-passing (i.e., transmission and receiving) simultaneously before fusion decoding stage. We show that our FSNet outperforms other state-of-the-arts for both the VOS and video salient object detection tasks.
  arXiv  Detail & Related papers  (2021-08-06T14:50:50Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)
• Anchor-free Small-scale Multispectral Pedestrian Detection [88.7497134369344]
  We propose a method for effective and efficient multispectral fusion of the two modalities in an adapted single-stage anchor-free base architecture. We aim at learning pedestrian representations based on object center and scale rather than direct bounding box predictions. Results show our method's effectiveness in detecting small-scaled pedestrians.
  arXiv  Detail & Related papers  (2020-08-19T13:13:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.