Cyclic Refiner: Object-Aware Temporal Representation Learning for Multi-View 3D Detection and Tracking

• URL: http://arxiv.org/abs/2407.03240v1
• Date: Wed, 3 Jul 2024 16:10:19 GMT
• Title: Cyclic Refiner: Object-Aware Temporal Representation Learning for Multi-View 3D Detection and Tracking
• Authors: Mingzhe Guo, Zhipeng Zhang, Liping Jing, Yuan He, Ke Wang, Heng Fan,
• Abstract summary: We propose a unified object-aware temporal learning framework for multi-view 3D detection and tracking tasks. The proposed model achieves consistent performance gains over baselines of different designs.
• Score: 37.186306646752975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a unified object-aware temporal learning framework for multi-view 3D detection and tracking tasks. Having observed that the efficacy of the temporal fusion strategy in recent multi-view perception methods may be weakened by distractors and background clutters in historical frames, we propose a cyclic learning mechanism to improve the robustness of multi-view representation learning. The essence is constructing a backward bridge to propagate information from model predictions (e.g., object locations and sizes) to image and BEV features, which forms a circle with regular inference. After backward refinement, the responses of target-irrelevant regions in historical frames would be suppressed, decreasing the risk of polluting future frames and improving the object awareness ability of temporal fusion. We further tailor an object-aware association strategy for tracking based on the cyclic learning model. The cyclic learning model not only provides refined features, but also delivers finer clues (e.g., scale level) for tracklet association. The proposed cycle learning method and association module together contribute a novel and unified multi-task framework. Experiments on nuScenes show that the proposed model achieves consistent performance gains over baselines of different designs (i.e., dense query-based BEVFormer, sparse query-based SparseBEV and LSS-based BEVDet4D) on both detection and tracking evaluation.

Related papers

• LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry [52.131996528655094]
  We present the Long-term Effective Any Point Tracking (LEAP) module. LEAP innovatively combines visual, inter-track, and temporal cues with mindfully selected anchors for dynamic track estimation. Based on these traits, we develop LEAP-VO, a robust visual odometry system adept at handling occlusions and dynamic scenes.
  arXiv  Detail & Related papers  (2024-01-03T18:57:27Z)
• LEF: Late-to-Early Temporal Fusion for LiDAR 3D Object Detection [40.267769862404684]
  We propose a late-to-early recurrent feature fusion scheme for 3D object detection using temporal LiDAR point clouds. Our main motivation is fusing object-aware latent embeddings into the early stages of a 3D object detector.
  arXiv  Detail & Related papers  (2023-09-28T21:58:25Z)
• Shape Anchor Guided Holistic Indoor Scene Understanding [9.463220988312218]
  We propose a shape anchor guided learning strategy (AncLearn) for robust holistic indoor scene understanding. AncLearn generates anchors that dynamically fit instance surfaces to (i) unmix noise and target-related features for offering reliable proposals at the detection stage. We embed AncLearn into a reconstruction-from-detection learning system (AncRec) to generate high-quality semantic scene models.
  arXiv  Detail & Related papers  (2023-09-20T08:30:20Z)
• Learning Monocular Depth in Dynamic Environment via Context-aware Temporal Attention [9.837958401514141]
  We present CTA-Depth, a Context-aware Temporal Attention guided network for multi-frame monocular Depth estimation. Our approach achieves significant improvements over state-of-the-art approaches on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-12T11:48:32Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention [50.11672196146829]
  3D object detection with surround-view images is an essential task for autonomous driving. We propose DETR4D, a Transformer-based framework that explores sparse attention and direct feature query for 3D object detection in multi-view images.
  arXiv  Detail & Related papers  (2022-12-15T14:18:47Z)
• A Dual-Cycled Cross-View Transformer Network for Unified Road Layout Estimation and 3D Object Detection in the Bird's-Eye-View [4.251500966181852]
  We propose a unified model for road layout estimation and 3D object detection inspired by the transformer architecture and the CycleGAN learning framework. We set up extensive learning scenarios to study the effect of multi-class learning for road layout estimation in various situations. Experiment results attest the effectiveness of our model; we achieve state-of-the-art performance in both the road layout estimation and 3D object detection tasks.
  arXiv  Detail & Related papers  (2022-09-19T08:43:38Z)
• BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving [92.05963633802979]
  We present BEVerse, a unified framework for 3D perception and prediction based on multi-camera systems. We show that the multi-task BEVerse outperforms single-task methods on 3D object detection, semantic map construction, and motion prediction.
  arXiv  Detail & Related papers  (2022-05-19T17:55:35Z)
• Fusing Local Similarities for Retrieval-based 3D Orientation Estimation of Unseen Objects [70.49392581592089]
  We tackle the task of estimating the 3D orientation of previously-unseen objects from monocular images. We follow a retrieval-based strategy and prevent the network from learning object-specific features. Our experiments on the LineMOD, LineMOD-Occluded, and T-LESS datasets show that our method yields a significantly better generalization to unseen objects than previous works.
  arXiv  Detail & Related papers  (2022-03-16T08:53:00Z)
• Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection [102.62963605429508]
  Point cloud semantic segmentation plays an essential role in autonomous driving. Current 3D semantic segmentation networks focus on convolutional architectures that perform great for well represented classes. We propose a novel Aware 3D Semantic Detection (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task.
  arXiv  Detail & Related papers  (2020-09-22T14:17:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.