DetAny4D: Detect Anything 4D Temporally in a Streaming RGB Video

• URL: http://arxiv.org/abs/2511.18814v1
• Date: Mon, 24 Nov 2025 06:42:17 GMT
• Title: DetAny4D: Detect Anything 4D Temporally in a Streaming RGB Video
• Authors: Jiawei Hou, Shenghao Zhang, Can Wang, Zheng Gu, Yonggen Ling, Taiping Zeng, Xiangyang Xue, Jingbo Zhang,
• Abstract summary: Existing open-set 4D object detection methods make predictions on a frame-by-frame basis without modeling temporal consistency.<n>DetAny4D is an open-set end-to-end framework that predicts 3D b-boxes directly from sequential inputs.<n>Extensive experiments show that DetAny4D achieves competitive detection accuracy and significantly improves temporal stability.
• Score: 29.912863749642156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable 4D object detection, which refers to 3D object detection in streaming video, is crucial for perceiving and understanding the real world. Existing open-set 4D object detection methods typically make predictions on a frame-by-frame basis without modeling temporal consistency, or rely on complex multi-stage pipelines that are prone to error propagation across cascaded stages. Progress in this area has been hindered by the lack of large-scale datasets that capture continuous reliable 3D bounding box (b-box) annotations. To overcome these challenges, we first introduce DA4D, a large-scale 4D detection dataset containing over 280k sequences with high-quality b-box annotations collected under diverse conditions. Building on DA4D, we propose DetAny4D, an open-set end-to-end framework that predicts 3D b-boxes directly from sequential inputs. DetAny4D fuses multi-modal features from pre-trained foundational models and designs a geometry-aware spatiotemporal decoder to effectively capture both spatial and temporal dynamics. Furthermore, it adopts a multi-task learning architecture coupled with a dedicated training strategy to maintain global consistency across sequences of varying lengths. Extensive experiments show that DetAny4D achieves competitive detection accuracy and significantly improves temporal stability, effectively addressing long-standing issues of jitter and inconsistency in 4D object detection. Data and code will be released upon acceptance.

Related papers

• MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence [50.11889361459544]
  Humans are born with vision-based 4D spatial-temporal intelligence.<n>Despite its importance, this capability remains a significant bottleneck for current large language models (MLLMs)
  arXiv  Detail & Related papers  (2026-02-28T07:23:36Z)
• Tracking-Guided 4D Generation: Foundation-Tracker Motion Priors for 3D Model Animation [21.075786141331974]
  We present emphTrack4DGen, a framework for generating dynamic 4D objects from sparse inputs.<n>In Stage One, we enforce dense, feature-level point correspondences inside the diffusion generator.<n>In Stage Two, we reconstruct a dynamic 4D-GS using a hybrid motion encoding.
  arXiv  Detail & Related papers  (2025-12-05T21:13:04Z)
• M^3Detection: Multi-Frame Multi-Level Feature Fusion for Multi-Modal 3D Object Detection with Camera and 4D Imaging Radar [12.877894178462297]
  M3Detection is a unified multi-frame 3D object detection framework that performs multi-level feature fusion on multi-modal data from camera and 4D radar.<n>We show that M3Detection achieves state-of-the-art 3D detection performance, its effectiveness in multi-frame detection with camera-4D imaging radar fusion.
  arXiv  Detail & Related papers  (2025-10-31T04:34:15Z)
• Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models [116.31344506738816]
  We present a novel framework, textbfDiffusion4D, for efficient and scalable 4D content generation. We develop a 4D-aware video diffusion model capable of synthesizing orbital views of dynamic 3D assets. Our method surpasses prior state-of-the-art techniques in terms of generation efficiency and 4D geometry consistency.
  arXiv  Detail & Related papers  (2024-05-26T17:47:34Z)
• 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency [118.15258850780417]
  We present textbf4DGen, a novel framework for grounded 4D content creation.<n>Our pipeline facilitates controllable 4D generation, enabling users to specify the motion via monocular video or adopt image-to-video generations.<n>Compared to existing video-to-4D baselines, our approach yields superior results in faithfully reconstructing input signals.
  arXiv  Detail & Related papers  (2023-12-28T18:53:39Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• 4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion [2.911052912709637]
  Four-dimensional (4D) radar--visual odometry (4DRVO) integrates complementary information from 4D radar and cameras. 4DRVO may exhibit significant tracking errors owing to sparsity of 4D radar point clouds. We present 4DRVO-Net, which is a method for 4D radar--visual odometry.
  arXiv  Detail & Related papers  (2023-08-12T14:00:09Z)
• Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion [14.15155927539293]
  We introduce Sparse4D, which does the iterative refinement of anchor boxes via sparsely sampling and fusing spatial-temporal features. In experiment, our method outperforms all sparse based methods and most BEV based methods on detection task in the nuScenes dataset.
  arXiv  Detail & Related papers  (2022-11-19T04:20:57Z)
• 4D Unsupervised Object Discovery [53.561750858325915]
  We propose 4D unsupervised object discovery, jointly discovering objects from 4D data -- 3D point clouds and 2D RGB images with temporal information. We present the first practical approach for this task by proposing a ClusterNet on 3D point clouds, which is jointly optimized with a 2D localization network.
  arXiv  Detail & Related papers  (2022-10-10T16:05:53Z)
• 4D-Net for Learned Multi-Modal Alignment [87.58354992455891]
  We present 4D-Net, a 3D object detection approach, which utilizes 3D Point Cloud and RGB sensing information, both in time. We are able to incorporate the 4D information by performing a novel connection learning across various feature representations and levels of abstraction, as well as by observing geometric constraints.
  arXiv  Detail & Related papers  (2021-09-02T16:35:00Z)
• DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes [54.239416488865565]
  We propose a fast single-stage 3D object detection method for LIDAR data. The core novelty of our method is a fast, single-pass architecture that both detects objects in 3D and estimates their shapes. We find that our proposed method achieves state-of-the-art results by 5% on object detection in ScanNet scenes, and it gets top results by 3.4% in the Open dataset.
  arXiv  Detail & Related papers  (2020-04-02T17:48:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.