Towards an End-to-End System for 3D Tracking of Physical Objects in Virtual Immersive Environments

• URL: http://arxiv.org/abs/2605.02901v1
• Date: Mon, 30 Mar 2026 12:00:13 GMT
• Title: Towards an End-to-End System for 3D Tracking of Physical Objects in Virtual Immersive Environments
• Abstract summary: This work aims to establish an end-to-end system for tracking of physical 3D objects for virtual reality (VR) applications.<n>We focus on training applications requiring real-time tracking of the position of small physical objects and their reflection in VR space.<n>We propose a system for object tracking via fiducial markers alongside a software harness.
• Score: 1.4897517581261097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work aims to establish an end-to-end system for tracking of physical 3D objects for virtual reality (VR) applications. We focus on training applications requiring real-time tracking of the position of small physical objects and their reflection in VR space. Out goal is to perform object tracking in a "plug and play" manner, without using complex systems with quite large tracking devices or manually implementing object tracking. We therefore propose a system for object tracking via fiducial markers alongside a software harness, to enable fast and efficient designation of objects to be tracked and data streaming solution for end-use applications. The system utilizes AruCo, AprilTag and an original Colored Control Points based fiducial system. It allows for easy tag detection and use of object position data, which are crucial for immersive training environments based on VR and eXtended Reality (XR). We evaluate various tag sizes, detection distances, and different camera devices against the theoretical limits. In effect, we create a complete solution for implementing marker-based, real-to-virtual object position mapping for various applications.

Related papers

• Dynamic Object Detection and Tracking in Construction: A Fisheye Camera and LiDAR Sensor Fusion Model [1.9875456417756243]
  We present a novel framework that enhances a quadruped robot equipped with a LiDAR sensor and an upward-facing fisheye camera for real-time dynamic object detection and tracking.<n>The proposed system demonstrates high precision, simplicity, and robustness, particularly in handling objects transitioning between dynamic and static states.
  arXiv  Detail & Related papers  (2026-07-08T01:42:14Z)
• Physics-Guided Fusion for Robust 3D Tracking of Fast Moving Small Objects [0.7285647284266376]
  This paper addresses the challenge of detecting and tracking rapidly moving small objects using an RGB-D camera.<n>Our system combines deep learning-based detection with physics-based tracking to overcome the limitations of existing approaches.<n>Our system has significant applications for improving robot perception on autonomous platforms.
  arXiv  Detail & Related papers  (2025-10-23T02:00:58Z)
• Open3DTrack: Towards Open-Vocabulary 3D Multi-Object Tracking [73.05477052645885]
  We introduce open-vocabulary 3D tracking, which extends the scope of 3D tracking to include objects beyond predefined categories.<n>We propose a novel approach that integrates open-vocabulary capabilities into a 3D tracking framework, allowing for generalization to unseen object classes.
  arXiv  Detail & Related papers  (2024-10-02T15:48:42Z)
• Once Detected, Never Lost: Surpassing Human Performance in Offline LiDAR based 3D Object Detection [50.959453059206446]
  This paper aims for high-performance offline LiDAR-based 3D object detection. We first observe that experienced human annotators annotate objects from a track-centric perspective. We propose a high-performance offline detector in a track-centric perspective instead of the conventional object-centric perspective.
  arXiv  Detail & Related papers  (2023-04-24T17:59:05Z)
• CXTrack: Improving 3D Point Cloud Tracking with Contextual Information [59.55870742072618]
  3D single object tracking plays an essential role in many applications, such as autonomous driving. We propose CXTrack, a novel transformer-based network for 3D object tracking. We show that CXTrack achieves state-of-the-art tracking performance while running at 29 FPS.
  arXiv  Detail & Related papers  (2022-11-12T11:29:01Z)
• TripletTrack: 3D Object Tracking using Triplet Embeddings and LSTM [0.0]
  3D object tracking is a critical task in autonomous driving systems. In this paper we investigate the use of triplet embeddings in combination with motion representations for 3D object tracking.
  arXiv  Detail & Related papers  (2022-10-28T15:23:50Z)
• CFTrack: Center-based Radar and Camera Fusion for 3D Multi-Object Tracking [9.62721286522053]
  We propose an end-to-end network for joint object detection and tracking based on radar and camera sensor fusion. Our proposed method uses a center-based radar-camera fusion algorithm for object detection and utilizes a greedy algorithm for object association. We evaluate our method on the challenging nuScenes dataset, where it achieves 20.0 AMOTA and outperforms all vision-based 3D tracking methods in the benchmark.
  arXiv  Detail & Related papers  (2021-07-11T23:56:53Z)
• EagerMOT: 3D Multi-Object Tracking via Sensor Fusion [68.8204255655161]
  Multi-object tracking (MOT) enables mobile robots to perform well-informed motion planning and navigation by localizing surrounding objects in 3D space and time. Existing methods rely on depth sensors (e.g., LiDAR) to detect and track targets in 3D space, but only up to a limited sensing range due to the sparsity of the signal. We propose EagerMOT, a simple tracking formulation that integrates all available object observations from both sensor modalities to obtain a well-informed interpretation of the scene dynamics.
  arXiv  Detail & Related papers  (2021-04-29T22:30:29Z)
• Learnable Online Graph Representations for 3D Multi-Object Tracking [156.58876381318402]
  We propose a unified and learning based approach to the 3D MOT problem. We employ a Neural Message Passing network for data association that is fully trainable. We show the merit of the proposed approach on the publicly available nuScenes dataset by achieving state-of-the-art performance of 65.6% AMOTA and 58% fewer ID-switches.
  arXiv  Detail & Related papers  (2021-04-23T17:59:28Z)
• Monocular Quasi-Dense 3D Object Tracking [99.51683944057191]
  A reliable and accurate 3D tracking framework is essential for predicting future locations of surrounding objects and planning the observer's actions in numerous applications such as autonomous driving. We propose a framework that can effectively associate moving objects over time and estimate their full 3D bounding box information from a sequence of 2D images captured on a moving platform.
  arXiv  Detail & Related papers  (2021-03-12T15:30:02Z)
• e-TLD: Event-based Framework for Dynamic Object Tracking [23.026432675020683]
  This paper presents a long-term object tracking framework with a moving event camera under general tracking conditions. The framework uses a discriminative representation for the object with online learning, and detects and re-tracks the object when it comes back into the field-of-view.
  arXiv  Detail & Related papers  (2020-09-02T07:08:56Z)
• Instant 3D Object Tracking with Applications in Augmented Reality [4.893345190925178]
  Tracking object poses in 3D is a crucial building block for Augmented Reality applications. We propose an instant motion tracking system that tracks an object's pose in space in real-time on mobile devices.
  arXiv  Detail & Related papers  (2020-06-23T17:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.