4D-based Robot Navigation Using Relativistic Image Processing

• URL: http://arxiv.org/abs/2410.22087v1
• Date: Tue, 29 Oct 2024 14:42:19 GMT
• Title: 4D-based Robot Navigation Using Relativistic Image Processing
• Abstract summary: 4D perception offers the possibility of predicting one's own position and changes in the environment over time. We present a 4D-based approach to robot navigation using relativistic image processing.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine perception is an important prerequisite for safe interaction and locomotion in dynamic environments. This requires not only the timely perception of surrounding geometries and distances but also the ability to react to changing situations through predefined, learned but also reusable skill endings of a robot so that physical damage or bodily harm can be avoided. In this context, 4D perception offers the possibility of predicting one's own position and changes in the environment over time. In this paper, we present a 4D-based approach to robot navigation using relativistic image processing. Relativistic image processing handles the temporal-related sensor information in a tensor model within a constructive 4D space. 4D-based navigation expands the causal understanding and the resulting interaction radius of a robot through the use of visual and sensory 4D information.

Related papers

• RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation [43.55126782618712]
  We introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image.<n>Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world manipulation tasks.
  arXiv  Detail & Related papers  (2026-07-07T17:58:15Z)
• Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs [85.01118931556098]
  We introduce IMU-to--4D, a framework that reconstructs human motion and 3D layouts purely from wearable sensors.<n>IMU-to--4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with coarse scene structure.<n>Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than SoTAd pipelines.
  arXiv  Detail & Related papers  (2026-04-23T17:59:16Z)
• Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation [46.24300848525144]
  Kinema4D is a new action-conditioned 4D generative robotic simulator that disentangles the robot-world interaction into: i.e. precise 4D representation of robot controls, ii. Generative 4D modeling of environmental reactions.<n>For the first time, it shows potential zero-shot transfer capability, providing a high-fidelity foundation for advancing next-generation embodied simulation.
  arXiv  Detail & Related papers  (2026-03-17T15:36:38Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction [70.06600045165905]
  3D spatial perception is fundamental to general robotic manipulation, yet obtaining reliable, high-quality 3D geometry remains challenging.<n>We introduce Robo3R, a manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time.<n>We observe consistent gains in performance, suggesting the promise of this alternative 3D sensing module for robotic manipulation.
  arXiv  Detail & Related papers  (2026-02-10T18:58:15Z)
• CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction [40.557276644446475]
  We present CARI4D, the first category-agnostic method that reconstructs spatially and temporarily consistent 4D human-object interaction at metric scale from monocular RGB videos.<n>Our model generalizes beyond the training categories and thus can be applied zero-shot to in-the-wild internet videos.
  arXiv  Detail & Related papers  (2025-12-12T19:11:11Z)
• VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation [54.81449795163812]
  We develop a general VLA model with 4D awareness for temporally coherent robotic manipulation.<n>We extract visual features, embed 1D time into 3D positions for 4D embeddings, and fuse them into a unified visual representation via a cross-attention mechanism.<n>Within this framework, the designed visual action jointly make robotic manipulation spatially-smooth and temporally temporally coherent.
  arXiv  Detail & Related papers  (2025-11-21T12:26:30Z)
• Streaming 4D Visual Geometry Transformer [63.99937807085461]
  We propose a streaming 4D visual geometry transformer to process the input sequence in an online manner.<n>We use temporal causal attention and cache the historical keys and values as implicit memory to enable efficient streaming long-term 4D reconstruction.<n>Experiments on various 4D geometry perception benchmarks demonstrate that our model increases the inference speed in online scenarios.
  arXiv  Detail & Related papers  (2025-07-15T17:59:57Z)
• Geometry-aware 4D Video Generation for Robot Manipulation [28.709339959536106]
  We propose a 4D video generation model that enforces multi-view 3D consistency of videos by supervising the model with cross-view pointmap alignment during training.<n>This geometric supervision enables the model to learn a shared 3D representation of the scene, allowing it to predict future video sequences from novel viewpoints.<n>Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets.
  arXiv  Detail & Related papers  (2025-07-01T18:01:41Z)
• Pre-training Auto-regressive Robotic Models with 4D Representations [43.80798244473759]
  ARM4R is an Auto-regressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better pre-trained robotic model. Our experiments show that ARM4R can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations.
  arXiv  Detail & Related papers  (2025-02-18T18:59:01Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [116.2042238179433]
  In this paper, we frame dynamic scenes as unconstrained 4D volume learning problems. We represent a target dynamic scene using a collection of 4D Gaussian primitives with explicit geometry and appearance features. This approach can capture relevant information in space and time by fitting the underlying photorealistic-temporal volume. Notably, our 4DGS model is the first solution that supports real-time rendering of high-resolution, novel views for complex dynamic scenes.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)
• RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics [26.42651735582044]
  We introduce RoboSpatial, a large-scale spatial understanding dataset consisting of real indoor and tabletop scenes captured as 3D scans and egocentric images annotated with rich spatial information relevant to robotics. Our experiments show that models trained with RoboSpatial outperform baselines on downstream tasks such as spatial affordance prediction, spatial relationship prediction, and robotics manipulation.
  arXiv  Detail & Related papers  (2024-11-25T16:21:34Z)
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [51.49400490437258]
  This work develops a method for imitating articulated object manipulation from a single monocular RGB human demonstration. We first propose 4D Differentiable Part Models (4D-DPM), a method for recovering 3D part motion from a monocular video. Given this 4D reconstruction, the robot replicates object trajectories by planning bimanual arm motions that induce the demonstrated object part motion. We evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part trajectories and RSRD's physical execution performance on 9 objects across 10 trials each on a bimanual YuMi robot.
  arXiv  Detail & Related papers  (2024-09-26T17:57:16Z)
• OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving [62.54220021308464]
  We propose a diffusion-based 4D occupancy generation model, OccSora, to simulate the development of the 3D world for autonomous driving. OccSora can generate 16s-videos with authentic 3D layout and temporal consistency, demonstrating its ability to understand the spatial and temporal distributions of driving scenes.
  arXiv  Detail & Related papers  (2024-05-30T17:59:42Z)
• TC4D: Trajectory-Conditioned Text-to-4D Generation [94.90700997568158]
  We propose TC4D: trajectory-conditioned text-to-4D generation, which factors motion into global and local components. We learn local deformations that conform to the global trajectory using supervision from a text-to-video model. Our approach enables the synthesis of scenes animated along arbitrary trajectories, compositional scene generation, and significant improvements to the realism and amount of generated motion.
  arXiv  Detail & Related papers  (2024-03-26T17:55:11Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)
• Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering [20.62295718847247]
  State estimation from measured data is crucial for robotic applications as autonomous systems rely on sensors to capture the motion and localize in the 3D world. In this work, we achieve image-based robot pose estimation and shape reconstruction from camera images. We demonstrate that our method of using geometrical shape primitives can achieve high accuracy in shape reconstruction for a soft continuum robot and pose estimation for a robot manipulator.
  arXiv  Detail & Related papers  (2023-02-27T18:51:29Z)
• Neural Scene Representation for Locomotion on Structured Terrain [56.48607865960868]
  We propose a learning-based method to reconstruct the local terrain for a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the estimates the topography in the robot's vicinity. We propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement.
  arXiv  Detail & Related papers  (2022-06-16T10:45:17Z)
• HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR [51.9200422793806]
  Using only body-mounted IMUs and LiDAR, HSC4D is space-free without any external devices' constraints and map-free without pre-built maps. Relationships between humans and environments are also explored to make their interaction more realistic.
  arXiv  Detail & Related papers  (2022-03-17T10:05:55Z)
• 4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface [7.637832293935966]
  We introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D.
  arXiv  Detail & Related papers  (2021-05-05T07:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.