mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction

• URL: http://arxiv.org/abs/2607.22973v2
• Date: Sat, 01 Aug 2026 17:01:11 GMT
• Title: mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction
• Abstract summary: We present mmSimPrior, a simulation-pretrained framework that factorizes transferable knowledge into signal, motion, and radar-to-motion mapping priors.<n>Experiments on mmSimPrior-Real and RT-Pose demonstrate consistent gains.
• Score: 12.731621912265986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Millimeter-wave (mmWave) radar enables privacy-preserving and illumination-robust human motion reconstruction, but training generalizable models typically requires costly paired radar-motion recordings. Simulation can scale such supervision, yet even physics-based simulators cannot fully reproduce real-world multipath, clutter, hardware-specific response statistics, or distance-dependent resolution degradation, leaving a sim-to-real gap. We present mmSimPrior, a simulation-pretrained framework that factorizes transferable knowledge into signal, motion, and radar-to-motion mapping priors. To learn transferable signal and motion priors, we pretrain a multimodal radar encoder with a physics-informed domain-randomization curriculum designed to mitigate the sim-to-real gap by approximating real-world propagation- and acquisition-level variations, while a joint-temporal tokenizer learns a discrete prior over plausible human motion. A dual-mode mapping module predicts either motion-code distributions for structurally constrained zero-shot reconstruction or continuous motion parameters for flexible adaptation from limited real data. We further construct a 4.2M-frame, 31K-sequence dataset suite and introduce a No-Overlap Setting that prevents any exact subject-environment-location-motion tuple from appearing in both the adaptation and test sets. Experiments on mmSimPrior-Real and RT-Pose demonstrate consistent gains: with only 24 paired real sequences, mmSimPrior-Reg reduces MPJPE by 24.7-39.0% over the strongest baseline across the three environments, while mmSimPrior-Cls reduces zero-shot MPJPE by 8.5% without fine-tuning.

Related papers

• Task-Relevant Feature-Dynamics Fidelity Enables Zero-Shot Sim-to-Real Transfer for Robotic Ultrasound Scanning [58.25007580703646]
  We develop a TR-oriented ultrasound simulator that combines a shared structural intermediate domain, trajectory-level fixed noise, and few-step conditional flow generation.<n>In phantom experiments, a policy trained exclusively in simulation succeeded in 390 of 400 zero-shot deployments across four target planes.
  arXiv  Detail & Related papers  (2026-08-30T02:37:00Z)
• Cross-simulator transfer with foundation model summaries: Towards robust SKA-era reionization inference [0.0]
  We show that a self-supervised Vision Transformer (ViT), pretrained label-free on a fast approximate simulator, produces transferable data summaries that generalize across simulators.<n>Without retraining, it can be reused as a frozen encoder to infer astrophysical parameters from a completely different simulator.
  arXiv  Detail & Related papers  (2026-08-26T19:37:18Z)
• HybridSim: A Physics-Learning Hybrid Digital Twin for mmWave Human Sensing [16.25180983132509]
  HybridSim is a physics-learning hybrid simulator that synthesizes mmWave radar signals from dynamic human meshes under a fixed indoor room configuration.<n>We show improved agreement with a physically based reference and consistent gains on downstream radar-based human sensing tasks when using HybridSim for site-specific data augmentation.
  arXiv  Detail & Related papers  (2026-07-17T10:20:31Z)
• Machine Learning-Based Reconstruction for Resistive Silicon Sensors [53.365453214937496]
  We study machine-learning-based reconstruction and compression for resistive silicon sensors.<n>We use full-waveform information from correlated pads to regularise the reconstruction and extract spatial information.<n>We first introduce recurrent neural network models based on LSTM layers, which provide a proof-of-concept implementation for full-waveform reconstruction.
  arXiv  Detail & Related papers  (2026-07-13T14:03:26Z)
• MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations [51.837372131319285]
  MotionMAR is a framework for motion reconstruction from sparse observations.<n>It first estimates the global trajectory of human motion and then gradually refines the temporal details.<n>MotionMAR achieves state-of-the-art accuracy on the AMASS dataset.
  arXiv  Detail & Related papers  (2026-06-22T08:15:56Z)
• TactSpace: Learning a Physics-enriched Shared Latent Space for Tactile Sim-to-Real Transfer [8.357674069719524]
  We propose a multi-modal representation learning framework that aligns heterogeneous tactile modalities within a shared latent space.<n>Our approach employs modality-specific encoders to project diverse tactile observations into a common embedding space.<n>Our results demonstrate zero-shot sim-to-real transfer across physically dissimilar representations.
  arXiv  Detail & Related papers  (2026-06-17T11:41:27Z)
• Physics-Informed Reinforcement Learning of Spatial Density Velocity Potentials for Map-Free Racing [73.88859384645264]
  Out-Of-Distribution (OOD) generalization to various racetracks utilizes Machine Learning (ML) to encode the mathematical relation between sensor data and vehicle actuation for end-to-end control.<n>This paper presents a DRL method that parameterizes nonlinear vehicle dynamics from the spectral distribution of depth measurements with a non-geometric, physics-informed reward, to infer vehicle time-optimal and overtaking racing controls with an Artificial Neural Network (ANN)<n>The policy outperforms human demonstrations by 12% in OOD tracks on proportionally scaled hardware, by maximizing the friction circle with tire dynamics that resemble an empirical Pacejka tire model.
  arXiv  Detail & Related papers  (2026-04-10T17:12:07Z)
• ReconPhys: Reconstruct Appearance and Physical Attributes from Single Video [51.2815592637053]
  ReconPhys is a feedforward framework that jointly learns physical attribute estimation and 3D Gaussian Splatting reconstruction from a single monocular video.<n>Our method employs a dual-branch architecture trained via a self-supervised strategy, eliminating the need for ground-truth physics labels.<n> Experiments on a large-scale synthetic dataset demonstrate superior performance.
  arXiv  Detail & Related papers  (2026-04-09T06:51:14Z)
• Graph Neural ODE Digital Twins for Control-Oriented Reactor Thermal-Hydraulic Forecasting Under Partial Observability [6.370970195988317]
  Real-time supervisory control of advanced reactors requires accurate forecasting of plant-wide thermalhydraulic states.<n>We present a physics-informed message-passing Differential Neural Network and a Neural Ordinary Equation (GNN-ODE)<n>The GNN-ODE surrogate achieves satisfactory results for the system dynamics prediction.
  arXiv  Detail & Related papers  (2026-04-08T16:58:14Z)
• Resonance4D: Frequency-Domain Motion Supervision for Preset-Free Physical Parameter Learning in 4D Dynamic Physical Scene Simulation [53.141181758015186]
  We present Resonance4D, a physics-driven 4D dynamic simulation framework that couples 3D Gaussian Splatting with the Material Point Method.<n>Our key insight is that dynamic consistency can be enforced without dense temporal generation by jointly constraining motion in complementary domains.<n>We show that Resonance4D achieves strong physical fidelity and motion consistency while reducing peak GPU memory from over 35,GB to around 20,GB.
  arXiv  Detail & Related papers  (2026-04-02T13:00:22Z)
• HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions [69.43316445143408]
  We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions.<n>Existing methods suffer from a perception-simulation gap, leading to instability in physics engines and failure in embodied AI applications.
  arXiv  Detail & Related papers  (2026-03-16T17:58:33Z)
• PRISM: Performer RS-IMLE for Single-pass Multisensory Imitation Learning [51.24484551729328]
  We introduce PRISM, a single-pass policy based on a batch-global rejection-sampling variant of IMLE.<n> PRISM couples a temporal multisensory encoder with a linear-attention generator using a Performer architecture.<n>We demonstrate the efficacy of PRISM on a diverse real-world hardware suite, including loco-manipulation using a Unitree Go2 with a 7-DoF arm D1 and tabletop manipulation with a UR5 manipulator.
  arXiv  Detail & Related papers  (2026-02-02T17:57:37Z)
• mmWEAVER: Environment-Specific mmWave Signal Synthesis from a Photo and Activity Description [2.1178416840822027]
  mmWeaver is a novel framework that synthesizes realistic, environment-specific complex mmWave signals.<n>By conditioning on semantic and geometric priors, mmWeaver generates diverse I/Q signals at multiple resolutions.<n>Experiments show that mmWeaver achieves a complex SSIM of 0.88 and a PSNR of 35 dB, outperforming existing methods in signal realism.
  arXiv  Detail & Related papers  (2025-12-10T03:32:48Z)
• Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework [57.994965436344195]
  Beamforming is a key technology in millimeter-wave (mmWave) communications that improves signal transmission by optimizing directionality and intensity.<n> multimodal sensing-aided beam prediction has gained significant attention, using various sensing data to predict user locations or network conditions.<n>Despite its promising potential, the adoption of multimodal sensing-aided beam prediction is hindered by high computational complexity, high costs, and limited datasets.
  arXiv  Detail & Related papers  (2025-04-07T15:38:25Z)
• A Lightweight Human Pose Estimation Approach for Edge Computing-Enabled Metaverse with Compressive Sensing [23.884862152830184]
  The ability to estimate 3D movements of users over edge computing-enabled networks, such as 5G/6G networks, is a key enabler for the new era of extended reality (XR) and Metaverse applications. Recent advancements in deep learning have shown advantages over optimization techniques for estimating 3D human poses given spare measurements from sensor signals. We propose a novel approach for redundancy removal and lightweight transmission of IMU signals over noisy wireless environments.
  arXiv  Detail & Related papers  (2024-08-26T02:57:23Z)
• PCGen: Point Cloud Generator for LiDAR Simulation [10.692184635629792]
  Existing methods generate data which are more noisy and complete than the real point clouds. We propose FPA raycasting and surrogate model raydrop. With minimal training data, the surrogate model can generalize to different geographies and scenes. Results show that object detection models trained by simulation data can achieve similar result as the real data trained model.
  arXiv  Detail & Related papers  (2022-10-17T04:13:21Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.