Taming Vision Priors for Data Efficient mmWave Channel Modeling

• URL: http://arxiv.org/abs/2603.13383v1
• Date: Wed, 11 Mar 2026 04:35:51 GMT
• Title: Taming Vision Priors for Data Efficient mmWave Channel Modeling
• Abstract summary: Differentiable ray tracing offers a physics-grounded solution for modeling millimeter-wave (mmWave) propagation.<n>Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements.<n>We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing.
• Score: 17.020987773588907
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Accurately modeling millimeter-wave (mmWave) propagation is essential for real-time AR and autonomous systems. Differentiable ray tracing offers a physics-grounded solution but still facing deployment challenges due to its over-reliance on exhaustive channel measurements or brittle, hand-tuned scene models for material properties. We present VisRFTwin, a scalable and data-efficient digital-twin framework that integrates vision-derived material priors with differentiable ray tracing. Multi-view images from commodity cameras are processed by a frozen Vision-Language Model to extract dense semantic embeddings, which are translated into initial estimates of permittivity and conductivity for scene surfaces. These priors initialize a Sionna-based differentiable ray tracer, which rapidly calibrates material parameters via gradient descent with only a few dozen sparse channel soundings. Once calibrated, the association between vision features and material parameters is retained, enabling fast transfer to new scenarios without repeated calibration. Evaluations across three real-world scenarios, including office interiors, urban canyons, and dynamic public spaces show that VisRFTwin reduces channel measurement needs by up to 10$\times$ while achieving a 59% lower median delay spread error than pure data-driven deep learning methods.

Related papers

• Low-Altitude Channel Multipath Prediction via Panoramic Perception and Vision-Language Model [33.916311823620596]
  Unmanned aerial vehicle (UAV) communication is expected to support a wide range of low-altitude applications in 6G mobile networks.<n>Existing multimodal channel prediction approaches mainly focus on large-scale metrics such as path loss.<n>This paper proposes PanoLAMP, a Panoramic perception and vision-language model-based Low-Altitude Multipath Prediction framework.
  arXiv  Detail & Related papers  (2026-07-24T03:59:32Z)
• SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents [55.26273954022457]
  Estimating the material property field of 3D assets is critical for physics-based simulation, robotics, and digital twin generation.<n>We present SLAT-Phys, an end-to-end method that predicts spatially varying material property fields of 3D assets directly from a single RGB image without explicit 3D reconstruction.
  arXiv  Detail & Related papers  (2026-03-25T06:14:03Z)
• Image-Conditioned Adaptive Parameter Tuning for Visual Odometry Frontends [21.343239924094764]
  Resource-constrained autonomous robots rely on sparse direct and semi-direct visual-(inertial)-odometry (VO) pipelines.<n>We propose the first image-conditioned reinforcement learning framework for online tuning of VO parameters.
  arXiv  Detail & Related papers  (2026-03-23T10:26:25Z)
• $D^3$-RSMDE: 40$\ imes$ Faster and High-Fidelity Remote Sensing Monocular Depth Estimation [72.9912717963138]
  Real-time, high-fidelity monocular depth estimation from remote sensing imagery is crucial for numerous applications.<n>Although using Vision Transformer (ViT) backbones for dense prediction is fast, they often exhibit poor perceptual quality.<n>We propose Depth Detail Diffusion for Remote Sensing Monocular Depth Estimation ($D3$-RSMDE) to overcome these limitations.<n>$D3$-RSMDE achieves a notable 11.85% reduction in the Learned Perceptual Image Patch Similarity (LPIPS) perceptual metric.
  arXiv  Detail & Related papers  (2026-03-17T10:50:36Z)
• Event-based Visual Deformation Measurement [76.25283405575108]
  Visual Deformation Measurement aims to recover dense deformation fields by tracking surface motion from camera observations.<n>Traditional image-based methods rely on minimal inter-frame motion to constrain the correspondence search space.<n>We propose an event-frame fusion framework that exploits events for temporally dense motion cues and frames for spatially dense precise estimation.
  arXiv  Detail & Related papers  (2026-02-16T01:04:48Z)
• Vision-Language-Model-Guided Differentiable Ray Tracing for Fast and Accurate Multi-Material RF Parameter Estimation [45.40179208702883]
  This paper proposes a vision-language-model (VLM) guided framework that accelerates and stabilizes multi-material parameter estimation.<n> Experiments in NVIDIA Sionna on indoor scenes show 2-4$times$ faster convergence and 10-100$times$ lower final parameter error.
  arXiv  Detail & Related papers  (2026-01-26T07:54:53Z)
• EvDiff: High Quality Video with an Event Camera [77.07279880903009]
  Reconstructing intensity images from events is a highly ill-posed task due to the inherent ambiguity of absolute brightness.<n>We propose EvDiff, an event-based diffusion model that follows a surrogate training framework to produce high-quality videos.
  arXiv  Detail & Related papers  (2025-11-21T18:49:18Z)
• Taming generative video models for zero-shot optical flow extraction [28.176290134216995]
  Self-supervised video models trained only for future frame prediction can be prompted, without fine-tuning, to output flow.<n>Inspired by the Counterfactual World Model (CWM) paradigm, we extend this idea to generative video models.<n> KL-tracing is a novel test-time procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and ungenerative predictive distributions.
  arXiv  Detail & Related papers  (2025-07-11T23:59:38Z)
• Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework [57.994965436344195]
  Beamforming is a key technology in millimeter-wave (mmWave) communications that improves signal transmission by optimizing directionality and intensity.<n> multimodal sensing-aided beam prediction has gained significant attention, using various sensing data to predict user locations or network conditions.<n>Despite its promising potential, the adoption of multimodal sensing-aided beam prediction is hindered by high computational complexity, high costs, and limited datasets.
  arXiv  Detail & Related papers  (2025-04-07T15:38:25Z)
• EvLight++: Low-Light Video Enhancement with an Event Camera: A Large-Scale Real-World Dataset, Novel Method, and More [7.974102031202597]
  EvLight++ is a novel event-guided low-light video enhancement approach designed for robust performance in real-world scenarios. EvLight++ significantly outperforms both single image- and video-based methods by 1.37 dB and 3.71 dB, respectively.
  arXiv  Detail & Related papers  (2024-08-29T04:30:31Z)
• Learning Radio Environments by Differentiable Ray Tracing [56.40113938833999]
  We introduce a novel gradient-based calibration method, complemented by differentiable parametrizations of material properties, scattering and antenna patterns. We have validated our method using both synthetic data and real-world indoor channel measurements, employing a distributed multiple-input multiple-output (MIMO) channel sounder.
  arXiv  Detail & Related papers  (2023-11-30T13:50:21Z)
• Differentiable Radio Frequency Ray Tracing for Millimeter-Wave Sensing [29.352303349003165]
  We propose DiffSBR, a differentiable framework for mmWave-based 3D reconstruction. DiffSBR incorporates a differentiable ray tracing engine to simulate radar point clouds from virtual 3D models. Experiments using various radar hardware validate DiffSBR's capability for fine-grained 3D reconstruction.
  arXiv  Detail & Related papers  (2023-11-22T06:13:39Z)
• A Generative Learning Approach for Spatio-temporal Modeling in Connected Vehicular Network [55.852401381113786]
  This paper proposes LaMI (Latency Model Inpainting), a novel framework to generate a comprehensive-temporal quality framework for wireless access latency of connected vehicles. LaMI adopts the idea from image inpainting and synthesizing and can reconstruct the missing latency samples by a two-step procedure. In particular, it first discovers the spatial correlation between samples collected in various regions using a patching-based approach and then feeds the original and highly correlated samples into a Varienational Autocoder (VAE)
  arXiv  Detail & Related papers  (2020-03-16T03:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.