FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System

• URL: http://arxiv.org/abs/2607.17803v1
• Date: Mon, 20 Jul 2026 10:42:16 GMT
• Title: FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System
• Abstract summary: Projector-camera (ProCams) systems achieve active scene perception and controllable appearance manipulation via structured illumination.<n>Existing inverse-rendering methods for ProCams deliver high-fidelity results but rely on time-consuming per-scene optimization.<n>We propose FF-ProCams, a Feed-Forward 3D Gaussian inverse-rendering framework for ProCams.
• Score: 17.828959053537798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Projector-camera (ProCams) systems achieve active scene perception and controllable appearance manipulation via structured illumination, serving as a core infrastructure for spatial augmented reality, projection mapping, and surface reflectance acquisition. Existing inverse-rendering methods for ProCams deliver high-fidelity results but rely on time-consuming per-scene optimization, while mainstream feed-forward 3D reconstruction models produce baked appearance that cannot adapt to spatially varying projector illumination. To resolve this accuracy-efficiency trade-off, we propose FF-ProCams, a Feed-Forward 3D Gaussian inverse-rendering framework for ProCams. A hybrid Mamba2-Transformer encoder aggregates cross-view geometric and photometric cues from sparse multi-view observations, and lightweight heads predict a relightable Gaussian representation in a single forward pass. We further design a projector-aware differentiable renderer to synthesize camera observations under arbitrary active illumination and ProCams poses. To enable feed-forward training, we construct a large-scale synthetic ProCams dataset covering diverse object geometries and surface materials. Experiments show FF-ProCams achieves high-fidelity projector-aware rendering, generalizes to unseen patterns, and supports novel projector-camera poses. Using only 8 input views, it outperforms optimization-based baselines with 297 views while reducing test-time reconstruction to 0.13 seconds (a three-to-five-order-of-magnitude speedup). The code and data are available at https://github.com/CPREgroup/FF-ProCams/.

Related papers

• Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras [43.56045236607455]
  We present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization.<n>Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance.<n>Experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU.
  arXiv  Detail & Related papers  (2026-09-14T04:50:52Z)
• ViewMorpher3D: A 3D-aware Diffusion Framework for Multi-Camera Novel View Synthesis in Autonomous Driving [20.935790354765604]
  We introduce ViewMorpher3D, a multi-view image enhancement framework based on image diffusion models.<n>Unlike single-view approaches, ViewMorpher3D jointly processes a set of rendered views conditioned on camera poses, 3D geometric priors, and temporally adjacent or spatially overlapping reference views.<n>Our framework accommodates variable numbers of cameras and flexible reference/target view configurations, making it adaptable to diverse sensor setups.
  arXiv  Detail & Related papers  (2026-01-12T13:44:14Z)
• 3DPR: Single Image 3D Portrait Relight using Generative Priors [101.74130664920868]
  3DPR is an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images.<n>We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets.<n>Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model.
  arXiv  Detail & Related papers  (2025-10-17T17:37:42Z)
• SViM3D: Stable Video Material Diffusion for Single Image 3D Generation [48.986972061812004]
  Video diffusion models have been successfully used to reconstruct 3D objects from a single image efficiently.<n>We extend a latent video diffusion model to output spatially varying PBR parameters and surface normals jointly with each generated view based on explicit camera control.<n>This unique setup allows for relighting and generating a 3D asset using our model as neural prior.
  arXiv  Detail & Related papers  (2025-10-09T14:29:47Z)
• DPCS: Path Tracing-Based Differentiable Projector-Camera Systems [49.69815958689441]
  Projector-camera systems (ProCams) simulation aims to model the physical project-and-capture process and associated scene parameters of a ProCams.<n>Recent advances use an end-to-end neural network to learn the project-and-capture process.<n>We introduce a novel path tracing-based differentiable projector-camera systems (DPCS), offering a differentiable ProCams simulation method.
  arXiv  Detail & Related papers  (2025-03-15T15:31:18Z)
• GS-ProCams: Gaussian Splatting-based Projector-Camera Systems [43.14716504756479]
  We present GS-ProCams, the first Gaussian Splatting-based framework for projector-camera systems (ProCams)<n>GS-ProCams is view-agnostic but also significantly enhances the efficiency of projection mapping (PM)<n>It also uses only 1/10 of the GPU memory for training and is 900 times faster in inference speed.
  arXiv  Detail & Related papers  (2024-12-16T13:26:52Z)
• Pose-Free Generalizable Rendering Transformer [72.47072706742065]
  PF-GRT is a Pose-Free framework for Generalizable Rendering Transformer. PF-GRT is parameterized using a local relative coordinate system. Experiments with zero-shot rendering on datasets reveal that it produces superior quality in generating photo-realistic images.
  arXiv  Detail & Related papers  (2023-10-05T17:24:36Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)
• DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems [91.45207885902786]
  We propose a novel end-to-end trainable model named DeProCams to learn the photometric and geometric mappings of ProCams. DeProCams explicitly decomposes the projector-camera image mappings into three subprocesses: shading attributes estimation, rough direct light estimation and photorealistic neural rendering. In our experiments, DeProCams shows clear advantages over previous arts with promising quality and being fully differentiable.
  arXiv  Detail & Related papers  (2020-03-06T05:49:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.