Feature-Guided Diffusion for Non-Differentiable Inverse Rendering

• URL: http://arxiv.org/abs/2607.17411v1
• Date: Sun, 19 Jul 2026 21:10:59 GMT
• Title: Feature-Guided Diffusion for Non-Differentiable Inverse Rendering
• Abstract summary: Inverse rendering is traditionally solved via differentiables and gradient descent.<n>We propose Feature-Informed Evolution (FIDE), a fully black-box framework for inverse rendering.
• Score: 7.921541321170412
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inverse rendering is traditionally solved via differentiable renderers and gradient descent, which requires substantial problem-specific engineering and is prone to getting stuck in local minima due to ambiguities. Derivative-free approaches alleviate engineering requirements, but often heavily depend on a good problem initialization. In this work, we propose Feature-Informed Diffusion Evolution (FIDE), a fully black-box framework that requires no gradients or specific initialization: the renderer is treated as an opaque function whose only requirement is to produce images. Our key insight is feature guiding: rather than reducing each candidate rendering to a scalar loss value, we use a Vision Transformer (ViT) to extract dense visual features from it. We subsequently use these features to train a diffusion-based candidate proposal model, allowing the network to use visual cues to predict parameters that would match the target image. The candidate solutions proposed by this diffusion model are then refined in a closed loop with a CMA evolution strategy, continuously narrowing the proposal region as optimization progresses. We validate across diverse inverse problems from path tracing, vector splines, Voronoi shaders, and robotics, and demonstrate that feature-guiding substantially improves convergence over scalar-loss baselines and reliably escapes local minima where gradient-based methods stall.

Related papers

• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• Intern-GS: Vision Model Guided Sparse-View 3D Gaussian Splatting [95.61137026932062]
  Intern-GS is a novel approach to enhance the process of sparse-view Gaussian splatting.<n>We show that Intern-GS achieves state-of-the-art rendering quality across diverse datasets.
  arXiv  Detail & Related papers  (2025-05-27T05:17:49Z)
• Locally Orderless Images for Optimization in Differentiable Rendering [80.09571356394574]
  We introduce a method that uses locally orderless images, where each pixel maps to a histogram of intensities that preserves local variations in appearance.<n>We validate our method on various inverse problems using both synthetic and real data.
  arXiv  Detail & Related papers  (2025-03-27T19:17:58Z)
• Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics [13.558059987555234]
  We show how to automatically optimize graphics for undefined or zero gradients. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. In contrast to other algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k variables.
  arXiv  Detail & Related papers  (2023-08-10T17:57:22Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• Plateau-reduced Differentiable Path Tracing [18.174063717952187]
  We show that inverse rendering might not converge due to inherent plateaus, i.e. regions of zero gradient, in the objective function. We propose to alleviate this by convolving the high-dimensional rendering function that maps parameters to images with an additional kernel that blurs the parameter space.
  arXiv  Detail & Related papers  (2022-11-30T18:58:53Z)
• Implicit Neural Representation for Mesh-Free Inverse Obstacle Scattering [21.459567997723376]
  Implicit representation of shapes as level sets of multilayer perceptrons has recently flourished in different shape analysis, compression, and reconstruction tasks. We introduce an implicit neural representation-based framework for solving the inverse obstacle scattering problem in a mesh-free fashion.
  arXiv  Detail & Related papers  (2022-06-04T17:16:09Z)
• RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation [110.4255414234771]
  Existing solutions require massive training data or lack generalizability to unknown rendering configurations. We propose a novel approach that marries domain randomization and differentiable rendering gradients to address this problem. Our approach achieves significantly lower reconstruction errors and has better generalizability among unknown rendering configurations.
  arXiv  Detail & Related papers  (2022-05-11T17:59:51Z)
• Differentiable Rendering with Perturbed Optimizers [85.66675707599782]
  Reasoning about 3D scenes from their 2D image projections is one of the core problems in computer vision. Our work highlights the link between some well-known differentiable formulations and randomly smoothed renderings. We apply our method to 3D scene reconstruction and demonstrate its advantages on the tasks of 6D pose estimation and 3D mesh reconstruction.
  arXiv  Detail & Related papers  (2021-10-18T08:56:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.