Learning to Rasterize Differentiably

• URL: http://arxiv.org/abs/2211.13333v2
• Date: Mon, 15 Jul 2024 17:36:45 GMT
• Title: Learning to Rasterize Differentiably
• Abstract summary: Differentiableization changes the standard formulation of primitiveization, by enabling flow from a pixel to its underlying triangles. Previous work has analyzed and compared several combinations of softening. In this work, we take it a step further and, instead of making a choice of softening operations, parameterize the continuous space of common softening operations.
• Score: 10.790255179159459
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differentiable rasterization changes the standard formulation of primitive rasterization -- by enabling gradient flow from a pixel to its underlying triangles -- using distribution functions in different stages of rendering, creating a "soft" version of the original rasterizer. However, choosing the optimal softening function that ensures the best performance and convergence to a desired goal requires trial and error. Previous work has analyzed and compared several combinations of softening. In this work, we take it a step further and, instead of making a combinatorial choice of softening operations, parameterize the continuous space of common softening operations. We study meta-learning tunable softness functions over a set of inverse rendering tasks (2D and 3D shape, pose and occlusion) so it generalizes to new and unseen differentiable rendering tasks with optimal softness.

Related papers

• GloSplat: Joint Pose-Appearance Optimization for Faster and More Accurate 3D Reconstruction [35.30036388020098]
  We present GloSplat, a framework that performs emphjoint pose-appearance optimization during 3D Gaussian Splatting training.<n>Unlike prior joint optimization methods, GloSplat preserves emphexplicit SfM feature tracks as first-class entities throughout training.<n>Experiments demonstrate that GloSplat-F achieves state-of-the-art among COLMAP-free methods while GloSplat-A surpasses all COLMAP-based baselines.
  arXiv  Detail & Related papers  (2026-03-05T06:02:50Z)
• Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding [86.55824709875598]
  We propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches.<n>Unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor to capture fine-grained 3D shape details.<n>We employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations.
  arXiv  Detail & Related papers  (2026-01-05T18:33:50Z)
• FlowLUT: Efficient Image Enhancement via Differentiable LUTs and Iterative Flow Matching [10.213645938731338]
  FlowLUT is a novel end-to-end model that integrates the efficiency of LUTs, multiple priors, and the parameter-independent characteristic of flow-matched reconstructed images.<n>A lightweight fusion prediction network runs on multiple 3D LUTs, with $mathcalO(1)$ complexity for scene-adaptive color correction.<n>The entire model is jointly optimized under a composite loss function enforcing perceptual and structural fidelity.
  arXiv  Detail & Related papers  (2025-09-28T03:22:01Z)
• BG-Triangle: Bézier Gaussian Triangle for 3D Vectorization and Rendering [60.240908644910874]
  Differentiable rendering enables efficient optimization by allowing gradients to be computed through the rendering process. Existing solutions approximate or re-formulate traditional rendering operations using smooth, probabilistic proxies. We present a novel hybrid representation that combines B'ezier triangle-based vector graphics primitives with Gaussian-based probabilistic models.
  arXiv  Detail & Related papers  (2025-03-18T06:53:52Z)
• LinPrim: Linear Primitives for Differentiable Volumetric Rendering [53.780682194322225]
  We introduce two new scene representations based on linear primitives. We present a different octaiableizer that runs efficiently on GPU. We demonstrate comparable performance to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-27T18:49:38Z)
• Towards Realistic Example-based Modeling via 3D Gaussian Stitching [31.710954782769377]
  We present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis. Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models. For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported.
  arXiv  Detail & Related papers  (2024-08-28T11:13:27Z)
• Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS [52.3215552448623]
  Novel View Synthesis (NVS) without Structure-from-Motion (SfM) pre-processed camera poses are crucial for promoting rapid response capabilities and enhancing robustness against variable operating conditions. Recent SfM-free methods have integrated pose optimization, designing end-to-end frameworks for joint camera pose estimation and NVS. Most existing works rely on per-pixel image loss functions, such as L2 loss. In this study, we propose a correspondence-guided SfM-free 3D Gaussian splatting for NVS.
  arXiv  Detail & Related papers  (2024-08-16T13:11:22Z)
• Euler's Elastica Based Cartoon-Smooth-Texture Image Decomposition [4.829677240798159]
  We propose a novel model for decomposing grayscale images into three distinct components. The structural part represents strong boundaries and regions with strong light-to-dark transitions; the smooth part, capturing soft shadows and shadows; and the oscillatory, characterizing textures and noise.
  arXiv  Detail & Related papers  (2024-07-03T03:42:33Z)
• Rasterized Edge Gradients: Handling Discontinuities Differentiably [25.85191317712521]
  We present a novel method for computing gradients at discontinuities for rendering approximations. Our method elegantly simplifies the traditionally complex problem through a carefully designed approximation strategy. We showcase our method in human head scene reconstruction, demonstrating handling of camera images and segmentation masks.
  arXiv  Detail & Related papers  (2024-05-03T22:42:00Z)
• Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization [105.14877281665011]
  We present SyNoRiM, a novel way to register multiple non-rigid shapes by synchronizing the maps relating learned functions defined on the point clouds. We demonstrate via extensive experiments that our method achieves a state-of-the-art performance in registration accuracy.
  arXiv  Detail & Related papers  (2021-11-25T02:37:59Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Differentiable Rendering with Perturbed Optimizers [85.66675707599782]
  Reasoning about 3D scenes from their 2D image projections is one of the core problems in computer vision. Our work highlights the link between some well-known differentiable formulations and randomly smoothed renderings. We apply our method to 3D scene reconstruction and demonstrate its advantages on the tasks of 6D pose estimation and 3D mesh reconstruction.
  arXiv  Detail & Related papers  (2021-10-18T08:56:23Z)
• Efficient and Differentiable Shadow Computation for Inverse Problems [64.70468076488419]
  Differentiable geometric computation has received increasing interest for image-based inverse problems. We propose an efficient yet efficient approach for differentiable visibility and soft shadow computation. As our formulation is differentiable, it can be used to solve inverse problems such as texture, illumination, rigid pose, and deformation recovery from images.
  arXiv  Detail & Related papers  (2021-04-01T09:29:05Z)
• A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding [57.1077544780653]
  We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
  arXiv  Detail & Related papers  (2020-06-26T08:34:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.