Meshtryoshka: Differentiable Rendering of Real-World Scenes via Mesh Rasterization

• URL: http://arxiv.org/abs/2606.28622v1
• Date: Fri, 26 Jun 2026 21:44:36 GMT
• Title: Meshtryoshka: Differentiable Rendering of Real-World Scenes via Mesh Rasterization
• Abstract summary: Meshtryoshka is a novel differentiable rendering framework that combines an off-the-shelf triangleizer with a 3D representation that resembles a matryoshka doll.<n>Our method performs competitively with surface-based differentiable rendering techniques.
• Score: 14.556821243323272
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differentiable rendering has emerged as a powerful approach for 3D reconstruction and novel view synthesis. State-of-the-art differentiable rendering methods combine a variety of custom representations of 3D geometry and appearance with specialized renderers. However, most downstream tasks in computer graphics rely on 3D meshes. While prior work has attempted differentiable rendering with mesh representations, these approaches are limited to object-centric scenes and fail to reconstruct large-scale, unbounded scenes. In this work, we introduce Meshtryoshka, a novel mesh differentiable rendering framework that combines an off-the-shelf triangle rasterizer with a 3D representation that consists of nested mesh shells which resemble a matryoshka doll. In every forward pass, the mesh shells are extracted anew from a 3D signed distance function via iso-surface extraction, and the opacities for each vertex are computed as a function of signed distance. Each mesh shell is then rasterized independently, and the final image is created via alpha compositing. Crucially, mesh vertex positions are updated only indirectly via gradients that flow through the opacity values into the signed distance function, and hence, our method is compatible with off-the-shelf mesh renderers that need not be differentiable with respect to vertex positions. On object-centric scenes, our method performs competitively with surface-based differentiable rendering techniques. Our differentiable mesh rendering method scales to unbounded, real-world 3D scenes, where it yields high-quality novel view synthesis results approaching those of state-of-the-art, non-mesh methods. Our method suggests that it may be possible to solve the differentiable rendering problem without relying on specialized renderers, only using conventional tools from the computer graphics toolbox.

Related papers

• MeshSplatting: Differentiable Rendering with Opaque Meshes [59.240722437975755]
  We present MeshSplatting, a mesh-based reconstruction approach that jointly optimize geometry and appearance through differentiable rendering.<n>On Mip-NeRF360, it boosts PSNR by +0.69 dB over the current state-of-the-art MiLo for mesh-based novel view synthesis.
  arXiv  Detail & Related papers  (2025-12-07T12:31:04Z)
• 2D Triangle Splatting for Direct Differentiable Mesh Training [4.161453036693641]
  2D Triangle Splatting (2DTS) is a novel method that replaces 3D Gaussian primitives with 2D triangle facelets.<n>By incorporating a compactness parameter into the triangle primitives, we enable direct training of photorealistic meshes.<n>Our approach produces reconstructed meshes with superior visual quality compared to existing mesh reconstruction methods.
  arXiv  Detail & Related papers  (2025-06-23T12:26:47Z)
• TriTex: Learning Texture from a Single Mesh via Triplane Semantic Features [78.13246375582906]
  We present a novel approach that learns a volumetric texture field from a single textured mesh by mapping semantic features to surface target colors.<n>Our approach achieves superior texture quality across 3D models in applications like game development.
  arXiv  Detail & Related papers  (2025-03-20T18:35:03Z)
• 3D Gaussian Splatting with Normal Information for Mesh Extraction and Improved Rendering [8.59572577251833]
  We propose a novel regularization method using the gradients of a signed distance function estimated from the Gaussians.<n>We demonstrate the effectiveness of our approach on datasets such as Mip-NeRF360, Tanks and Temples, and Deep-Blending.
  arXiv  Detail & Related papers  (2025-01-14T18:40:33Z)
• Hybrid Explicit Representation for Ultra-Realistic Head Avatars [55.829497543262214]
  We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time.<n> UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures.<n>Experiments that our modeled results exceed those of state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-18T04:01:26Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)
• Approximate Differentiable Rendering with Algebraic Surfaces [24.7500811470085]
  Fuzzy Metaballs is an approximate differentiable for a compact, interpretable representation. Our approximate focuses on rendering shapes via depth maps and silhouettes. Compared to mesh-based differentiables, our method has forward passes that are 5x faster and backwards passes that are 30x faster.
  arXiv  Detail & Related papers  (2022-07-21T16:59:54Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild [96.09941542587865]
  We present a novel 3D pose refinement approach based on differentiable rendering for objects of arbitrary categories in the wild. In this way, we precisely align 3D models to objects in RGB images which results in significantly improved 3D pose estimates. We evaluate our approach on the challenging Pix3D dataset and achieve up to 55% relative improvement compared to state-of-the-art refinement methods in multiple metrics.
  arXiv  Detail & Related papers  (2020-07-17T12:34:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.