Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization

• URL: http://arxiv.org/abs/2204.01386v1
• Date: Mon, 4 Apr 2022 11:07:03 GMT
• Title: Dressi: A Hardware-Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization
• Authors: Yusuke Takimoto, Hiroyuki Sato, Hikari Takehara, Keishiro Uragaki, Takehiro Tawara, Xiao Liang, Kentaro Oku, Wataru Kishimoto, Bo Zheng
• Abstract summary: Differentiable rendering (DR) enables various computer graphics and computer vision applications through gradient-based optimization with derivatives of the rendering equation. Mostization-based approaches are built on general-purpose automatic differentiation (AD) libraries and DR-specific modules handcrafted using runtime. We present a practical hardware-agnostic differentiable called Dressi, which is based on a new full AD design.
• Score: 6.443504994276216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable rendering (DR) enables various computer graphics and computer vision applications through gradient-based optimization with derivatives of the rendering equation. Most rasterization-based approaches are built on general-purpose automatic differentiation (AD) libraries and DR-specific modules handcrafted using CUDA. Such a system design mixes DR algorithm implementation and algorithm building blocks, resulting in hardware dependency and limited performance. In this paper, we present a practical hardware-agnostic differentiable renderer called Dressi, which is based on a new full AD design. The DR algorithms of Dressi are fully written in our Vulkan-based AD for DR, Dressi-AD, which supports all primitive operations for DR. Dressi-AD and our inverse UV technique inside it bring hardware independence and acceleration by graphics hardware. Stage packing, our runtime optimization technique, can adapt hardware constraints and efficiently execute complex computational graphs of DR with reactive cache considering the render pass hierarchy of Vulkan. HardSoftRas, our novel rendering process, is designed for inverse rendering with a graphics pipeline. Under the limited functionalities of the graphics pipeline, HardSoftRas can propagate the gradients of pixels from the screen space to far-range triangle attributes. Our experiments and applications demonstrate that Dressi establishes hardware independence, high-quality and robust optimization with fast speed, and photorealistic rendering.

Related papers

• Hardware-Rasterized Ray-Based Gaussian Splatting [21.088504768655618]
  We present a novel, hardwareized rendering approach for ray-based 3D Gaussian Splatting (RayGS) Our solution is the first enabling rendering RayGS models at sufficiently high frame rates to support quality-sensitive applications like Virtual and Mixed Reality. Our second contribution enables alias-free rendering for RayGS, by addressing MIP-related issues during training and testing.
  arXiv  Detail & Related papers  (2025-03-24T13:53:30Z)
• VR-Pipe: Streamlining Hardware Graphics Pipeline for Volume Rendering [1.9470707535768061]
  We implement the state-of-the-art radiance field method, 3D Gaussian splatting, using graphics APIs and evaluate it across synthetic and real-world scenes on today's graphics hardware. We present VR-Pipe, which seamlessly integrates two innovations into graphics hardware to streamline the hardware pipeline for volume rendering. Our evaluation shows that VR-Pipe greatly improves rendering performance, achieving up to a 2.78x speedup over the conventional graphics pipeline with negligible hardware overhead.
  arXiv  Detail & Related papers  (2025-02-24T11:46:36Z)
• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• FlashGS: Efficient 3D Gaussian Splatting for Large-scale and High-resolution Rendering [11.727367585102112]
  FlashGS is designed to facilitate the efficient differentiableization of 3D Gaussian Splatting. An extensive evaluation of FlashGS' performance has been conducted across a diverse spectrum of synthetic and real-world large-scale scenes. Results underscore the superior performance and resource optimization capabilities of FlashGS, positioning it as a formidable tool in the domain of 3D rendering.
  arXiv  Detail & Related papers  (2024-08-15T06:27:42Z)
• 3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes [50.36933474990516]
  This work considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized algorithm which encapsulates particles with bounding meshes. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision.
  arXiv  Detail & Related papers  (2024-07-09T17:59:30Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [55.15950594752051]
  We propose Image-GS, a content-adaptive image representation. Using anisotropic 2D Gaussians as the basis, Image-GS shows high memory efficiency, supports fast random access, and offers a natural level of detail stack. General efficiency and fidelity of Image-GS are validated against several recent neural image representations and industry-standard texture compressors. We hope this research offers insights for developing new applications that require adaptive quality and resource control, such as machine perception, asset streaming, and content generation.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces [21.946327323788275]
  3D rendering of dynamic face is a challenging problem. We present a novel representation that enables high-quality rendering of an actor's dynamic facial performances.
  arXiv  Detail & Related papers  (2024-04-22T00:44:13Z)
• Mondrian: On-Device High-Performance Video Analytics with Compressive Packed Inference [7.624476059109304]
  Mondrian is an edge system that enables high-performance object detection on high-resolution video streams. We devise a novel Compressive Packed Inference to minimize per-pixel processing costs.
  arXiv  Detail & Related papers  (2024-03-12T12:35:12Z)
• EvaSurf: Efficient View-Aware Implicit Textured Surface Reconstruction on Mobile Devices [53.28220984270622]
  We present an implicit textured $textbfSurf$ace reconstruction method on mobile devices. Our method can reconstruct high-quality appearance and accurate mesh on both synthetic and real-world datasets. Our method can be trained in just 1-2 hours using a single GPU and run on mobile devices at over 40 FPS (Frames Per Second)
  arXiv  Detail & Related papers  (2023-11-16T11:30:56Z)
• Flexible Techniques for Differentiable Rendering with 3D Gaussians [29.602516169951556]
  Neural Radiance Fields demonstrated photorealistic novel view is within reach, but was gated by performance requirements for fast reconstruction of real scenes and objects. We develop extensions to alternative shape representations, in particular, 3D watertight meshes and rendering per-ray normals. These reconstructions are quick, robust, and easily performed on GPU or CPU.
  arXiv  Detail & Related papers  (2023-08-28T17:38:31Z)
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing [66.00729477511219]
  Given a function represented as a computation graph, traditional architectures face challenges in efficiently computing its nth-order gradient. We introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture. We present results that demonstrate 1.8-4.8x and 1.5-3.6x speedup compared to CPU and GPU baselines respectively.
  arXiv  Detail & Related papers  (2023-08-11T04:24:39Z)
• Deep Direct Volume Rendering: Learning Visual Feature Mappings From Exemplary Images [57.253447453301796]
  We introduce Deep Direct Volume Rendering (DeepDVR), a generalization of Direct Volume Rendering (DVR) that allows for the integration of deep neural networks into the DVR algorithm. We conceptualize the rendering in a latent color space, thus enabling the use of deep architectures to learn implicit mappings for feature extraction and classification. Our generalization serves to derive novel volume rendering architectures that can be trained end-to-end directly from examples in image space.
  arXiv  Detail & Related papers  (2021-06-09T23:03:00Z)
• Modular Primitives for High-Performance Differentiable Rendering [37.12883917723895]
  We present a modular differentiable design that yields performance superior to previous methods by existing, highly optimized graphics pipelines. Our design supports all crucial operations in a modern graphics pipeline: leveraging large numbers of triangles, attribute, filtered texture lookups, as well as user-programmable shading and geometry processing.
  arXiv  Detail & Related papers  (2020-11-06T10:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.