PatchEX: High-Quality Real-Time Temporal Supersampling through Patch-based Parallel Extrapolation

• URL: http://arxiv.org/abs/2407.17501v1
• Date: Fri, 5 Jul 2024 13:59:05 GMT
• Title: PatchEX: High-Quality Real-Time Temporal Supersampling through Patch-based Parallel Extrapolation
• Authors: Akanksha Dixit, Smruti R. Sarangi,
• Abstract summary: This paper introduces PatchEX, a novel frame extrapolation method that aims to provide the quality of at the speed of extrapolation. PatchEX achieves a 65.29% and 48.46% improvement in PSNR over the latest extrapolation methods ExtraNet and ExtraSS, respectively.
• Score: 0.4143603294943439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-refresh rate displays have become very popular in recent years due to the need for superior visual quality in gaming, professional displays and specialized applications like medical imaging. However, high-refresh rate displays alone do not guarantee a superior visual experience; the GPU needs to render frames at a matching rate. Otherwise, we observe disconcerting visual artifacts such as screen tearing and stuttering. Temporal supersampling is an effective technique to increase frame rates by predicting new frames from other rendered frames. There are two methods in this space: interpolation and extrapolation. Interpolation-based methods provide good image quality at the cost of a higher latency because they also require the next rendered frame. On the other hand, extrapolation methods are much faster at the cost of quality. This paper introduces PatchEX, a novel frame extrapolation method that aims to provide the quality of interpolation at the speed of extrapolation. It smartly partitions the extrapolation task into sub-tasks and executes them in parallel to improve both quality and latency. It then uses a patch-based inpainting method and a custom shadow prediction approach to fuse the generated sub-frames. This approach significantly reduces the overall latency while maintaining the quality of the output. Our results demonstrate that PatchEX achieves a 65.29% and 48.46% improvement in PSNR over the latest extrapolation methods ExtraNet and ExtraSS, respectively, while being 6x and 2x faster, respectively.

Related papers

• ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler [53.98558445900626]
  Current image-to-video diffusion models, while powerful in generating videos from a single frame, need adaptation for two-frame conditioned generation. We introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
  arXiv  Detail & Related papers  (2024-10-08T03:01:54Z)
• GFFE: G-buffer Free Frame Extrapolation for Low-latency Real-time Rendering [14.496161390319065]
  We propose GFFE, with a novel framework and an efficient neural network, to generate new frames in real-time without introducing additional latency. We analyze the motion of dynamic fragments and different types of disocclusions, and design the corresponding modules. After filling disocclusions, a light-weight shading correction network is used to correct shading and improve overall quality.
  arXiv  Detail & Related papers  (2024-05-23T18:35:26Z)
• Low-latency Space-time Supersampling for Real-time Rendering [18.525909461287835]
  Existing techniques often suffer from quality and latency issues due to disjointed treatment of frame supersampling and extrapolation. In this paper, we recognize the shared context and mechanisms between frame supersampling and extrapolation, and present a novel framework, Space-time Supersampling (STSS) To implement an efficient architecture, we treat the aliasing and warping holes unified as reshading regions and put forth two key components to compensate the regions, namely Random Reshading Masking (RRM) and Efficient Reshading Module (ERM)
  arXiv  Detail & Related papers  (2023-12-18T02:37:30Z)
• ExWarp: Extrapolation and Warping-based Temporal Supersampling for High-frequency Displays [0.7734726150561089]
  High-frequency displays are gaining immense popularity because of their increasing use in video games and virtual reality applications. This paper proposes increasing the frame rate to provide a smooth experience on modern displays by predicting new frames based on past or future frames.
  arXiv  Detail & Related papers  (2023-07-24T08:32:27Z)
• Scaling Neural Face Synthesis to High FPS and Low Latency by Neural Caching [12.362614824541824]
  Recent neural rendering approaches greatly improve image quality, reaching near photorealism. The underlying neural networks have high runtime, precluding telepresence and virtual reality applications that require high resolution at low latency. We break this dependency by caching information from the previous frame to speed up the processing of the current one with an implicit warp. We test the approach on view-dependent rendering of 3D portrait avatars, as needed for telepresence, on established benchmark sequences.
  arXiv  Detail & Related papers  (2022-11-10T18:58:00Z)
• Differentiable Point-Based Radiance Fields for Efficient View Synthesis [57.56579501055479]
  We propose a differentiable rendering algorithm for efficient novel view synthesis. Our method is up to 300x faster than NeRF in both training and inference. For dynamic scenes, our method trains two orders of magnitude faster than STNeRF and renders at near interactive rate.
  arXiv  Detail & Related papers  (2022-05-28T04:36:13Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling [52.425236515695914]
  State-of-the-art methods are iterative solutions interpolating one frame at the time. This work introduces a true multi-frame interpolator. It utilizes a pyramidal style network in the temporal domain to complete the multi-frame task in one-shot.
  arXiv  Detail & Related papers  (2020-07-23T02:34:39Z)
• Is There Tradeoff between Spatial and Temporal in Video Super-Resolution? [50.70768797616581]
  Advanced algorithms have been proposed to exploit the temporal correlation between low-resolution (LR) video frames, and/or to super-resolve a frame with multiple LR frames. These methods pursue higher quality of super-resolved frames, where the quality is usually measured frame by frame in e.g. PSNR. It is a natural requirement to improve both frame-wise fidelity and between-frame consistency, which are termed spatial quality and temporal quality, respectively.
  arXiv  Detail & Related papers  (2020-03-13T07:49:05Z)
• Efficient Video Semantic Segmentation with Labels Propagation and Refinement [138.55845680523908]
  This paper tackles the problem of real-time semantic segmentation of high definition videos using a hybrid GPU / CPU approach. We propose an Efficient Video(EVS) pipeline that combines: (i) On the CPU, a very fast optical flow method, that is used to exploit the temporal aspect of the video and propagate semantic information from one frame to the next. On the popular Cityscapes dataset with high resolution frames (2048 x 1024), the proposed operating points range from 80 to 1000 Hz on a single GPU and CPU.
  arXiv  Detail & Related papers  (2019-12-26T11:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.