Learn how to Prune Pixels for Multi-view Neural Image-based Synthesis

• URL: http://arxiv.org/abs/2305.03572v1
• Date: Fri, 5 May 2023 14:29:24 GMT
• Title: Learn how to Prune Pixels for Multi-view Neural Image-based Synthesis
• Authors: Marta Milovanovi\'c, Enzo Tartaglione, Marco Cagnazzo, F\'elix Henry
• Abstract summary: We present LeHoPP, a method for input pixel pruning. We examine the importance of each input pixel concerning the rendered view, and we avoid the use of irrelevant pixels. Even without retraining the image-based rendering network, our approach shows a good trade-off between synthesis quality and pixel rate.
• Score: 10.571582038258443
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image-based rendering techniques stand at the core of an immersive experience for the user, as they generate novel views given a set of multiple input images. Since they have shown good performance in terms of objective and subjective quality, the research community devotes great effort to their improvement. However, the large volume of data necessary to render at the receiver's side hinders applications in limited bandwidth environments or prevents their employment in real-time applications. We present LeHoPP, a method for input pixel pruning, where we examine the importance of each input pixel concerning the rendered view, and we avoid the use of irrelevant pixels. Even without retraining the image-based rendering network, our approach shows a good trade-off between synthesis quality and pixel rate. When tested in the general neural rendering framework, compared to other pruning baselines, LeHoPP gains between $0.9$ dB and $3.6$ dB on average.

Related papers

• Exploring Multi-view Pixel Contrast for General and Robust Image Forgery Localization [4.8454936010479335]
  We propose a Multi-view Pixel-wise Contrastive algorithm (MPC) for image forgery localization. Specifically, we first pre-train the backbone network with the supervised contrastive loss. Then the localization head is fine-tuned using the cross-entropy loss, resulting in a better pixel localizer.
  arXiv  Detail & Related papers  (2024-06-19T13:51:52Z)
• PixRO: Pixel-Distributed Rotational Odometry with Gaussian Belief Propagation [8.049531918823758]
  In this paper, we address the task of frame-to-frame rotational estimation. Instead of reasoning about relative motion between frames using the full images, distribute the estimation at pixel-level. In this paradigm, each pixel produces an estimate of the global motion by only relying on local information and local message-passing with neighbouring pixels.
  arXiv  Detail & Related papers  (2024-06-14T05:28:45Z)
• Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur [68.24599239479326]
  We develop a hybrid neural rendering model that makes image-based representation and neural 3D representation join forces to render high-quality, view-consistent images. Our model surpasses state-of-the-art point-based methods for novel view synthesis.
  arXiv  Detail & Related papers  (2023-04-25T08:36:33Z)
• Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
  Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information. We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering. Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
  arXiv  Detail & Related papers  (2023-04-20T04:05:22Z)
• Dense Pixel-to-Pixel Harmonization via Continuous Image Representation [22.984119094424056]
  We propose a novel image Harmonization method based on Implicit neural Networks (HINet) Inspired by the Retinex theory, we decouple the harmonizations into two parts to respectively capture the content and environment of composite images. Extensive experiments have demonstrated the effectiveness of our method compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-03T02:52:28Z)
• Improving Pixel-Level Contrastive Learning by Leveraging Exogenous Depth Information [7.561849435043042]
  Self-supervised representation learning based on Contrastive Learning (CL) has been the subject of much attention in recent years. In this paper we will focus on the depth information, which can be obtained by using a depth network or measured from available data. We show that using this estimation information in the contrastive loss leads to improved results and that the learned representations better follow the shapes of objects.
  arXiv  Detail & Related papers  (2022-11-18T11:45:39Z)
• PixelFolder: An Efficient Progressive Pixel Synthesis Network for Image Generation [88.55256389703082]
  Pixel is a promising research paradigm for image generation, which can well exploit pixel-wise prior knowledge for generation. In this paper, we propose a progressive pixel synthesis network towards efficient image generation, as Pixel. With much less expenditure, Pixel obtains new state-of-the-art (SOTA) performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2022-04-02T10:55:11Z)
• LookinGood^{\pi}: Real-time Person-independent Neural Re-rendering for High-quality Human Performance Capture [13.026888802770902]
  We propose a novel neural re-rendering approach that is aimed to improve the rendering quality of the low-quality reconstructed results from human performance capture system in real-time. Our key idea is to utilize the rendered image of reconstructed geometry as the guidance to assist the prediction of person-specific details from few reference images. We demonstrate that our method outperforms state-of-the-art methods at producing high-fidelity images on unseen people.
  arXiv  Detail & Related papers  (2021-12-15T11:00:21Z)
• Image Quality Assessment using Contrastive Learning [50.265638572116984]
  We train a deep Convolutional Neural Network (CNN) using a contrastive pairwise objective to solve the auxiliary problem. We show through extensive experiments that CONTRIQUE achieves competitive performance when compared to state-of-the-art NR image quality models. Our results suggest that powerful quality representations with perceptual relevance can be obtained without requiring large labeled subjective image quality datasets.
  arXiv  Detail & Related papers  (2021-10-25T21:01:00Z)
• High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling [122.06593036862611]
  Existing image inpainting methods often produce artifacts when dealing with large holes in real applications. We propose an iterative inpainting method with a feedback mechanism. Experiments show that our method significantly outperforms existing methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2020-05-24T13:23:45Z)
• The Power of Triply Complementary Priors for Image Compressive Sensing [89.14144796591685]
  We propose a joint low-rank deep (LRD) image model, which contains a pair of complementaryly trip priors. We then propose a novel hybrid plug-and-play framework based on the LRD model for image CS. To make the optimization tractable, a simple yet effective algorithm is proposed to solve the proposed H-based image CS problem.
  arXiv  Detail & Related papers  (2020-05-16T08:17:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.