NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References

• URL: http://arxiv.org/abs/2501.06488v1
• Date: Sat, 11 Jan 2025 09:12:43 GMT
• Title: NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References
• Authors: Qiang Qu, Yiran Shen, Xiaoming Chen, Yuk Ying Chung, Weidong Cai, Tongliang Liu,
• Abstract summary: We propose NVS-SQA, a quality assessment method to learn no-reference quality representations through self-supervision. Traditional self-supervised learning predominantly relies on the "same instance, similar representation" assumption and extensive datasets. We employ photorealistic cues and quality scores as learning objectives, along with a specialized contrastive pair preparation process to improve the effectiveness and efficiency of learning.
• Score: 55.35182166250742
• License:
• Abstract: Neural View Synthesis (NVS), such as NeRF and 3D Gaussian Splatting, effectively creates photorealistic scenes from sparse viewpoints, typically evaluated by quality assessment methods like PSNR, SSIM, and LPIPS. However, these full-reference methods, which compare synthesized views to reference views, may not fully capture the perceptual quality of neurally synthesized scenes (NSS), particularly due to the limited availability of dense reference views. Furthermore, the challenges in acquiring human perceptual labels hinder the creation of extensive labeled datasets, risking model overfitting and reduced generalizability. To address these issues, we propose NVS-SQA, a NSS quality assessment method to learn no-reference quality representations through self-supervision without reliance on human labels. Traditional self-supervised learning predominantly relies on the "same instance, similar representation" assumption and extensive datasets. However, given that these conditions do not apply in NSS quality assessment, we employ heuristic cues and quality scores as learning objectives, along with a specialized contrastive pair preparation process to improve the effectiveness and efficiency of learning. The results show that NVS-SQA outperforms 17 no-reference methods by a large margin (i.e., on average 109.5% in SRCC, 98.6% in PLCC, and 91.5% in KRCC over the second best) and even exceeds 16 full-reference methods across all evaluation metrics (i.e., 22.9% in SRCC, 19.1% in PLCC, and 18.6% in KRCC over the second best).

Related papers

• NeRF-NQA: No-Reference Quality Assessment for Scenes Generated by NeRF and Neural View Synthesis Methods [13.403739247879766]
  We propose NeRF-NQA, the first no-reference quality assessment method for densely-observed scenes synthesized from the NVS and NeRF variants. NeRF-NQA employs a joint quality assessment strategy, integrating both viewwise and pointwise approaches. The viewwise approach assesses the spatial quality of each individual synthesized view and the overall inter-views consistency, while the pointwise approach focuses on the angular qualities of scene surface points.
  arXiv  Detail & Related papers  (2024-12-11T02:17:33Z)
• Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling [50.08315607506652]
  We propose a Constrained Active Sampling Framework (CASF) for reliable human judgment. Experiment results show CASF receives 93.18% top-ranked system recognition accuracy.
  arXiv  Detail & Related papers  (2024-06-12T07:44:36Z)
• How Quality Affects Deep Neural Networks in Fine-Grained Image Classification [0.799543372823325]
  We propose a No-Reference Image Quality Assessment (NRIQA) guided cut-off point selection (CPS) strategy to enhance the performance of a fine-grained classification system. We take the three most commonly adopted image augmentation configurations -- cropping, rotating, and blurring -- as the entry point. Concretely, the cut-off points yielded by those methods are aggregated via majority voting to inform the process of image subset selection.
  arXiv  Detail & Related papers  (2024-05-09T12:59:11Z)
• Less is More: Learning Reference Knowledge Using No-Reference Image Quality Assessment [58.09173822651016]
  We argue that it is possible to learn reference knowledge under the No-Reference Image Quality Assessment setting. We propose a new framework to learn comparative knowledge from non-aligned reference images. Experiments on eight standard NR-IQA datasets demonstrate the superior performance to the state-of-the-art NR-IQA methods.
  arXiv  Detail & Related papers  (2023-12-01T13:56:01Z)
• SSL-CPCD: Self-supervised learning with composite pretext-class discrimination for improved generalisability in endoscopic image analysis [3.1542695050861544]
  Deep learning-based supervised methods are widely popular in medical image analysis. They require a large amount of training data and face issues in generalisability to unseen datasets. We propose to explore patch-level instance-group discrimination and penalisation of inter-class variation using additive angular margin.
  arXiv  Detail & Related papers  (2023-05-31T21:28:08Z)
• Uncertainty-inspired Open Set Learning for Retinal Anomaly Identification [71.06194656633447]
  We establish an uncertainty-inspired open-set (UIOS) model, which was trained with fundus images of 9 retinal conditions. Our UIOS model with thresholding strategy achieved an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set. UIOS correctly predicted high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images.
  arXiv  Detail & Related papers  (2023-04-08T10:47:41Z)
• Conformer and Blind Noisy Students for Improved Image Quality Assessment [80.57006406834466]
  Learning-based approaches for perceptual image quality assessment (IQA) usually require both the distorted and reference image for measuring the perceptual quality accurately. In this work, we explore the performance of transformer-based full-reference IQA models. We also propose a method for IQA based on semi-supervised knowledge distillation from full-reference teacher models into blind student models.
  arXiv  Detail & Related papers  (2022-04-27T10:21:08Z)
• Contrastive Semi-supervised Learning for ASR [16.070972355201253]
  We propose Contrastive Semi-supervised Learning (CSL) for supervised learning of visual objects. CSL eschews directly predicting teacher-generated pseudo-labels in favor of utilizing them to select positive and negative examples. It reduces the WER by 8% compared to the standard Cross-Entropy pseudo-labeling (CE-PL) when 10hr of supervised data is used to annotate 75,000hr of videos.
  arXiv  Detail & Related papers  (2021-03-09T00:20:37Z)
• Comprehensive evaluation of no-reference image quality assessment algorithms on authentic distortions [0.0]
  No-reference image quality assessment predicts the quality of a given input image without any knowledge or information about its pristine (distortion free) counterpart. In this study, we evaluate several machine learning based NR-IQA methods and one opinion unaware method on databases consisting of authentic distortions.
  arXiv  Detail & Related papers  (2020-10-26T21:25:46Z)
• Learning Expectation of Label Distribution for Facial Age and Attractiveness Estimation [65.5880700862751]
  We analyze the essential relationship between two state-of-the-art methods (Ranking-CNN and DLDL) and show that the Ranking method is in fact learning label distribution implicitly. We propose a lightweight network architecture and propose a unified framework which can jointly learn facial attribute distribution and regress attribute value. Our method achieves new state-of-the-art results using the single model with 36$times$ fewer parameters and 3$times$ faster inference speed on facial age/attractiveness estimation.
  arXiv  Detail & Related papers  (2020-07-03T15:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.