Scanpath Prediction in Panoramic Videos via Expected Code Length Minimization

• URL: http://arxiv.org/abs/2305.02536v2
• Date: Fri, 5 May 2023 03:24:51 GMT
• Title: Scanpath Prediction in Panoramic Videos via Expected Code Length Minimization
• Authors: Mu Li, Kanglong Fan, Kede Ma
• Abstract summary: We present a new criterion for scanpath prediction based on principles from lossy data compression. This criterion suggests minimizing the expected code length of quantized scanpaths in a training set. We also introduce a proportional-integral-derivative (PID) controller-based sampler to generate realistic human-like scanpaths.
• Score: 27.06179638588126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting human scanpaths when exploring panoramic videos is a challenging task due to the spherical geometry and the multimodality of the input, and the inherent uncertainty and diversity of the output. Most previous methods fail to give a complete treatment of these characteristics, and thus are prone to errors. In this paper, we present a simple new criterion for scanpath prediction based on principles from lossy data compression. This criterion suggests minimizing the expected code length of quantized scanpaths in a training set, which corresponds to fitting a discrete conditional probability model via maximum likelihood. Specifically, the probability model is conditioned on two modalities: a viewport sequence as the deformation-reduced visual input and a set of relative historical scanpaths projected onto respective viewports as the aligned path input. The probability model is parameterized by a product of discretized Gaussian mixture models to capture the uncertainty and the diversity of scanpaths from different users. Most importantly, the training of the probability model does not rely on the specification of "ground-truth" scanpaths for imitation learning. We also introduce a proportional-integral-derivative (PID) controller-based sampler to generate realistic human-like scanpaths from the learned probability model. Experimental results demonstrate that our method consistently produces better quantitative scanpath results in terms of prediction accuracy (by comparing to the assumed "ground-truths") and perceptual realism (through machine discrimination) over a wide range of prediction horizons. We additionally verify the perceptual realism improvement via a formal psychophysical experiment and the generalization improvement on several unseen panoramic video datasets.

Related papers

• EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning [31.583764158565916]
  We present EyeFormer, a machine learning model for predicting scanpaths in a visual user interface. Our model has the unique capability of producing personalized predictions when given a few user scanpath samples. It can predict full scanpath information, including fixation positions and duration, across individuals and various stimulus types.
  arXiv  Detail & Related papers  (2024-04-15T22:26:27Z)
• Generalized Differentiable RANSAC [95.95627475224231]
  $nabla$-RANSAC is a differentiable RANSAC that allows learning the entire randomized robust estimation pipeline. $nabla$-RANSAC is superior to the state-of-the-art in terms of accuracy while running at a similar speed to its less accurate alternatives.
  arXiv  Detail & Related papers  (2022-12-26T15:13:13Z)
• A Probabilistic Time-Evolving Approach to Scanpath Prediction [8.669748138523758]
  We present a probabilistic time-evolving approach to scanpath prediction, based on Bayesian deep learning. Our model yields results that outperform those of current state-of-the-art approaches, and are almost on par with the human baseline.
  arXiv  Detail & Related papers  (2022-04-20T11:50:29Z)
• Masked prediction tasks: a parameter identifiability view [49.533046139235466]
  We focus on the widely used self-supervised learning method of predicting masked tokens. We show that there is a rich landscape of possibilities, out of which some prediction tasks yield identifiability, while others do not.
  arXiv  Detail & Related papers  (2022-02-18T17:09:32Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data [1.7474352892977458]
  We show how to detect out-of-distribution anomalies in road-driving scenes and remote sensing imagery. We leverage a jointly trained normalizing flow due to coverage-oriented learning objective and the capability to generate samples at different resolutions. The resulting models set the new state of the art on benchmarks for out-of-distribution detection in road-driving scenes and remote sensing imagery.
  arXiv  Detail & Related papers  (2021-12-23T20:35:10Z)
• PDC-Net+: Enhanced Probabilistic Dense Correspondence Network [161.76275845530964]
  Enhanced Probabilistic Dense Correspondence Network, PDC-Net+, capable of estimating accurate dense correspondences. We develop an architecture and an enhanced training strategy tailored for robust and generalizable uncertainty prediction. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-09-28T17:56:41Z)
• Efficient remedies for outlier detection with variational autoencoders [8.80692072928023]
  Likelihoods computed by deep generative models are a candidate metric for outlier detection with unlabeled data. We show that a theoretically-grounded correction readily ameliorates a key bias with VAE likelihood estimates. We also show that the variance of the likelihoods computed over an ensemble of VAEs also enables robust outlier detection.
  arXiv  Detail & Related papers  (2021-08-19T16:00:58Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• State-of-the-Art in Human Scanpath Prediction [22.030889583780514]
  We evaluate models based on how well they predict each fixation in a scanpath given the previous scanpath history. This makes model evaluation closely aligned with the biological processes thought to underly scanpath generation. We evaluate many existing models of scanpath prediction on the datasets MIT1003, MIT300, CAT2000 train and CAT200 test.
  arXiv  Detail & Related papers  (2021-02-24T12:01:28Z)
• Uncertainty Inspired RGB-D Saliency Detection [70.50583438784571]
  We propose the first framework to employ uncertainty for RGB-D saliency detection by learning from the data labeling process. Inspired by the saliency data labeling process, we propose a generative architecture to achieve probabilistic RGB-D saliency detection. Results on six challenging RGB-D benchmark datasets show our approach's superior performance in learning the distribution of saliency maps.
  arXiv  Detail & Related papers  (2020-09-07T13:01:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.