Energy-Based Generative Cooperative Saliency Prediction

• URL: http://arxiv.org/abs/2106.13389v1
• Date: Fri, 25 Jun 2021 02:11:50 GMT
• Title: Energy-Based Generative Cooperative Saliency Prediction
• Authors: Jing Zhang and Jianwen Xie and Zilong Zheng and Nick Barnes
• Abstract summary: We study the saliency prediction problem from the perspective of generative models. We propose a generative cooperative saliency prediction framework based on the generative cooperative networks. Experimental results show that our generative model can achieve state-of-the-art performance.
• Score: 44.85865238229076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conventional saliency prediction models typically learn a deterministic mapping from images to the corresponding ground truth saliency maps. In this paper, we study the saliency prediction problem from the perspective of generative models by learning a conditional probability distribution over saliency maps given an image, and treating the prediction as a sampling process. Specifically, we propose a generative cooperative saliency prediction framework based on the generative cooperative networks, where a conditional latent variable model and a conditional energy-based model are jointly trained to predict saliency in a cooperative manner. We call our model the SalCoopNets. The latent variable model serves as a fast but coarse predictor to efficiently produce an initial prediction, which is then refined by the iterative Langevin revision of the energy-based model that serves as a fine predictor. Such a coarse-to-fine cooperative saliency prediction strategy offers the best of both worlds. Moreover, we generalize our framework to the scenario of weakly supervised saliency prediction, where saliency annotation of training images is partially observed, by proposing a cooperative learning while recovering strategy. Lastly, we show that the learned energy function can serve as a refinement module that can refine the results of other pre-trained saliency prediction models. Experimental results show that our generative model can achieve state-of-the-art performance. Our code is publicly available at: \url{https://github.com/JingZhang617/SalCoopNets}.

Related papers

• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Pyramidal Predictive Network: A Model for Visual-frame Prediction Based on Predictive Coding Theory [1.4610038284393165]
  We propose a novel neural network model for the task of visual-frame prediction. The model is composed of a series of recurrent and convolutional units forming the top-down and bottom-up streams. It learns to predict future frames in a visual sequence, with ConvLSTMs on each layer in the network making local prediction from top to down.
  arXiv  Detail & Related papers  (2022-08-15T06:28:34Z)
• An Energy-Based Prior for Generative Saliency [62.79775297611203]
  We propose a novel generative saliency prediction framework that adopts an informative energy-based model as a prior distribution. With the generative saliency model, we can obtain a pixel-wise uncertainty map from an image, indicating model confidence in the saliency prediction. Experimental results show that our generative saliency model with an energy-based prior can achieve not only accurate saliency predictions but also reliable uncertainty maps consistent with human perception.
  arXiv  Detail & Related papers  (2022-04-19T10:51:00Z)
• Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
  We show that pre-trained language models with few examples show strong prediction bias across labels. Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features. These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
  arXiv  Detail & Related papers  (2022-04-17T15:55:18Z)
• Distributional Gradient Boosting Machines [77.34726150561087]
  Our framework is based on XGBoost and LightGBM. We show that our framework achieves state-of-the-art forecast accuracy.
  arXiv  Detail & Related papers  (2022-04-02T06:32:19Z)
• A General Divergence Modeling Strategy for Salient Object Detection [32.53501439077824]
  Salient object detection is subjective in nature, which implies that multiple estimations should be related to the same input image. Most existing salient object detection models are deterministic following a point to point estimation, making them incapable to estimate the predictive distribution. Although latent variable model based prediction network exists to model the prediction variants, the latent space based on the single clean saliency is less reliable in exploring the subjective nature of saliency.
  arXiv  Detail & Related papers  (2021-11-23T12:47:51Z)
• An Interpretable Probabilistic Model for Short-Term Solar Power Forecasting Using Natural Gradient Boosting [0.0]
  We propose a two stage probabilistic forecasting framework able to generate highly accurate, reliable, and sharp forecasts. The framework offers full transparency on both the point forecasts and the prediction intervals (PIs) To highlight the performance and the applicability of the proposed framework, real data from two PV parks located in Southern Germany are employed.
  arXiv  Detail & Related papers  (2021-08-05T12:59:38Z)
• Video Prediction via Example Guidance [156.08546987158616]
  In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states.
  arXiv  Detail & Related papers  (2020-07-03T14:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.