ScanGAN360: A Generative Model of Realistic Scanpaths for 360$^{\circ}$ Images

• URL: http://arxiv.org/abs/2103.13922v1
• Date: Thu, 25 Mar 2021 15:34:18 GMT
• Title: ScanGAN360: A Generative Model of Realistic Scanpaths for 360$^{\circ}$ Images
• Authors: Daniel Martin, Ana Serrano, Alexander W. Bergman, Gordon Wetzstein, Belen Masia
• Abstract summary: We present ScanGAN360, a new generative adversarial approach to generate scanpaths for 360$circ$ images. We accomplish this by leveraging the use of a spherical adaptation of dynamic-time warping as a loss function. The quality of our scanpaths outperforms competing approaches by a large margin and is almost on par with the human baseline.
• Score: 92.8211658773467
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Understanding and modeling the dynamics of human gaze behavior in 360$^\circ$ environments is a key challenge in computer vision and virtual reality. Generative adversarial approaches could alleviate this challenge by generating a large number of possible scanpaths for unseen images. Existing methods for scanpath generation, however, do not adequately predict realistic scanpaths for 360$^\circ$ images. We present ScanGAN360, a new generative adversarial approach to address this challenging problem. Our network generator is tailored to the specifics of 360$^\circ$ images representing immersive environments. Specifically, we accomplish this by leveraging the use of a spherical adaptation of dynamic-time warping as a loss function and proposing a novel parameterization of 360$^\circ$ scanpaths. The quality of our scanpaths outperforms competing approaches by a large margin and is almost on par with the human baseline. ScanGAN360 thus allows fast simulation of large numbers of virtual observers, whose behavior mimics real users, enabling a better understanding of gaze behavior and novel applications in virtual scene design.

Related papers

• Pathformer3D: A 3D Scanpath Transformer for 360° Images [1.8857725185112681]
  Existing scanpath prediction models for 360deg images execute scanpath prediction on 2D equirectangular projection plane, which always result in big error owing to the 2D plane's distortion and coordinate discontinuity. In this work, we perform scanpath prediction for 360deg images in 3D spherical coordinate system and proposed a novel 3D scanpath Transformer named Pathformer3D.
  arXiv  Detail & Related papers  (2024-07-15T09:24:27Z)
• DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting [56.101576795566324]
  We present a text-to-3D 360$circ$ scene generation pipeline. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement. Our method offers a globally consistent 3D scene within a 360$circ$ perspective.
  arXiv  Detail & Related papers  (2024-04-10T10:46:59Z)
• DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images [17.714378486267055]
  We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360deg images. Our evaluations show that DiffGaze outperforms state-of-the-art methods on all tasks.
  arXiv  Detail & Related papers  (2024-03-26T08:13:02Z)
• 360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries [24.345954562946385]
  This paper introduces a new benchmark dataset, 360Loc, composed of 360$circ$ images with ground truth poses for visual localization. We propose a virtual camera approach to generate lower-FoV query frames from 360$circ$ images. We demonstrate that omnidirectional visual localization is more robust in challenging large-scale scenes with symmetries and repetitive structures.
  arXiv  Detail & Related papers  (2023-11-29T06:42:12Z)
• Perceptual Quality Assessment of 360$^\circ$ Images Based on Generative Scanpath Representation [40.00063797833765]
  We introduce a unique generative scanpath representation (GSR) for effective quality inference of 360$circ$ images. GSR aggregates varied perceptual experiences of multi-hypothesis users under a predefined viewing condition. We then propose an efficient OIQA computational framework by learning the quality maps GSR.
  arXiv  Detail & Related papers  (2023-09-07T04:10:30Z)
• Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation [36.45222068699805]
  AOG-Net is proposed for 360-degree image generation by out-painting an incomplete image progressively with NFoV and text guidances joinly or individually. A global-local conditioning mechanism is devised to formulate the outpainting guidance in each autoregressive step. Comprehensive experiments on two commonly used 360-degree image datasets for both indoor and outdoor settings demonstrate the state-of-the-art performance of our proposed method.
  arXiv  Detail & Related papers  (2023-09-07T03:22:59Z)
• NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes [59.15910989235392]
  We introduce NeO 360, Neural fields for sparse view synthesis of outdoor scenes. NeO 360 is a generalizable method that reconstructs 360deg scenes from a single or a few posed RGB images. Our representation combines the best of both voxel-based and bird's-eye-view (BEV) representations.
  arXiv  Detail & Related papers  (2023-08-24T17:59:50Z)
• HQ3DAvatar: High Quality Controllable 3D Head Avatar [65.70885416855782]
  This paper presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. At test time, our method is driven by a monocular RGB video.
  arXiv  Detail & Related papers  (2023-03-25T13:56:33Z)
• NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360{\deg} Views [77.93662205673297]
  In this work, we study the challenging task of lifting a single image to a 3D object. We demonstrate the ability to generate a plausible 3D object with 360deg views that correspond well with a given reference image. We propose a novel framework, dubbed NeuralLift-360, that utilizes a depth-aware radiance representation.
  arXiv  Detail & Related papers  (2022-11-29T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.