SphereDiff: Tuning-free Omnidirectional Panoramic Image and Video Generation via Spherical Latent Representation

• URL: http://arxiv.org/abs/2504.14396v1
• Date: Sat, 19 Apr 2025 19:59:11 GMT
• Title: SphereDiff: Tuning-free Omnidirectional Panoramic Image and Video Generation via Spherical Latent Representation
• Authors: Minho Park, Taewoong Kang, Jooyeol Yun, Sungwon Hwang, Jaegul Choo,
• Abstract summary: We introduce SphereDiff, a novel approach for seamless 360-degree panoramic image and video generation using state-of-the-art diffusion models without additional tuning.<n>We extend MultiDiffusion to spherical latent space and propose a spherical latent sampling method to enable direct use of pretrained diffusion models.<n>Our method outperforms existing approaches in generating 360-degree panoramic content while maintaining high fidelity, making it a robust solution for immersive AR/VR applications.
• Score: 31.305851707485967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increasing demand for AR/VR applications has highlighted the need for high-quality 360-degree panoramic content. However, generating high-quality 360-degree panoramic images and videos remains a challenging task due to the severe distortions introduced by equirectangular projection (ERP). Existing approaches either fine-tune pretrained diffusion models on limited ERP datasets or attempt tuning-free methods that still rely on ERP latent representations, leading to discontinuities near the poles. In this paper, we introduce SphereDiff, a novel approach for seamless 360-degree panoramic image and video generation using state-of-the-art diffusion models without additional tuning. We define a spherical latent representation that ensures uniform distribution across all perspectives, mitigating the distortions inherent in ERP. We extend MultiDiffusion to spherical latent space and propose a spherical latent sampling method to enable direct use of pretrained diffusion models. Moreover, we introduce distortion-aware weighted averaging to further improve the generation quality in the projection process. Our method outperforms existing approaches in generating 360-degree panoramic content while maintaining high fidelity, making it a robust solution for immersive AR/VR applications. The code is available here. https://github.com/pmh9960/SphereDiff

Related papers

• Beyond Wide-Angle Images: Unsupervised Video Portrait Correction via Spatiotemporal Diffusion Adaptation [46.16087086554505]
  We propose an image portrait correction framework using diffusion models named ImagePD.<n>It integrates the long-range awareness of transformer and multi-step denoising of diffusion models into a unified framework.<n> Experiments demonstrate that the proposed methods outperform existing solutions quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2025-04-01T03:49:59Z)
• 3DEnhancer: Consistent Multi-View Diffusion for 3D Enhancement [66.8116563135326]
  We present 3DEnhancer, which employs a multi-view latent diffusion model to enhance coarse 3D inputs while preserving multi-view consistency.<n>Unlike existing video-based approaches, our model supports seamless multi-view enhancement with improved coherence across diverse viewing angles.
  arXiv  Detail & Related papers  (2024-12-24T17:36:34Z)
• 4Diffusion: Multi-view Video Diffusion Model for 4D Generation [55.82208863521353]
  Current 4D generation methods have achieved noteworthy efficacy with the aid of advanced diffusion generative models. We propose a novel 4D generation pipeline, namely 4Diffusion, aimed at generating spatial-temporally consistent 4D content from a monocular video.
  arXiv  Detail & Related papers  (2024-05-31T08:18:39Z)
• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation [29.818827785812086]
  Controllable generation of 3D assets is important for many practical applications like content creation in movies, games and engineering, as well as in AR/VR. We present a suitable representation for 3D diffusion models to enable disentanglement by introducing a hybrid point cloud and neural radiance field approach.
  arXiv  Detail & Related papers  (2023-12-21T18:46:27Z)
• DreamDrone: Text-to-Image Diffusion Models are Zero-shot Perpetual View Generators [56.994967294931286]
  We introduce DreamDrone, a novel zero-shot and training-free pipeline for generating flythrough scenes from textual prompts. We advocate explicitly warping the intermediate latent code of the pre-trained text-to-image diffusion model for high-quality image generation and unbounded generalization ability.
  arXiv  Detail & Related papers  (2023-12-14T08:42:26Z)
• CAD: Photorealistic 3D Generation via Adversarial Distillation [28.07049413820128]
  We propose a novel learning paradigm for 3D synthesis that utilizes pre-trained diffusion models. Our method unlocks the generation of high-fidelity and photorealistic 3D content conditioned on a single image and prompt.
  arXiv  Detail & Related papers  (2023-12-11T18:59:58Z)
• EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion [60.30030562932703]
  EpiDiff is a localized interactive multiview diffusion model. It generates 16 multiview images in just 12 seconds. It surpasses previous methods in quality evaluation metrics.
  arXiv  Detail & Related papers  (2023-12-11T05:20:52Z)
• Light Field Diffusion for Single-View Novel View Synthesis [32.59286750410843]
  Single-view novel view synthesis (NVS) is important but challenging in computer vision. Recent advancements in NVS have leveraged Denoising Diffusion Probabilistic Models (DDPMs) for their exceptional ability to produce high-fidelity images. We present Light Field Diffusion (LFD), a novel conditional diffusion-based approach that transcends the conventional reliance on camera pose matrices.
  arXiv  Detail & Related papers  (2023-09-20T03:27:06Z)
• Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction [60.52716381465063]
  We introduce Deceptive-NeRF/3DGS to enhance sparse-view reconstruction with only a limited set of input images. Specifically, we propose a deceptive diffusion model turning noisy images rendered from few-view reconstructions into high-quality pseudo-observations. Our system progressively incorporates diffusion-generated pseudo-observations into the training image sets, ultimately densifying the sparse input observations by 5 to 10 times.
  arXiv  Detail & Related papers  (2023-05-24T14:00:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.