Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos

• URL: http://arxiv.org/abs/2508.20221v1
• Date: Wed, 27 Aug 2025 19:01:47 GMT
• Title: Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos
• Authors: Mert Cokelek, Halit Ozsoy, Nevrez Imamoglu, Cagri Ozcinar, Inci Ayhan, Erkut Erdem, Aykut Erdem,
• Abstract summary: This study extends the domain of saliency prediction to 360-degree environments, addressing the complexities of spherical distortion and the integration of spatial audio.<n>Motivated by the lack of comprehensive datasets for 360-degree audio-visual saliency prediction, our study curates YT360-EyeTracking, a new dataset of 81 ODVs.<n>Our goal is to explore how to utilize audio-visual cues to effectively predict visual saliency in 360-degree videos.
• Score: 15.59763872743732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Omnidirectional videos (ODVs) are redefining viewer experiences in virtual reality (VR) by offering an unprecedented full field-of-view (FOV). This study extends the domain of saliency prediction to 360-degree environments, addressing the complexities of spherical distortion and the integration of spatial audio. Contextually, ODVs have transformed user experience by adding a spatial audio dimension that aligns sound direction with the viewer's perspective in spherical scenes. Motivated by the lack of comprehensive datasets for 360-degree audio-visual saliency prediction, our study curates YT360-EyeTracking, a new dataset of 81 ODVs, each observed under varying audio-visual conditions. Our goal is to explore how to utilize audio-visual cues to effectively predict visual saliency in 360-degree videos. Towards this aim, we propose two novel saliency prediction models: SalViT360, a vision-transformer-based framework for ODVs equipped with spherical geometry-aware spatio-temporal attention layers, and SalViT360-AV, which further incorporates transformer adapters conditioned on audio input. Our results on a number of benchmark datasets, including our YT360-EyeTracking, demonstrate that SalViT360 and SalViT360-AV significantly outperform existing methods in predicting viewer attention in 360-degree scenes. Interpreting these results, we suggest that integrating spatial audio cues in the model architecture is crucial for accurate saliency prediction in omnidirectional videos. Code and dataset will be available at https://cyberiada.github.io/SalViT360.

Related papers

• OmniAudio: Generating Spatial Audio from 360-Degree Video [91.56286471271513]
  We introduce a novel task, 360V2SA, to generate spatial audio from 360-degree videos.<n>We propose a novel framework OmniAudio, which leverages self-supervised pre-training using both spatial audio data and large-scale non-spatial data.<n>Experiments demonstrate that OmniAudio achieves state-of-the-art performance across both objective and subjective metrics.
  arXiv  Detail & Related papers  (2025-04-21T07:21:28Z)
• MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views [90.26609689682876]
  We introduce MVSplat360, a feed-forward approach for 360deg novel view synthesis (NVS) of diverse real-world scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views.
  arXiv  Detail & Related papers  (2024-11-07T17:59:31Z)
• How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model [50.15552768350462]
  This paper comprehensively investigates audio-visual attention in omnidirectional videos (ODVs) from both subjective and objective perspectives.<n>To advance the research on audio-visual saliency prediction for ODVs, we establish a new benchmark based on the AVS-ODV database.
  arXiv  Detail & Related papers  (2024-08-10T02:45:46Z)
• 360VFI: A Dataset and Benchmark for Omnidirectional Video Frame Interpolation [13.122586587748218]
  This paper introduces the benchmark dataset, 360VFI, for Omnidirectional Video Frame Interpolation. We present a practical implementation that introduces a distortion prior from omnidirectional video into the network to modulate distortions.
  arXiv  Detail & Related papers  (2024-07-19T06:50:24Z)
• Audio-visual Saliency for Omnidirectional Videos [58.086575606742116]
  We first establish the largest audio-visual saliency dataset for omnidirectional videos (AVS-ODV) We analyze the visual attention behavior of the observers under various omnidirectional audio modalities and visual scenes based on the AVS-ODV dataset.
  arXiv  Detail & Related papers  (2023-11-09T08:03:40Z)
• Spherical Vision Transformer for 360-degree Video Saliency Prediction [17.948179628551376]
  We propose a vision-transformer-based model for omnidirectional videos named SalViT360. We introduce a spherical geometry-aware self-attention mechanism that is capable of effective omnidirectional video understanding. Our approach is the first to employ tangent images for omnidirectional saliency prediction prediction, and our experimental results on three ODV saliency datasets demonstrate its effectiveness compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-08-24T18:07:37Z)
• Panoramic Vision Transformer for Saliency Detection in 360{\deg} Videos [48.54829780502176]
  We present a new framework named Panoramic Vision Transformer (PAVER) We design the encoder using Vision Transformer with deformable convolution, which enables us to plug pretrained models from normal videos into our architecture without additional modules or finetuning. We demonstrate the utility of our saliency prediction model with the omnidirectional video quality assessment task in VQA-ODV, where we consistently improve performance without any form of supervision.
  arXiv  Detail & Related papers  (2022-09-19T12:23:34Z)
• Pano-AVQA: Grounded Audio-Visual Question Answering on 360$^\circ$ Videos [42.32743253830288]
  We propose a benchmark named Pano-AVQA as a large-scale grounded audio-visual question answering dataset on panoramic videos. Using 5.4K 360$circ$ video clips harvested online, we collect two types of novel question-answer pairs with bounding-box grounding. We train several transformer-based models from Pano-AVQA, where the results suggest that our proposed spherical spatial embeddings and multimodal training objectives fairly contribute to a better semantic understanding of the panoramic surroundings on the dataset.
  arXiv  Detail & Related papers  (2021-10-11T09:58:05Z)
• ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos [5.831115928056554]
  This paper proposes ATSal, a novel attention based (head-eye) saliency model for 360degree videos. We compare the proposed approach to other state-of-the-art saliency models on two datasets: Salient360! and VR-EyeTracking. Experimental results on over 80 ODV videos (75K+ frames) show that the proposed method outperforms the existing state-of-the-art.
  arXiv  Detail & Related papers  (2020-11-20T19:19:48Z)
• Learning Representations from Audio-Visual Spatial Alignment [76.29670751012198]
  We introduce a novel self-supervised pretext task for learning representations from audio-visual content. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks.
  arXiv  Detail & Related papers  (2020-11-03T16:20:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.