PanoSAMic: Panoramic Image Segmentation from SAM Feature Encoding and Dual View Fusion

• URL: http://arxiv.org/abs/2601.07447v1
• Date: Mon, 12 Jan 2026 11:39:36 GMT
• Title: PanoSAMic: Panoramic Image Segmentation from SAM Feature Encoding and Dual View Fusion
• Abstract summary: PanoSAMic integrates the pre-trained Anything (SAM) encoder to make use of its extensive training.<n>We modify the SAM encoder to multi-stage features and introduce a novel Segment fusion module.<n>Our semantic decoder uses attention and dual view fusion to overcome the distortions and edge discontinuity often associated with panoramic images.
• Score: 16.433654468467324
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Existing image foundation models are not optimized for spherical images having been trained primarily on perspective images. PanoSAMic integrates the pre-trained Segment Anything (SAM) encoder to make use of its extensive training and integrate it into a semantic segmentation model for panoramic images using multiple modalities. We modify the SAM encoder to output multi-stage features and introduce a novel spatio-modal fusion module that allows the model to select the relevant modalities and best features from each modality for different areas of the input. Furthermore, our semantic decoder uses spherical attention and dual view fusion to overcome the distortions and edge discontinuity often associated with panoramic images. PanoSAMic achieves state-of-the-art (SotA) results on Stanford2D3DS for RGB, RGB-D, and RGB-D-N modalities and on Matterport3D for RGB and RGB-D modalities. https://github.com/dfki-av/PanoSAMic

Related papers

• MV-SAM: Multi-view Promptable Segmentation using Pointmap Guidance [79.57732829495843]
  We introduce MV-SAM, a framework for multi-view segmentation that achieves 3D consistency using pointmaps.<n>MV-SAM lifts images and prompts into 3D space, eliminating the need for explicit 3D networks or annotated 3D data.
  arXiv  Detail & Related papers  (2026-01-25T15:00:37Z)
• Multi-View Foundation Models [22.65339008198238]
  Foundation models are vital tools in various Computer Vision applications.<n>We propose a way to convert a Foundation Model into a Multi-View Foundation Model.<n>We show how to augment Transformers-based foundation models with intermediate 3D-aware attention layers.
  arXiv  Detail & Related papers  (2025-12-17T18:58:03Z)
• HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection [75.406055413928]
  We propose a novel prompt-driven segment anything model (HyPSAM) for RGB-T SOD.<n> DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction.<n>Experiments on three public datasets demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-09-23T07:32:11Z)
• OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentation [38.730173012619304]
  Segment Anything Model 2 (SAM2) has emerged as a strong base model in various pinhole imaging segmentation tasks.<n>Applying it to $360circ$ domain, the significant field-of-view gap between pinhole ($70circ times 70circ$) and panoramic images ($180circ times 360circ$) poses unique challenges.<n>We propose a novel OmniSAM framework, which makes the first attempt to apply SAM2 for panoramic semantic segmentation.
  arXiv  Detail & Related papers  (2025-03-10T09:21:08Z)
• SMLNet: A SPD Manifold Learning Network for Infrared and Visible Image Fusion [60.18614468818683]
  We propose a novel SPD (symmetric positive definite) manifold learning for multi-modal image fusion.<n>Our framework exhibits superior performance compared to the current state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-16T03:09:49Z)
• SSFam: Scribble Supervised Salient Object Detection Family [13.369217449092524]
  Scribble supervised salient object detection (SSSOD) constructs segmentation ability of attractive objects from surroundings under the supervision of sparse scribble labels. For the better segmentation, depth and thermal infrared modalities serve as the supplement to RGB images in the complex scenes. Our model demonstrates the remarkable performance among combinations of different modalities and refreshes the highest level of scribble supervised methods.
  arXiv  Detail & Related papers  (2024-09-07T13:07:59Z)
• FusionSAM: Visual Multi-Modal Learning with Segment Anything [37.61598617788102]
  We introduce the Segment Anything Model (SAM) into multimodal image segmentation for the first time.<n>We propose a novel framework that combines Latent Space Token Generation (LSTG) and Fusion Mask Prompting (FMP) modules.<n>Our method significantly outperforms SAM and SAM2 in multimodal autonomous driving scenarios.
  arXiv  Detail & Related papers  (2024-08-26T02:20:55Z)
• Segment Anything with Multiple Modalities [61.74214237816402]
  We develop MM-SAM, which supports cross-modal and multi-modal processing for robust and enhanced segmentation with different sensor suites. MM-SAM features two key designs, namely, unsupervised cross-modal transfer and weakly-supervised multi-modal fusion. It addresses three main challenges: 1) adaptation toward diverse non-RGB sensors for single-modal processing, 2) synergistic processing of multi-modal data via sensor fusion, and 3) mask-free training for different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-17T03:45:40Z)
• EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion [55.367269556557645]
  EvPlug learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model. We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation.
  arXiv  Detail & Related papers  (2023-12-28T10:05:13Z)
• Single Frame Semantic Segmentation Using Multi-Modal Spherical Images [0.0]
  We propose a transformer-based cross-modal fusion architecture to bridge the gap between multi-modal fusion and omnidirectional scene perception. We employ distortion-aware modules to address extreme object deformations and panorama distortions. Our technique achieved state-of-the-art mIoU performance: 60.60% on Stanford2D3DS (RGB-HHA), 71.97% Structured3D (RGB-D-N), and 35.92% Matterport3D (RGB-D)
  arXiv  Detail & Related papers  (2023-08-18T08:06:18Z)
• Semantic-SAM: Segment and Recognize Anything at Any Granularity [83.64686655044765]
  We introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. We consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels.
  arXiv  Detail & Related papers  (2023-07-10T17:59:40Z)
• Self-Supervised Representation Learning for RGB-D Salient Object Detection [93.17479956795862]
  We use Self-Supervised Representation Learning to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and un RGB-D datasets to perform pre-training, which make the network capture rich semantic contexts. For the inherent problem of cross-modal fusion in RGB-D SOD, we propose a multi-path fusion module.
  arXiv  Detail & Related papers  (2021-01-29T09:16:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.