MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors

• URL: http://arxiv.org/abs/2602.05330v1
• Date: Thu, 05 Feb 2026 05:51:28 GMT
• Title: MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors
• Authors: Jingdong Zhang, Xiaohang Zhan, Lingzhi Zhang, Yizhou Wang, Zhengming Yu, Jionghao Wang, Wenping Wang, Xin Li,
• Abstract summary: MTPano is a robust panoramic foundation model established by a label-free training pipeline.<n>We project panoramic images into perspective patches to generate accurate, domain-gap-free pseudo-labels.<n>To tackle the interference between task types, we categorize tasks into rotation-invariant and rotation-variant groups.
• Score: 42.124623200906626
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Comprehensive panoramic scene understanding is critical for immersive applications, yet it remains challenging due to the scarcity of high-resolution, multi-task annotations. While perspective foundation models have achieved success through data scaling, directly adapting them to the panoramic domain often fails due to severe geometric distortions and coordinate system discrepancies. Furthermore, the underlying relations between diverse dense prediction tasks in spherical spaces are underexplored. To address these challenges, we propose MTPano, a robust multi-task panoramic foundation model established by a label-free training pipeline. First, to circumvent data scarcity, we leverage powerful perspective dense priors. We project panoramic images into perspective patches to generate accurate, domain-gap-free pseudo-labels using off-the-shelf foundation models, which are then re-projected to serve as patch-wise supervision. Second, to tackle the interference between task types, we categorize tasks into rotation-invariant (e.g., depth, segmentation) and rotation-variant (e.g., surface normals) groups. We introduce the Panoramic Dual BridgeNet, which disentangles these feature streams via geometry-aware modulation layers that inject absolute position and ray direction priors. To handle the distortion from equirectangular projections (ERP), we incorporate ERP token mixers followed by a dual-branch BridgeNet for interactions with gradient truncation, facilitating beneficial cross-task information sharing while blocking conflicting gradients from incompatible task attributes. Additionally, we introduce auxiliary tasks (image gradient, point map, etc.) to fertilize the cross-task learning process. Extensive experiments demonstrate that MTPano achieves state-of-the-art performance on multiple benchmarks and delivers competitive results against task-specific panoramic specialist foundation models.

Related papers

• Saccadic Vision for Fine-Grained Visual Classification [10.681604440788854]
  Fine-grained visual classification (FGVC) requires distinguishing between visually similar categories through subtle, localized features.<n>Existing part-based methods rely on complex localization networks that learn mappings from pixel to sample space.<n>We propose a two-stage process that first extracts peripheral features and generates a sample map.<n>We employ contextualized selective attention to weigh the impact of each fixation patch before fusing peripheral and focus representations.
  arXiv  Detail & Related papers  (2025-09-19T07:03:37Z)
• PanMatch: Unleashing the Potential of Large Vision Models for Unified Matching Models [80.65273820998875]
  We present PanMatch, a versatile foundation model for robust correspondence matching.<n>Our key insight is that any two-frame correspondence matching task can be addressed within a 2D displacement estimation framework.<n>PanMatch achieves multi-task integration by endowing displacement estimation algorithms with unprecedented generalization capabilities.
  arXiv  Detail & Related papers  (2025-07-11T08:18:52Z)
• Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgetting [49.87694319431288]
  Generalist segmentation models are increasingly favored for diverse tasks involving various objects from different image sources. We propose a Comprehensive Generative (CGR) framework that restores appearance and semantic knowledge by synthesizing image-mask pairs. Experiments on incremental tasks (cardiac, fundus and prostate segmentation) show its clear advantage for alleviating concurrent appearance and semantic forgetting.
  arXiv  Detail & Related papers  (2024-06-28T10:05:58Z)
• DeTra: A Unified Model for Object Detection and Trajectory Forecasting [68.85128937305697]
  Our approach formulates the union of the two tasks as a trajectory refinement problem. To tackle this unified task, we design a refinement transformer that infers the presence, pose, and multi-modal future behaviors of objects. In our experiments, we observe that ourmodel outperforms the state-of-the-art on Argoverse 2 Sensor and Open dataset.
  arXiv  Detail & Related papers  (2024-06-06T18:12:04Z)
• Task Indicating Transformer for Task-conditional Dense Predictions [16.92067246179703]
  We introduce a novel task-conditional framework called Task Indicating Transformer (TIT) to tackle this challenge. Our approach designs a Mix Task Adapter module within the transformer block, which incorporates a Task Indicating Matrix through matrix decomposition. We also propose a Task Gate Decoder module that harnesses a Task Indicating Vector and gating mechanism to facilitate adaptive multi-scale feature refinement.
  arXiv  Detail & Related papers  (2024-03-01T07:06:57Z)
• 360 Layout Estimation via Orthogonal Planes Disentanglement and Multi-view Geometric Consistency Perception [56.84921040837699]
  Existing panoramic layout estimation solutions tend to recover room boundaries from a vertically compressed sequence, yielding imprecise results. We propose an orthogonal plane disentanglement network (termed DOPNet) to distinguish ambiguous semantics. We also present an unsupervised adaptation technique tailored for horizon-depth and ratio representations. Our solution outperforms other SoTA models on both monocular layout estimation and multi-view layout estimation tasks.
  arXiv  Detail & Related papers  (2023-12-26T12:16:03Z)
• ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer [71.82644727907146]
  We propose a novel $underlineComP$lementary $underlinetr$ansformer, $textbfComPtr$, for diverse bi-source dense prediction tasks.<n>ComPtr treats different inputs equally and builds an efficient dense interaction model in the form of sequence-to-sequence on top of the transformer.
  arXiv  Detail & Related papers  (2023-07-23T15:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.