Panoramic Affordance Prediction

• URL: http://arxiv.org/abs/2603.15558v1
• Date: Mon, 16 Mar 2026 17:21:49 GMT
• Title: Panoramic Affordance Prediction
• Abstract summary: Affordance prediction serves as a critical bridge between perception and action in embodied AI.<n>We present the first exploration into Panoramic Affordance Prediction, utilizing 360-degree imagery.<n>We propose PAP, a training-free, coarse-to-fine pipeline inspired by the human foveal visual system.
• Score: 94.50813972018504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Affordance prediction serves as a critical bridge between perception and action in embodied AI. However, existing research is confined to pinhole camera models, which suffer from narrow Fields of View (FoV) and fragmented observations, often missing critical holistic environmental context. In this paper, we present the first exploration into Panoramic Affordance Prediction, utilizing 360-degree imagery to capture global spatial relationships and holistic scene understanding. To facilitate this novel task, we first introduce PAP-12K, a large-scale benchmark dataset containing over 1,000 ultra-high-resolution (12k, 11904 x 5952) panoramic images with over 12k carefully annotated QA pairs and affordance masks. Furthermore, we propose PAP, a training-free, coarse-to-fine pipeline inspired by the human foveal visual system to tackle the ultra-high resolution and severe distortion inherent in panoramic images. PAP employs recursive visual routing via grid prompting to progressively locate targets, applies an adaptive gaze mechanism to rectify local geometric distortions, and utilizes a cascaded grounding pipeline to extract precise instance-level masks. Experimental results on PAP-12K reveal that existing affordance prediction methods designed for standard perspective images suffer severe performance degradation and fail due to the unique challenges of panoramic vision. In contrast, PAP framework effectively overcomes these obstacles, significantly outperforming state-of-the-art baselines and highlighting the immense potential of panoramic perception for robust embodied intelligence.

Related papers

• EAGLE-360: Embodied Active Global-to-Local Exploration in 360$^\circ$ [25.40870252692496]
  We propose a novel Embodied Active Global-to-Local Exploration framework for 360$circ$ visual search.<n>We construct a large-scale dataset, comprising 14,000+ 4K panoramas and 70,000+ rounds of high-quality VQA dialogues.<n>Experiments demonstrate that the framework achieves nearly an 8-fold increase in accuracy over the base model.
  arXiv  Detail & Related papers  (2026-07-02T17:47:27Z)
• OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning [94.47284498900895]
  We introduce OmniCoT, a panoramic spatial reasoning suite designed to enable MLLMs to use global evidence and perform multi-step inference across viewpoints.<n>For training, OmniCoT-T is purpose-built with structured stepwise Chain-of-Thought annotations that explicitly link intermediate reasoning steps to panoramic evidence.
  arXiv  Detail & Related papers  (2026-06-29T14:38:20Z)
• Gaussian-Constrained LeJEPA Representations for Unsupervised Scene Discovery and Pose Consistency [0.0]
  This paper investigates the application of Gaussian-constrained representations inspired by LeJEPA (Joint Embedding Predictive Architecture) to address these challenges.<n>We present three progressively refined pipelines, culminating in a LeJEPA-inspired approach that enforces isotropic Gaussian constraints on learned image embeddings.
  arXiv  Detail & Related papers  (2026-01-31T19:52:54Z)
• Generalizable Collaborative Search-and-Capture in Cluttered Environments via Path-Guided MAPPO and Directional Frontier Allocation [7.6658082440595186]
  Collaborative pursuit-evasion in cluttered environments presents significant challenges due to sparse rewards and constrained Fields of View.<n>We propose PGF-MAPPO, a hierarchical framework bridging topological planning with reactive control.<n>Experiments demonstrate that PGF-MAPPO achieves superior capture efficiency against faster evaders.
  arXiv  Detail & Related papers  (2025-12-10T08:09:12Z)
• Panoramic Out-of-Distribution Segmentation for Autonomous Driving [25.32165408678185]
  We introduce Panoramic Out-of-distribution (PanOoS) to achieve comprehensive and safe scene understanding.<n> POS adapts to the characteristics of panoramic images through text-guided prompt distribution learning.<n> POS achieves leading closed-set segmentation capabilities and advances the development of panoramic understanding.
  arXiv  Detail & Related papers  (2025-05-06T13:51:26Z)
• MCPDepth: Omnidirectional Depth Estimation via Stereo Matching from Multi-Cylindrical Panoramas [49.891712558113845]
  We introduce Multi-Cylindrical Panoramic Depth Estimation (MCPDepth)<n>MCPDepth is a two-stage framework designed to enhance omnidirectional depth estimation.<n>Our method improves the mean absolute error (MAE) by 18.8% on the outdoor dataset Deep360 and by 19.9% on the real dataset 3D60.
  arXiv  Detail & Related papers  (2024-08-03T03:35:37Z)
• Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning [93.6645991946674]
  We introduce panoramic panoptic segmentation, as the most holistic scene understanding. A complete surrounding understanding provides a maximum of information to a mobile agent. We propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain.
  arXiv  Detail & Related papers  (2022-06-21T20:07:15Z)
• Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning [97.37544023666833]
  We introduce panoramic panoptic segmentation as the most holistic scene understanding. A complete surrounding understanding provides a maximum of information to the agent. We propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain.
  arXiv  Detail & Related papers  (2021-03-01T09:37:27Z)
• Revisiting Pixel-Wise Supervision for Face Anti-Spoofing [75.89648108213773]
  Face anti-spoofing (FAS) plays a vital role in securing face recognition systems from presentation attacks (PAs) Traditionally, deep models supervised by binary loss are weak in describing intrinsic and discriminative spoofing patterns. Recent, pixel-wise supervision has been proposed for the FAS task, intending to provide more fine-grained pixel/patch-level cues.
  arXiv  Detail & Related papers  (2020-11-24T11:25:58Z)
• Towards Image-based Automatic Meter Reading in Unconstrained Scenarios: A Robust and Efficient Approach [60.63996472100845]
  We present an end-to-end approach for Automatic Meter Reading (AMR) focusing on unconstrained scenarios. Our main contribution is the insertion of a new stage in the AMR pipeline, called corner detection and counter classification. We show that our AMR system achieves impressive recognition rates (i.e., > 99%) when rejecting readings made with lower confidence values.
  arXiv  Detail & Related papers  (2020-09-21T21:21:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.