P2P-Former: Bridging the Supervision Gap Between Point and Proposal in Temporal Action Localization
Abstract Overview
This paper proposes P2P-Former, a Transformer-based framework for point-supervised temporal action localization that aims to bridge the gap between sparse point annotations and proposal-level learning. The method addresses two stated challenges: generating reliable pseudo proposals from point supervision and injecting point-level cues into a proposal-based detector. To do this, it combines point-aware proposal generation with one-to-one matching, a point-guided semantic enhancement module built from clustered point features, and a point-to-proposal constraint loss. Experiments on THUMOS14 and ActivityNet-1.3 show consistent improvements over prior point-supervised methods, with ablations supporting the contribution of each component.
Novelty
The distinctive aspect of the work is its explicit reformulation of point-supervised temporal action localization as a point-to-proposal learning problem rather than relying mainly on snippet-level classification and thresholding. Its main novelties are the point-aware reliable proposal generation with Hungarian matching, prototype-based semantic enhancement from point annotations, and a loss that constrains predicted proposals to include and align with annotated points.
Results
On THUMOS14, the method reports 65.8 average mAP over IoU 0.1:0.1:0.7, exceeding prior point-supervised results in the table, including HR-Pro at 60.3 and QROT at 52.1 for the 0.3:0.1:0.7 average comparison shown. On ActivityNet-1.3, it reports 31.1 average mAP over IoU 0.5:0.05:0.95, above the listed point-supervised baselines such as QROT at 27.7. Ablation results show that combining all three proposed components improves the THUMOS14 average mAP over IoU 0.1:0.1:0.7 from 62.6 to 65.8.
Key Points
- P2P-Former generates pseudo ground-truth proposals by re-scoring candidate proposals with point-aware importance and selecting one proposal per point via Hungarian matching.
- The framework enriches snippet features using category prototypes clustered from point-annotated features and injects them through Transformer-based interactions.
- The full model achieves the best reported point-supervised performance in the provided tables on both THUMOS14 and ActivityNet-1.3, and ablations indicate that PPG, PSE, and PPCL are complementary.