Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

• URL: http://arxiv.org/abs/2602.18996v1
• Date: Sun, 22 Feb 2026 00:53:03 GMT
• Title: Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction
• Authors: Shannan Yan, Leqi Zheng, Keyu Lv, Jingchen Ni, Hongyang Wei, Jiajun Zhang, Guangting Wang, Jing Lyu, Chun Yuan, Fengyun Rao,
• Abstract summary: We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios.<n>We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video.<n> Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance.
• Score: 47.01100029571904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios. We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video. To encourage robust, view-invariant representations, we introduce a cycle-consistency training objective: the predicted mask in the target view is projected back to the source view to reconstruct the original query mask. This bidirectional constraint provides a strong self-supervisory signal without requiring ground-truth annotations and enables test-time training (TTT) at inference. Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance. The code is available at https://github.com/shannany0606/CCMP.

Related papers

• Beyond Single Images: Retrieval Self-Augmented Unsupervised Camouflaged Object Detection [18.382178646073474]
  We propose RISE, a paradigm that exploits the entire training dataset to generate pseudo-labels for single images.<n>It is important to recognize that using only training images without annotations exerts a pronounced challenge in crafting high-quality prototype libraries.<n>In the KNN retrieval stage, to alleviate the effect of artifacts in feature maps, we propose Multi-View KNN Retrieval.
  arXiv  Detail & Related papers  (2025-10-21T09:12:26Z)
• Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation [52.337472185022136]
  We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description.<n>We propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation.<n>We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art consistency.
  arXiv  Detail & Related papers  (2025-01-06T14:49:26Z)
• ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives [109.11714588441511]
  The Ego-Exo object correspondence task aims to understand object relations across ego-exo perspectives through segmentation.<n> PSALM, a recently proposed segmentation method, stands out as a notable exception with its demonstrated zero-shot ability on this task.<n>We propose ObjectRelator, a novel approach featuring two key modules: Multimodal Condition Fusion and SSL-based Cross-View Object Alignment.
  arXiv  Detail & Related papers  (2024-11-28T12:01:03Z)
• Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction [6.798515070856465]
  We present a novel self-supervised approach for representation learning, particularly for the task of Visual Relationship Detection (VRD) Motivated by the effectiveness of Masked Image Modeling (MIM), we propose Masked Bounding Box Reconstruction (MBBR)
  arXiv  Detail & Related papers  (2023-11-08T16:59:26Z)
• Rethinking Amodal Video Segmentation from Learning Supervised Signals with Object-centric Representation [47.39455910191075]
  Video amodal segmentation is a challenging task in computer vision. Recent studies have achieved promising performance by using motion flow to integrate information across frames under a self-supervised setting. This paper presents a rethinking to previous works. We particularly leverage the supervised signals with object-centric representation.
  arXiv  Detail & Related papers  (2023-09-23T04:12:02Z)
• Learning Referring Video Object Segmentation from Weak Annotation [78.45828085350936]
  Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. We propose a new annotation scheme that reduces the annotation effort by 8 times, while providing sufficient supervision for RVOS. Our scheme only requires a mask for the frame where the object first appears and bounding boxes for the rest of the frames.
  arXiv  Detail & Related papers  (2023-08-04T06:50:52Z)
• MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining [138.86293836634323]
  MaskCLIP incorporates a newly proposed masked self-distillation into contrastive language-image pretraining. MaskCLIP achieves superior results in linear probing, finetuning, and zero-shot performance with the guidance of the language encoder.
  arXiv  Detail & Related papers  (2022-08-25T17:59:58Z)
• Object-wise Masked Autoencoders for Fast Pre-training [13.757095663704858]
  We show that current masked image encoding models learn the underlying relationship between all objects in the whole scene, instead of a single object representation. We introduce a novel object selection and division strategy to drop non-object patches for learning object-wise representations by selective reconstruction with interested region masks. Experiments on four commonly-used datasets demonstrate the effectiveness of our model in reducing the compute cost by 72% while achieving competitive performance.
  arXiv  Detail & Related papers  (2022-05-28T05:13:45Z)
• Self-Supervised Visual Representations Learning by Contrastive Mask Prediction [129.25459808288025]
  We propose a novel contrastive mask prediction (CMP) task for visual representation learning. MaskCo contrasts region-level features instead of view-level features, which makes it possible to identify the positive sample without any assumptions. We evaluate MaskCo on training datasets beyond ImageNet and compare its performance with MoCo V2.
  arXiv  Detail & Related papers  (2021-08-18T02:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.