WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation

• URL: http://arxiv.org/abs/2505.18686v2
• Date: Thu, 29 May 2025 00:58:26 GMT
• Title: WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation
• Authors: Yang Liu, Silin Cheng, Xinwei He, Sebastien Ourselin, Lei Tan, Gen Luo,
• Abstract summary: We propose WeakMCN, a novel multi-task collaborative network that effectively combines WREC and WRES with a dual-branch architecture.<n>In WeakMCN, we propose two innovative designs to facilitate multi-task collaboration, namely Dynamic Visual Feature Enhancement(DVFE) and Collaborative Consistency Module( CCM)
• Score: 11.906318282459942
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Weakly supervised referring expression comprehension(WREC) and segmentation(WRES) aim to learn object grounding based on a given expression using weak supervision signals like image-text pairs. While these tasks have traditionally been modeled separately, we argue that they can benefit from joint learning in a multi-task framework. To this end, we propose WeakMCN, a novel multi-task collaborative network that effectively combines WREC and WRES with a dual-branch architecture. Specifically, the WREC branch is formulated as anchor-based contrastive learning, which also acts as a teacher to supervise the WRES branch. In WeakMCN, we propose two innovative designs to facilitate multi-task collaboration, namely Dynamic Visual Feature Enhancement(DVFE) and Collaborative Consistency Module(CCM). DVFE dynamically combines various pre-trained visual knowledge to meet different task requirements, while CCM promotes cross-task consistency from the perspective of optimization. Extensive experimental results on three popular REC and RES benchmarks, i.e., RefCOCO, RefCOCO+, and RefCOCOg, consistently demonstrate performance gains of WeakMCN over state-of-the-art single-task alternatives, e.g., up to 3.91% and 13.11% on RefCOCO for WREC and WRES tasks, respectively. Furthermore, experiments also validate the strong generalization ability of WeakMCN in both semi-supervised REC and RES settings against existing methods, e.g., +8.94% for semi-REC and +7.71% for semi-RES on 1% RefCOCO. The code is publicly available at https://github.com/MRUIL/WeakMCN.

Related papers

• Learning Multi-Branch Cooperation for Enhanced Click-Through Rate Prediction at Taobao [51.84189885218365]
  We introduce a novel Multi-Branch Cooperation Network (MBCnet)<n>MBCnet enables multiple branch networks to collaborate with each other for better complex feature interaction modeling.<n> Experiments on large-scale industrial datasets and online A/B test at app demonstrate MBCnet's superior performance.
  arXiv  Detail & Related papers  (2024-11-20T06:10:06Z)
• Multi-branch Collaborative Learning Network for 3D Visual Grounding [66.67647903507927]
  3D referring expression comprehension (3DREC) and segmentation (3DRES) have overlapping objectives, indicating their potential for collaboration. We argue that employing separate branches for 3DREC and 3DRES tasks enhances the model's capacity to learn specific information for each task.
  arXiv  Detail & Related papers  (2024-07-07T13:27:14Z)
• Continual Referring Expression Comprehension via Dual Modular Memorization [133.46886428655426]
  Referring Expression (REC) aims to localize an image region of a given object described by a natural-language expression. Existing REC algorithms make a strong assumption that training data feeding into a model are given upfront, which degrades its practicality for real-world scenarios. In this paper, we propose Continual Referring Expression (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks. In order to continuously improve the model on sequential tasks without forgetting prior learned knowledge and without repeatedly re-training from a scratch, we propose an effective baseline method named Dual Modular Memorization
  arXiv  Detail & Related papers  (2023-11-25T02:58:51Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generation [21.27400500728834]
  Reference Expression (RES) and Reference Expression Generation (REG) are mutually inverse tasks that can be naturally jointly trained. We propose a unified mutual supervision framework that enables two tasks to improve each other.
  arXiv  Detail & Related papers  (2022-11-15T06:08:39Z)
• A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension [81.57558029858954]
  We build a one-stage referring expression comprehension network called SimREC. We conduct over 100 experimental trials on three benchmark datasets of REC. With much less training overhead and parameters, SimREC can still achieve better performance than a set of large-scale pre-trained models.
  arXiv  Detail & Related papers  (2022-04-17T03:04:03Z)
• 3D Human Action Representation Learning via Cross-View Consistency Pursuit [52.19199260960558]
  We propose a Cross-view Contrastive Learning framework for unsupervised 3D skeleton-based action Representation (CrosSCLR) CrosSCLR consists of both single-view contrastive learning (SkeletonCLR) and cross-view consistent knowledge mining (CVC-KM) modules, integrated in a collaborative learning manner.
  arXiv  Detail & Related papers  (2021-04-29T16:29:41Z)
• Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation [135.67558811281984]
  We propose a novel Multi-task Collaborative Network (MCN) to achieve a joint learning offerring expression comprehension (REC) and segmentation (RES) In MCN, RES can help REC to achieve better language-vision alignment, while REC can help RES to better locate the referent. We address a key challenge in this multi-task setup, i.e., the prediction conflict, with two innovative designs namely, Consistency Energy Maximization (CEM) and Adaptive Soft Non-Located Suppression (ASNLS)
  arXiv  Detail & Related papers  (2020-03-19T14:25:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.