Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation

• URL: http://arxiv.org/abs/2405.15169v1
• Date: Fri, 24 May 2024 03:07:38 GMT
• Title: Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation
• Authors: Weize Li, Zhicheng Zhao, Haochen Bai, Fei Su,
• Abstract summary: Generalized Referring Expression introduces new challenges by allowing expressions to describe multiple objects or lack specific object references. Existing RES methods, usually rely on sophisticated encoder-decoder and feature fusion modules. We propose a novel Model with Adaptive Binding Prototypes (MABP) that adaptively binds queries to object features in the corresponding region.
• Score: 18.806738617249426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Expression Segmentation (RES) has attracted rising attention, aiming to identify and segment objects based on natural language expressions. While substantial progress has been made in RES, the emergence of Generalized Referring Expression Segmentation (GRES) introduces new challenges by allowing expressions to describe multiple objects or lack specific object references. Existing RES methods, usually rely on sophisticated encoder-decoder and feature fusion modules, and are difficult to generate class prototypes that match each instance individually when confronted with the complex referent and binary labels of GRES. In this paper, reevaluating the differences between RES and GRES, we propose a novel Model with Adaptive Binding Prototypes (MABP) that adaptively binds queries to object features in the corresponding region. It enables different query vectors to match instances of different categories or different parts of the same instance, significantly expanding the decoder's flexibility, dispersing global pressure across all queries, and easing the demands on the encoder. Experimental results demonstrate that MABP significantly outperforms state-of-the-art methods in all three splits on gRefCOCO dataset. Meanwhile, MABP also surpasses state-of-the-art methods on RefCOCO+ and G-Ref datasets, and achieves very competitive results on RefCOCO. Code is available at https://github.com/buptLwz/MABP

Related papers

• CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentation [37.96005100341482]
  Generalized Referring Expression (GRES) amplifies the formulation of classic RES by involving complex multiple/non-target scenarios. Recent approaches address GRES by directly extending the well-adopted RES frameworks with object-existence identification. We propose a textbfCounting-Aware textbfHierarchical textbfDecoding framework (CoHD) for GRES.
  arXiv  Detail & Related papers  (2024-05-24T15:53:59Z)
• GSVA: Generalized Segmentation via Multimodal Large Language Models [72.57095903188922]
  Generalized Referring Expression (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. Current solutions to GRES remain unsatisfactory since segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt. We propose Generalized Vision Assistant (GSVA) to address this gap.
  arXiv  Detail & Related papers  (2023-12-15T02:54:31Z)
• Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation [59.37587762543934]
  This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS) Existing methods suffer from a granularity inconsistency regarding the usage of group tokens. We propose the prototypical guidance network (PGSeg) that incorporates multi-modal regularization.
  arXiv  Detail & Related papers  (2023-10-29T13:18:00Z)
• Referring Expression Comprehension Using Language Adaptive Inference [15.09309604460633]
  This paper explores the adaptation between expressions and REC models for dynamic inference. We propose a framework named Language Adaptive Subnets (LADS), which can extract language-adaptives from the REC model conditioned on the referring expressions. Experiments on RefCOCO, RefCO+, RefCOCOg, and Referit show that the proposed method achieves faster inference speed and higher accuracy against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-06-06T07:58:59Z)
• GRES: Generalized Referring Expression Segmentation [32.12725360752345]
  We introduce a new benchmark called Generalized Referring Expression (GRES) GRES allows expressions to refer to an arbitrary number of target objects. We construct the first large-scale GRES dataset called gRefCOCO that contains multi-target, no-target, and single-target expressions.
  arXiv  Detail & Related papers  (2023-06-01T17:57:32Z)
• Reflection Invariance Learning for Few-shot Semantic Segmentation [53.20466630330429]
  Few-shot semantic segmentation (FSS) aims to segment objects of unseen classes in query images with only a few annotated support images. This paper proposes a fresh few-shot segmentation framework to mine the reflection invariance in a multi-view matching manner. Experiments on both PASCAL-$5textiti$ and COCO-$20textiti$ datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-01T15:14:58Z)
• Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation [102.25240608024063]
  Referring image segments an image from a language expression. We develop an algorithm that shifts from being localization-centric to segmentation-language. Compared to its counterparts, our method is more versatile yet effective.
  arXiv  Detail & Related papers  (2023-03-11T08:42:40Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• I^3Net: Implicit Instance-Invariant Network for Adapting One-Stage Object Detectors [64.93963042395976]
  Implicit Instance-Invariant Network (I3Net) is tailored for adapting one-stage detectors. I3Net implicitly learns instance-invariant features via exploiting the natural characteristics of deep features in different layers. Experiments reveal that I3Net exceeds the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2021-03-25T11:14:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.