GraCo: Granularity-Controllable Interactive Segmentation

• URL: http://arxiv.org/abs/2405.00587v2
• Date: Thu, 16 May 2024 12:17:32 GMT
• Title: GraCo: Granularity-Controllable Interactive Segmentation
• Authors: Yian Zhao, Kehan Li, Zesen Cheng, Pengchong Qiao, Xiawu Zheng, Rongrong Ji, Chang Liu, Li Yuan, Jie Chen,
• Abstract summary: Granularity-Controllable Interactive (GraCo) is a novel approach that allows precise control of prediction granularity by introducing additional parameters to input. GraCo exploits the semantic property of the pre-trained IS model to automatically generate abundant mask-granularity pairs. Experiments on intricate scenarios at object and part levels demonstrate that our GraCo has significant advantages over previous methods.
• Score: 52.9695642626127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive Segmentation (IS) segments specific objects or parts in the image according to user input. Current IS pipelines fall into two categories: single-granularity output and multi-granularity output. The latter aims to alleviate the spatial ambiguity present in the former. However, the multi-granularity output pipeline suffers from limited interaction flexibility and produces redundant results. In this work, we introduce Granularity-Controllable Interactive Segmentation (GraCo), a novel approach that allows precise control of prediction granularity by introducing additional parameters to input. This enhances the customization of the interactive system and eliminates redundancy while resolving ambiguity. Nevertheless, the exorbitant cost of annotating multi-granularity masks and the lack of available datasets with granularity annotations make it difficult for models to acquire the necessary guidance to control output granularity. To address this problem, we design an any-granularity mask generator that exploits the semantic property of the pre-trained IS model to automatically generate abundant mask-granularity pairs without requiring additional manual annotation. Based on these pairs, we propose a granularity-controllable learning strategy that efficiently imparts the granularity controllability to the IS model. Extensive experiments on intricate scenarios at object and part levels demonstrate that our GraCo has significant advantages over previous methods. This highlights the potential of GraCo to be a flexible annotation tool, capable of adapting to diverse segmentation scenarios. The project page: https://zhao-yian.github.io/GraCo.

Related papers

• DFIMat: Decoupled Flexible Interactive Matting in Multi-Person Scenarios [32.77825044757212]
  We propose DFIMat, a decoupled framework that enables flexible interactive matting. Specifically, we first decouple the task into 2 sub-ones: localizing target instances by understanding scene semantics and the flexible user inputs, and conducting refinement for instance-level matting. We observe a clear performance gain from decoupling, as it makes sub-tasks easier to learn, and the flexible multi-type input further enhances both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2024-10-13T10:02:58Z)
• Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model [19.861556031795725]
  We introduce a Multi-Granularity Large Multimodal Model (MGLMM) MGLMM is capable of seamlessly adjusting the granularity of Captioning (SegCap) following user instructions. It excels at tackling more than eight downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-09-20T11:13:31Z)
• N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields [112.02885337510716]
  Nested Neural Feature Fields (N2F2) is a novel approach that employs hierarchical supervision to learn a single feature field. We leverage a 2D class-agnostic segmentation model to provide semantically meaningful pixel groupings at arbitrary scales in the image space. Our approach outperforms the state-of-the-art feature field distillation methods on tasks such as open-vocabulary 3D segmentation and localization.
  arXiv  Detail & Related papers  (2024-03-16T18:50:44Z)
• M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition [80.21796574234287]
  M$3$Net is a matching-based framework for few-shot fine-grained (FS-FG) action recognition. It incorporates textitmulti-view encoding, textitmulti-view matching, and textitmulti-view fusion to facilitate embedding encoding, similarity matching, and decision making. Explainable visualizations and experimental results demonstrate the superiority of M$3$Net in capturing fine-grained action details.
  arXiv  Detail & Related papers  (2023-08-06T09:15:14Z)
• Semantic-SAM: Segment and Recognize Anything at Any Granularity [83.64686655044765]
  We introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. We consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels.
  arXiv  Detail & Related papers  (2023-07-10T17:59:40Z)
• Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation [49.6153714376745]
  We address the problem of referring image segmentation that aims to generate a mask for the object specified by a natural language expression. We propose Multi-Modal Mutual Attention ($mathrmM3Att$) and Multi-Modal Mutual Decoder ($mathrmM3Dec$) that better fuse information from the two input modalities.
  arXiv  Detail & Related papers  (2023-05-24T16:26:05Z)
• Multi-granularity Interaction Simulation for Unsupervised Interactive Segmentation [38.08152990071453]
  We introduce a Multi-granularity Interaction Simulation (MIS) approach to open up a promising direction for unsupervised interactive segmentation. Our MIS significantly outperforms non-deep learning unsupervised methods and is even comparable with some previous deep-supervised methods without any annotation.
  arXiv  Detail & Related papers  (2023-03-23T16:19:43Z)
• DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation [21.717520350930705]
  Transformer-based models have been widely demonstrated to be successful in computer vision tasks. However, they are often dominated by features of large patterns leading to the loss of local details. We propose a Dual-Aggregation Transformer Network called DuAT, which is characterized by two innovative designs. Our proposed model outperforms state-of-the-art methods in the segmentation of skin lesion images, and polyps in colonoscopy images.
  arXiv  Detail & Related papers  (2022-12-21T07:54:02Z)
• BoundarySqueeze: Image Segmentation as Boundary Squeezing [104.43159799559464]
  We propose a novel method for fine-grained high-quality image segmentation of both objects and scenes. Inspired by dilation and erosion from morphological image processing techniques, we treat the pixel level segmentation problems as squeezing object boundary. Our method yields large gains on COCO, Cityscapes, for both instance and semantic segmentation and outperforms previous state-of-the-art PointRend in both accuracy and speed under the same setting.
  arXiv  Detail & Related papers  (2021-05-25T04:58:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.