Sub-Semantic Image Segmentation

• URL: http://arxiv.org/abs/2606.14754v1
• Date: Sun, 07 Jun 2026 18:39:55 GMT
• Title: Sub-Semantic Image Segmentation
• Abstract summary: We propose a new category of sub-semantic image segmentation that blurs the line between the two.<n>In sub-semantic image segmentation, language is not used to name whole objects. Instead, it is used to partition an image into stable appearance patterns that can be described by language.
• Score: 13.414225469091724
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Images can be segmented based on visual cues (i.e., texture segmentation) or into objects (i.e., semantic segmentation). We propose a new category of sub-semantic image segmentation that blurs the line between the two. In sub-semantic image segmentation, language is not used to name whole objects. Instead, it is used to partition an image into stable appearance patterns that can be described by language. To do that, we couple a general-purpose vision-language model to SAM 3, a promptable segmentation backbone whose native text pathway can ground rich descriptions into masks. Simple coupling fails for a number of reasons that we identify in the paper, and we overcome them by introducing DETECTURE that resolves three concrete failure modes -- language leakage between texture regions, prompt competition inside the segmentation backbone, and semantic distortion at the language-to-mask interface. Since there is no dataset of sub-semantic image segmentation, we introduce one, termed TextureADE. The new dataset is derived from the ADE20K dataset using a system we designed. We compare DETECTURE to a number of baselines and find that it achieves the strongest performance on several datasets using different metrics. Code is available at https://github.com/Scientific-Computing-Lab/TextureDetecture.

Related papers

• Text4Seg++: Advancing Image Segmentation via Generative Language Modeling [52.07442359419673]
  We propose a novel text-as-mask paradigm that casts image segmentation as a text generation problem.<n>Key innovation is semantic descriptors, a new textual representation of segmentation masks.<n>Experiments on natural and remote sensing datasets show that Text4Seg++ consistently outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-09-08T04:07:14Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification [59.779532652634295]
  We propose an embarrassingly simple approach to better align image and text features with no need of additional data formats other than image-text pairs. We parse objects and attributes from the description, which are highly likely to exist in the image. Experiments substantiate the average 5.2% improvement of our framework over existing alternatives.
  arXiv  Detail & Related papers  (2023-12-21T18:59:06Z)
• Text and Click inputs for unambiguous open vocabulary instance segmentation [21.03169732771627]
  We propose a new segmentation process, Text + Click, where a model takes as input an image, a text phrase describing a class to segment, and a single foreground click specifying the instance to segment. We demonstrate that the combination of a single user-specified foreground click and a text prompt allows a model to better disambiguate overlapping or co-occurring semantic categories.
  arXiv  Detail & Related papers  (2023-11-24T19:37:57Z)
• Beyond One-to-One: Rethinking the Referring Image Segmentation [117.53010476628029]
  Referring image segmentation aims to segment the target object referred by a natural language expression. We propose a Dual Multi-Modal Interaction (DMMI) Network, which contains two decoder branches. In the text-to-image decoder, text embedding is utilized to query the visual feature and localize the corresponding target. Meanwhile, the image-to-text decoder is implemented to reconstruct the erased entity-phrase conditioned on the visual feature.
  arXiv  Detail & Related papers  (2023-08-26T11:39:22Z)
• Hierarchical Open-vocabulary Universal Image Segmentation [48.008887320870244]
  Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions. We propose a decoupled text-image fusion mechanism and representation learning modules for both "things" and "stuff" Our resulting model, named HIPIE tackles, HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a unified framework.
  arXiv  Detail & Related papers  (2023-07-03T06:02:15Z)
• Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs [10.484851004093919]
  We tackle open-world semantic segmentation, which aims at learning to segment arbitrary visual concepts in images. Existing open-world segmentation methods have shown impressive advances by employing contrastive learning (CL) to learn diverse visual concepts. We propose a novel Text-grounded Contrastive Learning framework that enables a model to directly learn region-text alignment.
  arXiv  Detail & Related papers  (2022-12-01T18:59:03Z)
• StructToken : Rethinking Semantic Segmentation with Structural Prior [14.056789487558731]
  We present a new paradigm for semantic segmentation, named structure-aware extraction. It generates the segmentation results via the interactions between a set of learned structure tokens and the image feature, which aims to progressively extract the structural information of each category from the feature. Our StructToken outperforms the state-of-the-art on three widely-used benchmarks, including ADE20K, Cityscapes, and COCO-Stuff-10K.
  arXiv  Detail & Related papers  (2022-03-23T17:58:31Z)
• Prompt-Based Multi-Modal Image Segmentation [81.58378196535003]
  We propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder.
  arXiv  Detail & Related papers  (2021-12-18T21:27:19Z)
• Locate then Segment: A Strong Pipeline for Referring Image Segmentation [73.19139431806853]
  Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent interaction mechanism to fuse the visual-linguistic features to directly generate the final segmentation mask. We present a "Then-Then-Segment" scheme to tackle these problems. Our framework is simple but surprisingly effective.
  arXiv  Detail & Related papers  (2021-03-30T12:25:27Z)
• Exploring Cross-Image Pixel Contrast for Semantic Segmentation [130.22216825377618]
  We propose a pixel-wise contrastive framework for semantic segmentation in the fully supervised setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. Our method can be effortlessly incorporated into existing segmentation frameworks without extra overhead during testing.
  arXiv  Detail & Related papers  (2021-01-28T11:35:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.