Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

• URL: http://arxiv.org/abs/2603.19026v1
• Date: Thu, 19 Mar 2026 15:25:36 GMT
• Title: Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
• Abstract summary: This paper investigates whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E)<n>Our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs.<n>Comprehensive experiments validate that SELF1E achieves performance competitive with specialist mask decoder-based methods.
• Score: 75.67455028589505
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.

Related papers

• ARGenSeg: Image Segmentation with Autoregressive Image Generation Model [46.837184955843355]
  We propose a novel AutoRegressive Generation-based paradigm for image (ARGenSeg)<n>Our method surpasses prior state-of-the-art approaches on multiple segmentation datasets with a remarkable boost in inference speed.
  arXiv  Detail & Related papers  (2025-10-23T17:58:26Z)
• Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation [9.120581644616488]
  We propose a novel semantic visual projector that uses semantic superpixels to identify "visual words" in an image.<n>By compressing and projecting semantic superpixels as visual tokens, our approach adaptively shortens the token sequence according to scene.<n> Experiments show that our method cuts visual tokens by 93% without compromising performance.
  arXiv  Detail & Related papers  (2025-09-17T04:04:08Z)
• Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder [18.236863512276187]
  We propose a novel framework that exploits the inherent visual detail features encoded in the MLLM vision encoder without introducing an extra visual encoder.<n>Besides, we propose a detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully integrates the detail-related visual feature with the semantic-related feature output by the large language model (LLM) of MLLM.<n>Our method generally surpasses both SAM-based and SAM-free competitors, striking a better balance between performance and cost.
  arXiv  Detail & Related papers  (2025-08-06T06:06:52Z)
• HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model [6.641903410779405]
  We propose the Hierarchical Mask Tokenizer (HiMTok), which represents segmentation masks with up to 32 tokens.<n>HiMTok allows for compact and coarse-to-fine mask representations, aligning well with the next-token-prediction paradigm.<n>We develop a 3-stage training recipe for progressive learning of segmentation and visual capabilities, featuring a hierarchical mask loss for effective coarse-to-fine learning.
  arXiv  Detail & Related papers  (2025-03-17T10:29:08Z)
• SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories [52.57696897619189]
  We introduce the Human-Like Mask Modeling Task (HLMAT), a new paradigm where MLLMs mimic human annotators using interactive segmentation tools.<n>HLMAT enables MLLMs to iteratively generate text-based click points, achieving high-quality masks without architectural changes or implicit tokens.<n>HLMAT provides a protocol for assessing fine-grained pixel understanding in MLLMs and introduces a vision-centric, multi-step decision-making task.
  arXiv  Detail & Related papers  (2025-03-11T17:08:54Z)
• PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model [49.80313655590392]
  PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. It incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization.
  arXiv  Detail & Related papers  (2024-03-21T17:50:47Z)
• Generalizable Entity Grounding via Assistance of Large Language Model [77.07759442298666]
  We propose a novel approach to densely ground visual entities from a long caption. We leverage a large multimodal model to extract semantic nouns, a class-a segmentation model to generate entity-level segmentation, and a multi-modal feature fusion module to associate each semantic noun with its corresponding segmentation mask.
  arXiv  Detail & Related papers  (2024-02-04T16:06:05Z)
• PixelLM: Pixel Reasoning with Large Multimodal Model [110.500792765109]
  PixelLM is an effective and efficient LMM for pixel-level reasoning and understanding. It produces masks from the hidden embeddings of the codebook tokens, which encode detailed target-relevant information. PixelLM excels across various pixel-level image reasoning and understanding tasks, outperforming well-established methods in multiple benchmarks.
  arXiv  Detail & Related papers  (2023-12-04T03:05:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.