Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

• URL: http://arxiv.org/abs/2608.02791v1
• Date: Mon, 03 Aug 2026 18:42:11 GMT
• Title: Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
• Abstract summary: STAMP combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference.<n>It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass.<n>A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation.
• Score: 10.813064862132379
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction. Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary <SEG> trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference. However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass. A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s. Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.

Related papers

• DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation [5.5825182710791275]
  Reasoning segmentation aims to predict pixel-wise masks for targets given complex language queries.<n>We propose DGSeg, a framework that learns to fuse predictions guided by semantic and spatial cues.<n>Experiments demonstrate that DGSeg consistently outperforms strong baselines on multiple benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T08:10:03Z)
• Bottleneck Tokens for Unified Multimodal Retrieval [16.707536543758344]
  Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps.<n>First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token as the sequence-level representation.<n>Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it.<n>We introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism.
  arXiv  Detail & Related papers  (2026-04-13T07:12:12Z)
• Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction [10.813064862132379]
  We present STAMP: Simultaneous Textual All-Mask Prediction, an MLLM that embodies this paradigm.<n>After generating a textual response, STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches.<n>This design maintains the MLLM's dialogue ability by avoiding conflicting objectives, enables high segmentation performance by leveraging rich, bidirectional spatial context for all mask tokens, and achieves exceptional speed.
  arXiv  Detail & Related papers  (2025-11-29T08:52:41Z)
• Text4Seg++: Advancing Image Segmentation via Generative Language Modeling [52.07442359419673]
  We propose a novel text-as-mask paradigm that casts image segmentation as a text generation problem.<n>Key innovation is semantic descriptors, a new textual representation of segmentation masks.<n>Experiments on natural and remote sensing datasets show that Text4Seg++ consistently outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-09-08T04:07:14Z)
• LESS: Label-Efficient and Single-Stage Referring 3D Segmentation [55.06002976797879]
  Referring 3D is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. We propose a novel Referring 3D pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. We achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels.
  arXiv  Detail & Related papers  (2024-10-17T07:47:41Z)
• Bridge the Points: Graph-based Few-shot Segment Anything Semantically [79.1519244940518]
  Recent advancements in pre-training techniques have enhanced the capabilities of vision foundation models. Recent studies extend the SAM to Few-shot Semantic segmentation (FSS) We propose a simple yet effective approach based on graph analysis.
  arXiv  Detail & Related papers  (2024-10-09T15:02:28Z)
• MaskUno: Switch-Split Block For Enhancing Instance Segmentation [0.0]
  We propose replacing mask prediction with a Switch-Split block that processes refined ROIs, classifies them, and assigns them to specialized mask predictors. An increase in the mean Average Precision (mAP) of 2.03% was observed for the high-performing DetectoRS when trained on 80 classes.
  arXiv  Detail & Related papers  (2024-07-31T10:12:14Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)
• Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models [61.46999584579775]
  General purpose segmentation models are able to generate (semantic) segmentation masks from a variety of prompts. In particular, input images are pre-processed by an image encoder to obtain embedding vectors which are later used for mask predictions. We show that even imperceptible perturbations of radius $epsilon=1/255$ are often sufficient to drastically modify the masks predicted with point, box and text prompts.
  arXiv  Detail & Related papers  (2023-11-24T12:57:34Z)
• Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation [63.910211095033596]
  Few-shot segmentation aims to segment unseen-class objects given only a handful of densely labeled samples. We propose a simple yet versatile framework in the spirit of divide-and-conquer. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information.
  arXiv  Detail & Related papers  (2022-04-21T06:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.