FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation

• URL: http://arxiv.org/abs/2608.13980v1
• Date: Fri, 14 Aug 2026 05:40:36 GMT
• Title: FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation
• Abstract summary: Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into pixel-level masks.<n>In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token.<n>We introduce FIRM, a Fine-grained Intra-token Representation of Masks.
• Score: 27.47065769615997
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into precise pixel-level masks. MLLMs encode an image as visual tokens, each of which merges a group of image patches. In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token. Assigning a single binary mask label to such a token loses its internal spatial structure, causing nearby targets to merge and object boundaries to become coarse. To bridge this representational gap, we introduce FIRM, a Fine-grained Intra-token Representation of Masks. For each visual token, FIRM predicts a mask code that specifies an $r\times r$ binary sub-cell pattern rather than a single foreground/background label. Given a target identified by the MLLM, the complete grid of mask codes is predicted in one mask pass. Fixed lookup converts the predicted codes into a discrete sub-cell mask, while marginalizing the code distribution yields a soft structural field. To further recover fine-grained boundaries within each sub-cell, we introduce a lightweight continuous renderer that refines this field using pre-merge visual features and image details. Across five reasoning and referring segmentation benchmarks on satellite and UAV images, FIRM achieves leading results, including $70.5/80.5$ gIoU/cIoU on LaSeRS and a $3.0$-point average gain on EarthReason. These results demonstrate the value of explicitly representing intra-token mask patterns for fine-grained MLLM segmentation.

Related papers

• SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization [64.6372217552272]
  Public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks.<n>We propose SIGMA, which performs semantic-feature differencing in a vision foundation backbone.<n>It produces a 1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets.
  arXiv  Detail & Related papers  (2026-05-27T03:55:13Z)
• Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token [75.67455028589505]
  This paper investigates whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E)<n>Our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs.<n>Comprehensive experiments validate that SELF1E achieves performance competitive with specialist mask decoder-based methods.
  arXiv  Detail & Related papers  (2026-03-19T15:25:36Z)
• SAMTok: Representing Any Mask with Two Words [70.74140779649856]
  We present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens.<n>By treating masks as new language tokens, SAMTok enables base MLLMs to learn pixel-wise capabilities.<n>QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation.
  arXiv  Detail & Related papers  (2026-01-22T16:44:09Z)
• LlamaSeg: Image Segmentation via Autoregressive Mask Generation [46.17509085054758]
  We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions.<n>We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs.
  arXiv  Detail & Related papers  (2025-05-26T02:22:41Z)
• HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model [6.641903410779405]
  We propose the Hierarchical Mask Tokenizer (HiMTok), which represents segmentation masks with up to 32 tokens.<n>HiMTok allows for compact and coarse-to-fine mask representations, aligning well with the next-token-prediction paradigm.<n>We develop a 3-stage training recipe for progressive learning of segmentation and visual capabilities, featuring a hierarchical mask loss for effective coarse-to-fine learning.
  arXiv  Detail & Related papers  (2025-03-17T10:29:08Z)
• High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation [109.19165503929992]
  We present MaskCLIP++, which uses ground-truth masks instead of generated masks to enhance the mask classification capability of CLIP.<n>After low-cost fine-tuning, MaskCLIP++ significantly improves the mask classification performance on multi-domain datasets.<n>We achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets.
  arXiv  Detail & Related papers  (2024-12-16T05:44:45Z)
• CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation [104.5033800500497]
  Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects. We propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method.
  arXiv  Detail & Related papers  (2022-12-04T08:53:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.