FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation

• URL: http://arxiv.org/abs/2605.29461v1
• Date: Thu, 28 May 2026 06:55:35 GMT
• Title: FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation
• Abstract summary: FlowSeg introduces semantic guidance via a bidirectional semantic flow between intermediate decoding states and condition embeddings.<n>This design yields semantically grounded mask representations and visually aligned language conditions, enabling more reliable matching.
• Score: 76.08991871243333
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-conditioned segmentation has recently advanced rapidly by coupling large language models with iterative mask generation frameworks. However, we identify a persistent failure mode in current propose-then-select pipelines. Although high-quality mask candidates are often generated, the final prediction may fail to match the given linguistic condition. This failure arises because language semantics are typically used as static prompts or post-hoc matching signals, rather than participating in the iterative mask generation process. Through systematic analysis, we show that many errors stem from semantic misalignment rather than poor mask quality. To address this issue, we propose FlowSeg, which introduces dynamic semantic guidance via a bidirectional semantic flow between intermediate decoding states and LLM-derived condition embeddings throughout the generation process. Language conditions actively guide mask refinement at each stage, while condition embeddings are progressively updated by emerging visual evidence. This design yields semantically grounded mask representations and visually aligned language conditions, enabling more reliable matching. We further incorporate a lightweight boundary-aware refinement to selectively enhance uncertain regions without perturbing confident interiors. Extensive experiments on referring expression segmentation and reasoning segmentation tasks demonstrate that FlowSeg consistently improves language-mask alignment and achieves state-of-the-art performance. Project page: https://zkzhang98.github.io/FlowSeg_page

Related papers

• Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis [13.853304646673637]
  Multimodal Sentiment Analysis (MSA) aims to interpret complex human emotions by integrating natural language with non-verbal modalities.<n>SentiLLM is a framework that distills continuous raw signals into compact, semantically meaningful tokens.<n>Our method achieves superior performance on four datasets, demonstrating the effectiveness of the structural abstraction paradigm in MSA.
  arXiv  Detail & Related papers  (2026-07-30T07:23:27Z)
• Masked Language Flow Models [10.204151640924454]
  Flow Language Models (FLMs) sidestep this limitation by learning a continuous flow that transports noise toward clean sequences.<n>We introduce Masked Language Flow Models (MLFMs) which incorporate masking into FLMs using a continuous interpolant to bridge partially masked and clean sequences.<n>We find, for the first time, that flow-based language models can be scaled to solve downstream reasoning and instruction-following tasks.
  arXiv  Detail & Related papers  (2026-06-26T00:16:40Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding [41.70825455674463]
  SWIM (See What I Mean) is a training strategy that aligns vision and language representations to enable fine-grained object understanding solely from textual prompts.<n>We show that SWIM substantially improves text-visual alignment and achieves superior performance over visual-prompt-based methods on fine-grained object understanding benchmarks.
  arXiv  Detail & Related papers  (2026-05-18T08:09:37Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models [2.3863052459868297]
  We propose a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation.<n> Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2026-03-16T14:28:39Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• Test-Time Computing for Referring Multimodal Large Language Models [143.49848714354698]
  We propose ControlMLLM++, a novel test-time adaptation framework.<n>It injects learnable visual prompts into frozen multimodal large language models.
  arXiv  Detail & Related papers  (2026-02-23T04:42:10Z)
• Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction [10.813064862132379]
  We present STAMP: Simultaneous Textual All-Mask Prediction, an MLLM that embodies this paradigm.<n>After generating a textual response, STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches.<n>This design maintains the MLLM's dialogue ability by avoiding conflicting objectives, enables high segmentation performance by leveraging rich, bidirectional spatial context for all mask tokens, and achieves exceptional speed.
  arXiv  Detail & Related papers  (2025-11-29T08:52:41Z)
• Text4Seg++: Advancing Image Segmentation via Generative Language Modeling [52.07442359419673]
  We propose a novel text-as-mask paradigm that casts image segmentation as a text generation problem.<n>Key innovation is semantic descriptors, a new textual representation of segmentation masks.<n>Experiments on natural and remote sensing datasets show that Text4Seg++ consistently outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-09-08T04:07:14Z)
• Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder [5.57393627015653]
  Video segmenter and grounder approaches, exemplified by Sa2VA, directly fuse features within segmentation models.<n>This often results in an undesirable entanglement of dynamic visual information and static semantics, thereby degrading segmentation accuracy.<n>We propose DeSa2VA, a decoupling-enhanced prompting scheme integrating text pre-training and a linear decoupling module to address the information processing limitations inherent in SAM-2.
  arXiv  Detail & Related papers  (2025-06-28T13:30:36Z)
• GLOS: Sign Language Generation with Temporally Aligned Gloss-Level Conditioning [60.86278956347739]
  GLOS is a sign language generation framework with temporally aligned gloss-level conditioning.<n>Our method generates signs with correct lexical order and high semantic accuracy, outperforming prior methods on CSL-Daily and Phoenix-2014T.
  arXiv  Detail & Related papers  (2025-06-09T06:09:03Z)
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training [152.63467944568094]
  We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks.
  arXiv  Detail & Related papers  (2020-02-28T15:28:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.