Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation

• URL: http://arxiv.org/abs/2603.12538v1
• Date: Fri, 13 Mar 2026 00:37:20 GMT
• Title: Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation
• Abstract summary: Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression.<n>We propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation.<n> SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework.
• Score: 0.3437656066916039
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression. Although pretrained vision-language models have improved semantic grounding, many existing methods still rely on uniform refinement strategies that do not fully match the diverse reasoning requirements of referring expressions. Because of this mismatch, predictions often contain fragmented regions, inaccurate boundaries, or even the wrong object, especially when pretrained backbones are frozen for computational efficiency. To address these limitations, we propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation. SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework. First, we design SERA-Adapter, which inserts an expression-conditioned adapter into selected backbone blocks to improve spatial coherence and boundary precision through expert-guided refinement and cross-modal attention. We then introduce SERA-Fusion, which strengthens intermediate visual representations by reshaping token features into spatial grids and applying geometry-preserving expert transformations before multimodal interaction. In addition, a lightweight routing mechanism adaptively weights expert contributions while remaining compatible with pretrained representations. To make this routing stable under frozen encoders, SERA uses a parameter-efficient tuning strategy that updates only normalization and bias terms, affecting less than 1% of the backbone parameters. Experiments on standard referring image segmentation benchmarks show that SERA consistently outperforms strong baselines, with especially clear gains on expressions that require accurate spatial localization and precise boundary delineation.

Related papers

• Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation [14.492779733437082]
  Referring Remote Sensing Image seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image.<n>Existing RRSIS models have benefited from large-scale foundation models, but they predominantly rely on full fine-tuning.<n>We propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment.
  arXiv  Detail & Related papers  (2026-06-29T12:54:10Z)
• ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images [51.53960096934913]
  Remote sensing visual grounding (RSVG) aims to locate specific objects in high-resolution RS imagery using free-form natural language descriptions.<n>Recent advances in multimodal large language models (MLLMs) show great potential for such open-vocabulary RSVG.<n>We propose Exemplar-driven Calibrated Refinement (ExACT), a training-free framework driven by a one-shot visual prompting mechanism.
  arXiv  Detail & Related papers  (2026-06-27T13:50:39Z)
• Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models [22.803631190636313]
  We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved at the input.<n>The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.
  arXiv  Detail & Related papers  (2026-06-18T08:40:41Z)
• Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation [30.988795633147138]
  We investigate whether language-conditioned visual semantic grounding can be leveraged for zero-shot referring image segmentation.<n>We propose a training-free framework that repurposes pretrained image editing models for RIS by exploiting their intermediate representations.<n>Our approach decomposes localization into two complementary components: attention-based spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment.
  arXiv  Detail & Related papers  (2026-05-13T07:48:05Z)
• ProxyImg: Towards Highly-Controllable Image Representation via Hierarchical Disentangled Proxy Embedding [44.20713526887855]
  We propose a hierarchical proxy-based parametric image representation that disentangles semantic, geometric, and textural attributes into independent parameter spaces.<n>Our method achieves state-of-the-art rendering fidelity with significantly fewer parameters, while enabling intuitive, interactive, and physically plausible manipulation.
  arXiv  Detail & Related papers  (2026-02-02T09:53:45Z)
• DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation [16.57245702815661]
  Open-vocabulary semantic segmentation aims to assign labels to every pixel in an image based on text labels.<n>Existing approaches typically utilize vision-language models (VLMs), such as CLIP, for dense prediction.<n>We introduce DiSa, a novel saliency-aware foreground-background disentangled framework.
  arXiv  Detail & Related papers  (2026-01-27T21:15:10Z)
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• DiffRIS: Enhancing Referring Remote Sensing Image Segmentation with Pre-trained Text-to-Image Diffusion Models [9.109484087832058]
  DiffRIS is a novel framework that harnesses the semantic understanding capabilities of pre-trained text-to-image diffusion models for RRSIS tasks.<n>Our framework introduces two key innovations: a context perception adapter (CP-adapter) and a cross-modal reasoning decoder (PCMRD)
  arXiv  Detail & Related papers  (2025-06-23T02:38:56Z)
• Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation [12.893224628061516]
  The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression.<n>We propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges.<n>Our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets.
  arXiv  Detail & Related papers  (2025-01-01T14:24:04Z)
• Cross-Modal Bidirectional Interaction Model for Referring Remote Sensing Image Segmentation [50.433911327489554]
  The goal of referring remote sensing image segmentation (RRSIS) is to generate a pixel-level mask of the target object identified by the referring expression.<n>To address the aforementioned challenges, a novel RRSIS framework is proposed, termed the cross-modal bidirectional interaction model (CroBIM)<n>To further forster the research of RRSIS, we also construct RISBench, a new large-scale benchmark dataset comprising 52,472 image-language-label triplets.
  arXiv  Detail & Related papers  (2024-10-11T08:28:04Z)
• Guided Interpretable Facial Expression Recognition via Spatial Action Unit Cues [55.97779732051921]
  A new learning strategy is proposed to explicitly incorporate au cues into classifier training. We show that our strategy can improve layer-wise interpretability without degrading classification performance.
  arXiv  Detail & Related papers  (2024-02-01T02:13:49Z)
• GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding [101.32590239809113]
  Generalized Perception NeRF (GP-NeRF) is a novel pipeline that makes the widely used segmentation model and NeRF work compatibly under a unified framework. We propose two self-distillation mechanisms, i.e., the Semantic Distill Loss and the Depth-Guided Semantic Distill Loss, to enhance the discrimination and quality of the semantic field.
  arXiv  Detail & Related papers  (2023-11-20T15:59:41Z)
• AF$_2$: Adaptive Focus Framework for Aerial Imagery Segmentation [86.44683367028914]
  Aerial imagery segmentation has some unique challenges, the most critical one among which lies in foreground-background imbalance. We propose Adaptive Focus Framework (AF$), which adopts a hierarchical segmentation procedure and focuses on adaptively utilizing multi-scale representations. AF$ has significantly improved the accuracy on three widely used aerial benchmarks, as fast as the mainstream method.
  arXiv  Detail & Related papers  (2022-02-18T10:14:45Z)
• Improving Semantic Segmentation via Decoupled Body and Edge Supervision [89.57847958016981]
  Existing semantic segmentation approaches either aim to improve the object's inner consistency by modeling the global context, or refine objects detail along their boundaries by multi-scale feature fusion. In this paper, a new paradigm for semantic segmentation is proposed. Our insight is that appealing performance of semantic segmentation requires textitexplicitly modeling the object textitbody and textitedge, which correspond to the high and low frequency of the image. We show that the proposed framework with various baselines or backbone networks leads to better object inner consistency and object boundaries.
  arXiv  Detail & Related papers  (2020-07-20T12:11:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.