An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

• URL: http://arxiv.org/abs/2606.00987v1
• Date: Sun, 31 May 2026 04:01:10 GMT
• Title: An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation
• Abstract summary: We introduce textbfMulti-temporal Referring (MTRS), a new task that aims to segment language-described temporal changes from multi-temporal images.<n>MTRS extends conventional referring segmentation and change detection by jointly requiring temporal correspondence reasoning, language grounding, and pixel-level mask prediction.<n>We show that MTRefSeg-R1 achieves strong and often superior performance compared with existing LVLM baselines.
• Score: 55.291219073365546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temporal visual reasoning remains underexplored. To bridge this gap, we introduce \textbf{Multi-temporal Referring Segmentation (MTRS)}, a new task that aims to segment language-described temporal changes from multi-temporal images. MTRS extends conventional referring segmentation and change detection by jointly requiring temporal correspondence reasoning, language grounding, and pixel-level mask prediction. We propose \textbf{CRAFT-Agent}, an automated data construction pipeline with human auditing, and build \textbf{MTRefSeg-21K}, the first MTRS benchmark, containing 21K high-quality multi-temporal image-text-mask triplets across diverse scenes, viewpoints, and domains. Benchmarking a broad set of VLM- and LVLM-based models reveals that direct inference performs poorly, while task-specific fine-tuning remains limited. To address this, we propose \textbf{MTRefSeg-R1}, a change-aware LVLM framework trained with a two-stage strategy. It first learns general temporal-change perception from 20K vision-only bi-temporal samples, and is then fine-tuned on MTRefSeg-21K for fine-grained language-guided temporal localization. MTRefSeg-R1 explicitly models cross-temporal visual differences, aligns language instructions with temporal variations, and predicts referred change masks. Extensive experiments show that MTRefSeg-R1 achieves strong and often superior performance compared with existing LVLM baselines, demonstrating the challenge and potential of MTRS.

Related papers

• SAGE: Variate-Wise Semantic Augmentation for Vision-Language Time Series Forecasting [6.3330628137530605]
  Time series forecasting models operate on raw numerical sequences, lacking the semantic knowledge that domain experts implicitly leverage.<n>We propose SAGE, an end-to-end CLIP-based framework that jointly models temporal, cross-variable, textual, and visual information.
  arXiv  Detail & Related papers  (2026-08-27T09:04:45Z)
• Masked Diffusion Vision-Language Models for Temporal Action Localization [10.018515245238216]
  Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos.<n>Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right.<n>We adapt vision-language models to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising.
  arXiv  Detail & Related papers  (2026-05-28T12:39:04Z)
• Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models [58.9387276447485]
  Vision Inference Former (VIF) is a lightweight architectural module that establishes a bridge between pure visual representations and the model's output space.<n>We conduct experiments on 14 benchmark tasks covering general reasoning, OCR, table understanding, vision-centric evaluation, and hallucination.
  arXiv  Detail & Related papers  (2026-05-18T10:04:22Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding [6.673833357112721]
  textbfBridge-STG is an end-to-end framework that decouples temporal and spatial localization while maintaining semantic coherence.<n>Bridge-STG achieves state-of-the-art performance among MLLM-based methods.
  arXiv  Detail & Related papers  (2026-04-09T09:14:00Z)
• Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining [86.49790441700195]
  ClusterSTM is a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining.<n>We introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction.
  arXiv  Detail & Related papers  (2026-03-24T08:48:15Z)
• Temporal Grounding as a Learning Signal for Referring Video Object Segmentation [29.646697516547558]
  Referring Video Object (RVOS) aims to segment and track objects in videos based on natural language expressions, requiring precise alignment between visual content and textual queries.<n>Existing methods often suffer from semantic misalignment, largely due to indiscriminate frame sampling and supervision of all visible objects during training.<n>We introduce MeViS-M, a dataset built upon the challenging MeViS benchmark, where we manually annotate temporal spans when each object is referred to by the expression.
  arXiv  Detail & Related papers  (2025-08-16T07:34:43Z)
• Teaching Time Series to See and Speak: Forecasting with Aligned Visual and Textual Perspectives [22.10401153489018]
  Time series forecasting traditionally relies on unimodal numerical inputs.<n>We propose a multimodal contrastive learning framework that transforms raw time series into structured visual and textual perspectives.
  arXiv  Detail & Related papers  (2025-06-30T17:59:14Z)
• Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey [35.600870905903996]
  We present the first comprehensive review of RS-STVLMs.<n>We discuss progress in representative tasks, such as change captioning, change question, answering captions and change grounding.<n>We aim to illuminate current achievements and promising directions for future research in vision-language understanding for remote sensing.
  arXiv  Detail & Related papers  (2024-12-03T16:56:10Z)
• Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation [62.275143240798236]
  Video semantic segmentation dataset has limited categories per video. Less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. Our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.
  arXiv  Detail & Related papers  (2023-09-14T20:31:06Z)
• Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation [79.1669476932147]
  Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction. We introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation.
  arXiv  Detail & Related papers  (2021-11-10T16:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.