Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation

• URL: http://arxiv.org/abs/2604.07916v1
• Date: Thu, 09 Apr 2026 07:37:09 GMT
• Title: Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentation
• Abstract summary: We present Tarot-SAM3, a training-free framework that can accurately segment from any referring expression.<n>Tarot-SAM3 consists of two key phases. First, the Reasoning Expression Interpreter (ERI) phase introduces reasoning-assisted prompt options.<n>Second, the Mask Self-Refining (MSR) phase selects the best mask across prompt types and performs self-refinement.
• Score: 18.568343383992072
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Expression Segmentation (RES) aims to segment image regions described by natural-language expressions, serving as a bridge between vision and language understanding. Existing RES methods, however, rely heavily on large annotated datasets and are limited to either explicit or implicit expressions, hindering their ability to generalize to any referring expression. Recently, the Segment Anything Model 3 (SAM3) has shown impressive robustness in Promptable Concept Segmentation. Nonetheless, applying it to RES remains challenging: (1) SAM3 struggles with longer or implicit expressions; (2) naive coupling of SAM3 with a multimodal large language model (MLLM) makes the final results overly dependent on the MLLM's reasoning capability, without enabling refinement of SAM3's segmentation outputs. To this end, we present Tarot-SAM3, a novel training-free framework that can accurately segment from any referring expression. Specifically, Tarot-SAM3 consists of two key phases. First, the Expression Reasoning Interpreter (ERI) phase introduces reasoning-assisted prompt options to support structured expression parsing and evaluation-aware rephrasing. This transforms arbitrary queries into robust heterogeneous prompts for generating reliable masks with SAM3. Second, the Mask Self-Refining (MSR) phase selects the best mask across prompt types and performs self-refinement by leveraging rich feature relationships from DINOv3 to compare discriminative regions among ERI outputs. It then infers region affiliation to the target, thereby correcting over- and under-segmentation. Extensive experiments demonstrate that Tarot-SAM3 achieves strong performance on both explicit and implicit RES benchmarks, as well as open-world scenarios. Ablation studies further validate the effectiveness of each phase.

Related papers

• Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3) [0.866627581195388]
  We propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction.<n>In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability.<n>In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification.
  arXiv  Detail & Related papers  (2026-08-09T18:11:09Z)
• EOVSAM: Efficient Open-Vocabulary Segmentation with SAM 3 in One Pass [79.69317943751223]
  We propose an Efficient Open-Vocabulary segmentation framework with SAM 3 (EOVSAM)<n>EOVSAM adapts SAM 3 for single-pass prediction.<n>It consistently improves segmentation accuracy over vanilla SAM 3 on all evaluated datasets.
  arXiv  Detail & Related papers  (2026-08-03T14:15:37Z)
• InstructSAM: Segment Any Instance with Any Instructions [70.32433456722613]
  InstructSAM is a framework designed for multi-instance segmentation under arbitrary instructions.<n>A bank of learnable instance queries is injected into a vision-language model (VLM) and SAM3.<n>A hybrid-attention mechanism promotes interaction among these queries, visual tokens, and instruction tokens.
  arXiv  Detail & Related papers  (2026-05-25T17:58:03Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation [37.55469031169627]
  The Segment Anything Model (SAM) excels at general image segmentation but has limited ability to understand natural language.<n>We propose SSP-SAM, a framework that fully utilizes SAM's segmentation capabilities by integrating a Semantic-Spatial Prompt (SSP) encoder.<n>SSP-SAM enhances the referent representation for the prompt generator, resulting in high-quality SSPs that enable SAM to generate precise masks guided by language.
  arXiv  Detail & Related papers  (2026-03-18T08:26:07Z)
• SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images [51.42466259821335]
  We present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training.<n>First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder.<n>Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene.
  arXiv  Detail & Related papers  (2025-12-09T15:42:28Z)
• SAM3-I: Segment Anything with Instructions [86.92593395772029]
  We present SAM3-I, an enhanced framework that unifies concept-level understanding and instructionlevel reasoning within the SAM family.<n>We design a structured instruction taxonomy spanning concept, simple, and complex levels, and develop a scalable data engine to construct a dataset with diverse instruction-mask pairs.
  arXiv  Detail & Related papers  (2025-12-04T09:00:25Z)
• SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation [88.80792308991867]
  Segment Anything model (SAM) has shown ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges.<n>This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation.<n> Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains.
  arXiv  Detail & Related papers  (2024-07-23T17:47:25Z)
• EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model [39.967352995143855]
  We introduce the Early Vision-language Fusion-based SAM (EVF-SAM)<n>EVF-SAM is a simple yet effective referring segmentation method which exploits multimodal prompts (i.e., image and text)<n>Experiments show that the proposed EVF-SAM based on BEIT-3 can obtain state-of-the-art performance on RefCOCO/+/g for referring expression segmentation.
  arXiv  Detail & Related papers  (2024-06-28T17:38:18Z)
• AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning [61.666973416903005]
  Segment Anything Model (SAM) has demonstrated its impressive generalization capabilities in open-world scenarios with the guidance of prompts. We propose a novel framework, termed AlignSAM, designed for automatic prompting for aligning SAM to an open context.
  arXiv  Detail & Related papers  (2024-06-01T16:21:39Z)
• Boosting Segment Anything Model Towards Open-Vocabulary Learning [69.24734826209367]
  Segment Anything Model (SAM) has emerged as a new paradigmatic vision foundation model.<n>Despite SAM finding applications and adaptations in various domains, its primary limitation lies in the inability to grasp object semantics.<n>We present Sambor to seamlessly integrate SAM with the open-vocabulary object detector in an end-to-end framework.
  arXiv  Detail & Related papers  (2023-12-06T17:19:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.