Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges

• URL: http://arxiv.org/abs/2507.09562v1
• Date: Sun, 13 Jul 2025 10:10:17 GMT
• Title: Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges
• Authors: Yidong Jiang,
• Abstract summary: The Segment Anything Model (SAM) has revolutionized image segmentation through its innovative prompt-based approach.<n>This paper presents the first comprehensive survey focusing specifically on prompt engineering techniques for SAM and its variants.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Segment Anything Model (SAM) has revolutionized image segmentation through its innovative prompt-based approach, yet the critical role of prompt engineering in its success remains underexplored. This paper presents the first comprehensive survey focusing specifically on prompt engineering techniques for SAM and its variants. We systematically organize and analyze the rapidly growing body of work in this emerging field, covering fundamental methodologies, practical applications, and key challenges. Our review reveals how prompt engineering has evolved from simple geometric inputs to sophisticated multimodal approaches, enabling SAM's adaptation across diverse domains including medical imaging and remote sensing. We identify unique challenges in prompt optimization and discuss promising research directions. This survey fills an important gap in the literature by providing a structured framework for understanding and advancing prompt engineering in foundation models for segmentation.

Related papers

• A Survey of Context Engineering for Large Language Models [31.68644305980195]
  This survey introduces Context Engineering, a formal discipline that transcends simple prompt design.<n>We first examine the foundational components: context retrieval and generation, context processing and context management.<n>We then explore how these components are architecturally integrated to create sophisticated system implementations.
  arXiv  Detail & Related papers  (2025-07-17T17:50:36Z)
• Vision Generalist Model: A Survey [87.49797517847132]
  We provide a comprehensive overview of the vision generalist models, delving into their characteristics and capabilities within the field.<n>We take a brief excursion into related domains, shedding light on their interconnections and potential synergies.
  arXiv  Detail & Related papers  (2025-06-11T17:23:41Z)
• Large Language Model Agent: A Survey on Methodology, Applications and Challenges [88.3032929492409]
  Large Language Model (LLM) agents, with goal-driven behaviors and dynamic adaptation capabilities, potentially represent a critical pathway toward artificial general intelligence.<n>This survey systematically deconstructs LLM agent systems through a methodology-centered taxonomy.<n>Our work provides a unified architectural perspective, examining how agents are constructed, how they collaborate, and how they evolve over time.
  arXiv  Detail & Related papers  (2025-03-27T12:50:17Z)
• Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey [124.23247710880008]
  multimodal CoT (MCoT) reasoning has recently garnered significant research attention.<n>Existing MCoT studies design various methodologies to address the challenges of image, video, speech, audio, 3D, and structured data.<n>We present the first systematic survey of MCoT reasoning, elucidating the relevant foundational concepts and definitions.
  arXiv  Detail & Related papers  (2025-03-16T18:39:13Z)
• The Prompt Canvas: A Literature-Based Practitioner Guide for Creating Effective Prompts in Large Language Models [0.0]
  This paper argues for the creation of an overarching framework that synthesizes existing methodologies into a cohesive overview for practitioners.<n>We present the Prompt Canvas, a structured framework resulting from an extensive literature review on prompt engineering.
  arXiv  Detail & Related papers  (2024-12-06T15:35:18Z)
• From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models [56.9134620424985]
  Cross-modal reasoning (CMR) is increasingly recognized as a crucial capability in the progression toward more sophisticated artificial intelligence systems. The recent trend of deploying Large Language Models (LLMs) to tackle CMR tasks has marked a new mainstream of approaches for enhancing their effectiveness. This survey offers a nuanced exposition of current methodologies applied in CMR using LLMs, classifying these into a detailed three-tiered taxonomy.
  arXiv  Detail & Related papers  (2024-09-19T02:51:54Z)
• Segment Anything for Videos: A Systematic Survey [52.28931543292431]
  The recent wave of foundation models has witnessed tremendous success in computer vision (CV) and beyond. The segment anything model (SAM) has sparked a passion for exploring task-agnostic visual foundation models. This work conducts a systematic review on SAM for videos in the era of foundation models.
  arXiv  Detail & Related papers  (2024-07-31T02:24:53Z)
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications [11.568575664316143]
  This paper provides a structured overview of recent advancements in prompt engineering, categorized by application area.<n>We provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized.<n>This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.
  arXiv  Detail & Related papers  (2024-02-05T19:49:13Z)
• A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering [49.732628643634975]
  The Segment Anything Model (SAM), developed by Meta AI Research, offers a robust framework for image and video segmentation. This survey provides a comprehensive exploration of the SAM family, including SAM and SAM 2, highlighting their advancements in granularity and contextual understanding.
  arXiv  Detail & Related papers  (2023-05-12T07:21:59Z)
• Evolutionary Multitask Optimization: a Methodological Overview, Challenges and Future Research Directions [8.14509634354919]
  We consider multitasking in the context of solving multiple optimization problems simultaneously by conducting a single search process. The emerging paradigm of Evolutionary Multitasking tackles multitask optimization scenarios by using as inspiration concepts drawn from Evolutionary Computation.
  arXiv  Detail & Related papers  (2021-02-04T11:48:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.