Agentic DraCor and the Art of Docstring Engineering: Evaluating MCP-empowered LLM Usage of the DraCor API

• URL: http://arxiv.org/abs/2508.13774v1
• Date: Tue, 19 Aug 2025 12:21:21 GMT
• Title: Agentic DraCor and the Art of Docstring Engineering: Evaluating MCP-empowered LLM Usage of the DraCor API
• Abstract summary: This paper reports on the implementation and evaluation of a Model Context Protocol server for DraCor.<n>It enables Large Language Models (LLM) to autonomously interact with the DraCor API.
• Score: 0.10555513406636088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper reports on the implementation and evaluation of a Model Context Protocol (MCP) server for DraCor, enabling Large Language Models (LLM) to autonomously interact with the DraCor API. We conducted experiments focusing on tool selection and application by the LLM, employing a qualitative approach that includes systematic observation of prompts to understand how LLMs behave when using MCP tools, evaluating "Tool Correctness", "Tool-Calling Efficiency", and "Tool-Use Reliability". Our findings highlight the importance of "Docstring Engineering", defined as reflexively crafting tool documentation to optimize LLM-tool interaction. Our experiments demonstrate both the promise of agentic AI for research in Computational Literary Studies and the essential infrastructure development needs for reliable Digital Humanities infrastructures.

Related papers

• Automated Summarization of Software Documents: An LLM-based Multi-Agent Approach [4.230025065044209]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) are revolutionizing software engineering (SE) by advancing automation, decision-making, and knowledge processing.<n>We present Metagente, an LLM-based MAS designed to generate concise and accurate summaries of software documentation.
  arXiv  Detail & Related papers  (2026-06-23T15:18:17Z)
• LitBench: A Graph-Centric Large Language Model Benchmarking Tool For Literature Tasks [31.14225125626119]
  We introduce LitBench, a benchmarking tool designed to enable the development and evaluation of domain-specific language models.<n>At its core, LitBench uses a data curation process that generates domain-specific literature sub-graphs.<n>In addition to dataset curation, LitBench defines a comprehensive suite of literature tasks, ranging from node and edge level analyses to advanced applications.
  arXiv  Detail & Related papers  (2026-02-10T04:12:29Z)
• InsightGUIDE: An Opinionated AI Assistant for Guided Critical Reading of Scientific Literature [0.13258332549279203]
  InsightGUIDE is a novel AI-powered tool designed to function as a reading assistant, not a replacement.<n>Our system provides concise, structured insights that act as a "map" to a paper's key elements by embedding an expert's reading methodology directly into its core AI logic.
  arXiv  Detail & Related papers  (2025-09-24T19:10:52Z)
• MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models [11.809732662992982]
  This paper introduces MCP-RADAR, the first comprehensive benchmark specifically designed to evaluate Large Language Models (LLMs) performance in the Model Context Protocol (MCP) framework.<n>Unlike conventional benchmarks that rely on subjective human evaluations or binary success metrics, MCP-RADAR employs objective, quantifiable measurements across multiple task domains.
  arXiv  Detail & Related papers  (2025-05-22T14:02:37Z)
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning [63.31585771716123]
  Large language models (LLMs) have shown remarkable reasoning capabilities via large-scale reinforcement learning (RL)<n>We introduce Tool-Star, an RL-based framework designed to empower LLMs to autonomously invoke multiple external tools during stepwise reasoning.<n>Tool-Star integrates six types of tools and incorporates systematic designs in both data synthesis and training.
  arXiv  Detail & Related papers  (2025-05-22T09:00:19Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification [0.0]
  This study evaluates the efficacy of Large Language Models (LLMs) as innovative solutions for mitigating misinformation on platforms like Twitter.<n>LLMs offer a pre-trained, adaptable approach that bypasses the extensive training and overfitting issues associated with traditional machine learning models.<n>We present a comparative analysis of LLMs' performance using a specialized dataset and propose a framework for their application in public health communication.
  arXiv  Detail & Related papers  (2024-12-21T05:02:26Z)
• From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions [60.733557487886635]
  This paper focuses on bridging the comprehension gap between Large Language Models and external tools.<n>We propose a novel framework, DRAFT, aimed at Dynamically Refining tool documentation.<n>This methodology pivots on an innovative trial-and-error approach, consisting of three distinct learning phases.
  arXiv  Detail & Related papers  (2024-10-10T17:58:44Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone.<n>We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench.<n>We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks [33.51887014808227]
  This paper explores the impact of both internal and external factors on the performance of tool learning frameworks. We find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration.
  arXiv  Detail & Related papers  (2024-07-03T11:06:05Z)
• Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents [56.822238860147024]
  Augmenting large language models with external tools has emerged as a promising approach to extend their utility.<n>Previous methods manually parse tool documentation and create in-context demonstrations, transforming tools into structured formats for LLMs to use in their step-by-step reasoning.<n>We propose AutoTools, a framework that enables LLMs to automate the tool-use workflow.
  arXiv  Detail & Related papers  (2024-05-26T11:40:58Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
  We make a systematic review of the literature, including the general methodology of supervised fine-tuning (SFT)<n>We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies.
  arXiv  Detail & Related papers  (2023-08-21T15:35:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.