A Reproducible Optimisation Protocol for Calibrating Prompt-Based Large Language Model Workflows in Evidence Synthesis

• URL: http://arxiv.org/abs/2605.06937v1
• Date: Thu, 07 May 2026 20:55:07 GMT
• Title: A Reproducible Optimisation Protocol for Calibrating Prompt-Based Large Language Model Workflows in Evidence Synthesis
• Abstract summary: This article presents a reproducible calibration workflow for prompt-based large language models (LLMs) in structured evidence-synthesis tasks.<n>The method separates the rules that define the scientific task from the mutable prompt harness that frames and applies them.<n>It optimises that harness against labelled or reference examples and an explicit task metric, then preserves the calibrated workflow as an inspectable artefact.
• Score: 1.624454100511275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This methods article presents a reproducible calibration workflow for prompt-based large language models (LLMs) in structured evidence-synthesis tasks. The method separates the rules that define the scientific task from the mutable prompt harness that frames and applies them. It optimises that harness against labelled or reference examples and an explicit task metric, then preserves the calibrated workflow as an inspectable artefact with its specification, metric, settings, and evaluation traces. The example code instantiates the protocol with DSPy and GEPA tools, but the underlying logic can transfer to other prompt-optimisation frameworks that support structured task definitions, metric-guided search, and artefact reuse. Title and abstract screening is the worked validation case because it provides labelled benchmark data and clear evaluation metrics. The demonstrated workflow uses a smaller student LLM for performing the scientific task execution and a larger reflection LLM to steer the prompt optimisation process during calibration. This work shows compilation, artefact round-tripping, and how optimisation budget affects a smaller student model.

Related papers

• GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science [39.146761527401424]
  GRACE-DS is a Reward-guided Agent Correction Environment in Data Science for pre-deployment evaluation of LLM-powered AutoML agents.<n>It exposes agents to realistic workflow stages, from planning and data inspection through feature engineering, model development, validation, and code repair to final submission.<n>These results establish GRACE-DS as a robust platform for assessing the capacity of LLM-based AutoML agents to execute machine learning under production-like conditions.
  arXiv  Detail & Related papers  (2026-06-14T19:58:06Z)
• Reflective Prompt Tuning through Language Model Function-Calling [16.552558967042312]
  Reflective Prompt Tuning (RPT) is a framework that simulates the iterative workflow of human prompt engineers.<n>RPT improves over initial prompts by up to 12.9 points, remains competitive with state of the art, and improves confidence calibration.<n>Our analyses show that RPT is especially effective on multi-hop and mathematical reasoning.
  arXiv  Detail & Related papers  (2026-05-20T22:21:29Z)
• Prompt-Driven Code Summarization: A Systematic Literature Review [4.238464304669881]
  Large language models (LLMs) offer a promising solution by automatically generating natural language descriptions from source code.<n>This systematic review literature consolidates existing evidence, categorizes prompting paradigms, examines their effectiveness, and identifies gaps to guide future research and practical adoption.
  arXiv  Detail & Related papers  (2026-04-16T03:25:28Z)
• Optimizing LLM Prompt Engineering with DSPy Based Declarative Learning [0.0]
  Large Language Models (LLMs) have shown strong performance across a wide range of natural language processing tasks.<n>Their effectiveness is highly dependent on prompt design, structure, and embedded reasoning.<n>This paper presents a systematic study of DSPy-based declarative learning for prompt optimization.
  arXiv  Detail & Related papers  (2026-04-06T17:17:57Z)
• From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents [39.457720579458]
  Large language model (LLM)-based systems are becoming increasingly popular for solving tasks.<n>This survey reviews recent methods for designing and optimizing such computation.
  arXiv  Detail & Related papers  (2026-03-23T17:42:59Z)
• From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs [48.83701310501069]
  Large language models (LLMs) have achieved notable performance in code synthesis.<n>We introduce a performance-aware, closed-loop solution that enables LLMs to autonomously engineer optimal transformations.<n>We fine-tune LLMs with Low-Rank Adaptation on a novel repository of more than 6,000 empirically evaluated PyTorch augmentation functions.
  arXiv  Detail & Related papers  (2026-01-07T11:13:02Z)
• MedDCR: Learning to Design Agentic Workflows for Medical Coding [55.51674334874892]
  Medical coding converts free-text clinical notes into standardized diagnostic and procedural codes.<n>We present MedDCR, a closed-loop framework that treats design as a learning problem.<n>On benchmark datasets, MedDCR outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-17T13:30:51Z)
• Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis [1.624454100511275]
  Large language models (LLMs) offer significant potential to accelerate systematic literature reviews.<n>Current approaches often rely on brittle, manually crafted prompts that compromise reliability and rigour.<n>This research proposes a structured, domain-specific framework that embeds task declarations, test suites, and automated prompt tuning into a reproducible SLR.
  arXiv  Detail & Related papers  (2025-08-22T21:37:49Z)
• In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement [71.60563181678323]
  Large language models (LLMs) have achieved great success across diverse tasks, and fine-tuning is sometimes needed to further enhance generation quality.<n>To handle these challenges, a direct solution is to generate high-confidence'' data from unsupervised downstream tasks.<n>We propose a novel approach, pseudo-supervised demonstrations aligned prompt optimization (PAPO) algorithm, which jointly refines both the prompt and the overall pseudo-supervision.
  arXiv  Detail & Related papers  (2024-10-04T03:39:28Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting [68.19544657508509]
  Large language models (LLMs) are adopted as a fundamental component of language technologies. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt format in few-shot settings. We propose an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights.
  arXiv  Detail & Related papers  (2023-10-17T15:03:30Z)
• How Fine-Tuning Allows for Effective Meta-Learning [50.17896588738377]
  We present a theoretical framework for analyzing representations derived from a MAML-like algorithm. We provide risk bounds on the best predictor found by fine-tuning via gradient descent, demonstrating that the algorithm can provably leverage the shared structure. This separation result underscores the benefit of fine-tuning-based methods, such as MAML, over methods with "frozen representation" objectives in few-shot learning.
  arXiv  Detail & Related papers  (2021-05-05T17:56:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.