PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines

• URL: http://arxiv.org/abs/2605.18812v1
• Date: Tue, 12 May 2026 08:23:12 GMT
• Title: PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines
• Abstract summary: We present PASC (Pipeline-Aware Split Conformal), which reduces multi-stage joint coverage to a single scalar conformal prediction problem.<n>On a three-stage NER -> NED -> entity-typing pipeline over CoNLL-2003, PASC achieves 96.4% end-to-end coverage versus 93.4% for Bonferroni and 86.5% for independent CP.
• Score: 0.11280931253550518
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern NLP and LLM systems are pipelines: named entity recognition (NER) -> entity disambiguation (NED) -> entity typing, retrieval-augmented generation (retriever -> reader), and agentic chains of planner -> tool -> critic. Errors compound across stages, but existing uncertainty quantification methods either calibrate each stage independently (no joint coverage) or apply a Bonferroni union bound (joint coverage, but conservative). We present PASC (Pipeline-Aware Split Conformal), which reduces multi-stage joint coverage to a single scalar conformal prediction problem on the joint maximum nonconformity score. PASC provides a finite-sample distribution-free guarantee that all K stages are simultaneously covered with probability at least 1 - alpha, and is nearly tight up to a 1/(n+1) factor. On a three-stage NER -> NED -> entity-typing pipeline over CoNLL-2003, PASC achieves 96.4% end-to-end coverage versus 93.4% for Bonferroni and 86.5% for independent CP, at identical average prediction set size (1.083). Under distribution shift to WNUT-17 Twitter and WikiNEuRal Wikipedia data, PASC empirically maintains the target coverage in the tested shift settings while independent CP collapses to 59%. PASC requires a single quantile computation, runs 1.7x faster than Bonferroni, and scales to K = 6 stages where independent CP drops to 0.53 end-to-end coverage. The same joint-maximum-score reduction applies directly to compound LLM systems and agent pipelines.

Related papers

• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Metric Aggregation Divergence: A Hidden Validity Threat in Agent-Based Policy Optimization and a Contractual Remedy [7.1976264551575895]
  Metric aggregation divergence (MAD) is the silent inconsistency that arises when distinct pipeline stages in an agent-based model.<n>We introduce the metric contract - a single callable runtime enforced at dispatch time - as the remedy.
  arXiv  Detail & Related papers  (2026-06-27T18:17:33Z)
• Report the Floor: A Training-Free Conformal Interval Is a Mandatory Baseline for Probabilistic Time-Series Forecasting [0.0]
  We show that the simplest possible conformal interval - a last-value point forecast wrapped in a finite-sample split-conformal residual quantile - is a far stronger baseline than its near-total absence.<n>This ConformalNaive interval decisively beats the naive value-quantile baselines.<n>We give ConformalNaive+, a one-line, training-free, horizon-adaptive selector that attains the better of two complementary floors at every horizon with restored coverage.
  arXiv  Detail & Related papers  (2026-06-08T13:30:18Z)
• Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa [0.0]
  PII detection systems are frequently trained within narrow source or domain boundaries, limiting coverage when deployed on heterogeneous text.<n>We study model fine-tuning on a corrected multi-source PIIBench preparation spanning 82 retained entity types across ten source datasets.<n>We evaluate three DeBERTa-based approaches: direct token classification fine-tuning, a source-conditioned hierarchical model (SC+H), and a three-phase curriculum extension (SC+H+Curr)
  arXiv  Detail & Related papers  (2026-05-25T13:12:42Z)
• Decoupled Conformal Optimisation: Efficient Prediction Sets via Independent Tuning and Calibration [20.21455697379946]
  We propose Decoupled Conformal optimisation (DCO) as a train-tune-calibrate design principle.<n>DCO uses an independent tuning split for efficiency-oriented structural selection and a fresh calibration split for the final conformal quantile.<n>We show that DCO tracks the nominal coverage level closely while often reducing average prediction-set size or interval width relative to PAC-style calibration.
  arXiv  Detail & Related papers  (2026-05-18T13:10:21Z)
• A Penalty-Free Pipeline for Direct Quantum-Annealer Portfolio Optimization [0.0]
  Direct quantum-annealer portfolio optimization is commonly formulated as a penalty-encoded QUBO submitted to D-Wave hardware.<n>We show that this standard formulation fails on current devices and identify the structural reason.<n>We propose dropping the penalty entirely: build an objective-only QUBO from the expected returns and sample it on hardware.
  arXiv  Detail & Related papers  (2026-05-17T19:50:04Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Physics-Informed Causal MDPs for Sequential Constraint Repair in Engineering Simulation Pipelines [0.0]
  We introduce PI-CMDP, a framework for CMDPs whose constraint dependencies form a layered DAG.<n>We instantiate PI-CMDP on constraint repair in engineering simulation pipelines.
  arXiv  Detail & Related papers  (2026-04-20T07:40:15Z)
• The Autocorrelation Blind Spot: Why 42% of Turn-Level Findings in LLM Conversation Analysis May Be Spurious [35.76482964927589]
  We characterize the autocorrelation structure of 66 turn-level metrics across 202 multi-turn conversations.<n>We show that 42% of associations that appear significant under standard pooled testing fail to survive cluster-robust correction.<n>We present a two-stage correction framework combining Chelton (1983) effective degrees of freedom with conversation-level block bootstraps.
  arXiv  Detail & Related papers  (2026-04-15T20:54:39Z)
• Token Coherence: Adapting MESI Cache Protocols to Minimize Synchronization Overhead in Multi-Agent LLM Systems [0.0]
  Multi-agent LLM orchestration incurs synchronization costs scaling as O(n x S x |D|) in agents, steps, and artifact size under naive broadcast.<n>I argue this pathology is a structural residue of full-state rebroadcast, not an inherent property of multi-agent coordination.<n>I construct the Artifact Coherence System (ACS) and prove the Token Coherence Theorem.
  arXiv  Detail & Related papers  (2026-03-16T12:20:06Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training [54.8494905524997]
  Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes.<n>We propose SENTINEL, a verification mechanism for pipeline parallelism (PP) training without duplication.<n>Experiments demonstrate successful training of up to 4B- parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
  arXiv  Detail & Related papers  (2026-03-03T23:51:10Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• Robust Conformal Prediction with a Single Binary Certificate [58.450154976190795]
  Conformal prediction (CP) converts any model's output to prediction sets with a guarantee to cover the true label with (adjustable) high probability.<n>We propose a robust conformal prediction that produces smaller sets even with significantly lower MC samples.
  arXiv  Detail & Related papers  (2025-03-07T08:41:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.