Compliance-Scored Best-of-N Guardrail Orchestration for Multimodal Document Generation in Payments Dispute Defense

• URL: http://arxiv.org/abs/2606.01513v1
• Date: Mon, 01 Jun 2026 00:24:30 GMT
• Title: Compliance-Scored Best-of-N Guardrail Orchestration for Multimodal Document Generation in Payments Dispute Defense
• Abstract summary: High-stakes enterprise document generation includes financial dispute narratives, compliance notices, and audit summaries.<n>Prior to a unified guardrail layer, production systems often stitched together separate PII redaction, content moderation, and format validation steps.<n>We present a guardrail orchestration layer for text and image inputs that couples multi-candidate generation with an explicit compliance score used for early exit.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-stakes enterprise document generation, including financial dispute narratives, compliance notices, and audit summaries, demands schema correctness, policy compliance, and low-latency operation at scale. Prior to a unified guardrail layer, production systems often stitched together separate PII redaction, content moderation, and format validation steps, leading to fragmented logic, slower request paths, and higher operational cost. We present a guardrail orchestration layer for text and image inputs that couples multi-candidate generation with an explicit compliance score used for early exit. The framework runs configurable parallel generation heads, scores candidates against weighted guardrails including PII detection, content moderation, schema constraints, and domain rules, and returns the best-scoring output with selection metadata. The available operational readout reports 5 attempts within 20 seconds and 91 percent compliance. For payments dispute defense summaries, we analyze aggregate operational scenario readouts rather than a randomized A/B test. Variable cohorts show higher count win rates than controls overall, 301/659 versus 536/1548, corresponding to +11.0 percentage points with 95 percent confidence interval [6.6, 15.5] and p < 0.001, and for adjusted item-not-received cases, +7.5 percentage points with 95 percent confidence interval [0.2, 15.7] and p = 0.045. Fraud and local evidence-ranking deltas are directionally positive but not statistically significant from the aggregate count data. We also report reviewer-calibrated Responsible-AI evidence-quality signals from 770 generated-evidence reviews and a 70-case OCR slice, and document the reproducibility boundary through the request interface, scoring logic, pseudocode, and operational evidence boundary.

Related papers

• When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor [0.0]
  We present a case study of one such agent implementing a multi-component data system against a detailed pre-existing specification.<n>We catalog five such defects, categorized by constraint violated and detection method.<n>Across retrieval budgets from 1 to 10 and 100 questions against a pooled corpus of 2994 paragraphs, filtered recall reaches its ceiling by a budget of 3, expected once candidates are restricted to the gold paragraphs themselves.
  arXiv  Detail & Related papers  (2026-09-02T01:43:09Z)
• Counterfactual Bias Testing for Application Tracking System [1.007765066765576]
  This paper uses task-specialized LLM agents to synthesize identity-neutral base resumes and inject demographic treatments.<n>It ranks candidates against a job description via a fine-tuned sentence-embedding model and cosine similarity.<n>It computes a nine-metric fairness suite spanning counterfactual (score delta, mean absolute rank change, flip rate), group-fairness (top-K retention, four-fifths/impact ratio), and merit-aware (Recall@K, nDCG@K, equal opportunity, equalized odds) families.
  arXiv  Detail & Related papers  (2026-08-27T10:01:36Z)
• Grounded Normative Rule Generation with Structured Search [47.28595196084396]
  We formalize the problem as Grounded Normative Rule Synthesis (GNRS)<n>We introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG)<n>We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharterBench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses.
  arXiv  Detail & Related papers  (2026-08-23T05:43:42Z)
• Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text [3.2712318205295907]
  Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in qualitatively different ways.<n>We study this problem as fine-grained inconsistency.<n>A fine-tuned 300M encoder reaches 61.9% accuracy, compared with 61.5% for a LoRA-adapted Qwen3.5-9B model and 61.3% for GPT-adapted generative models.
  arXiv  Detail & Related papers  (2026-07-29T01:03:53Z)
• ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents [4.018837636907658]
  We introduce a sandboxed, trace-based benchmark that measures benchmark-defined endpoint policy compliance.<n>We find that aggregate logged-spread rankings change with evidence-stage composition and denominator choice.<n>These results show that a terminal policy label is not a sufficient statistic for operational post-exposure containment.
  arXiv  Detail & Related papers  (2026-07-27T04:51:20Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• Hierarchical Online Prompt Mutation with Dual-Loop Feedback for Guardrailed Evidence Document Generation: A Production-Evaluation Case Study [0.0]
  HOPM is a hierarchical online prompt mutation framework evaluated on a real marketplace dispute-evidence workflow.<n>The paper includes control setup, sample sizes, confidence intervals, paired tests, prompt-token categories, pseudocode, schema, rubric, guardrail taxonomy, and a constructed example.
  arXiv  Detail & Related papers  (2026-05-31T22:17:44Z)
• Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks [0.8699677835130409]
  Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points.<n>We present a repeated-run evaluation protocol with metrics for run-level accuracy, retry-free coverage, and per-problem variability.
  arXiv  Detail & Related papers  (2026-05-30T23:03:05Z)
• TypedCSIP: Typed Counterfactual Pretraining for Chinese Legislative Conflict Classification [3.5137191090796054]
  TypedCSIP is a typed counterfactual pretraining method for the conflict-classification task of the LCR-CN benchmark.<n>We exploit LCR-CN's expert-written minimal revisions as training-time counterfactual supervision.
  arXiv  Detail & Related papers  (2026-05-25T06:26:46Z)
• Privacy Policy Enforcement Guardrails for Data-Sensitive Retrieval-Augmented Generation [40.44980598761587]
  PII filters often miss contextual data leakage in RAG systems, such as non-regulated attribute clusters that collectively identify individuals.<n>We introduce a Privacy Policy Enforcement framework using dual one-class density estimators with fused text embeddings and a abstain region for out-of-distribution inputs.<n>Our proposed T3+OCSVM detector, trained on safe and borderline-safe data, achieves a borderline AUROC of 0.93+ while reducing false positives by 44-55 percentage points and maintaining millisecond latency.
  arXiv  Detail & Related papers  (2026-05-16T15:08:31Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt Engineering Quality Assurance [0.0]
  This paper reports a single-system empirical case study of agent-driven auditing applied to AEGIS.<n>We report a seven-category post-hoc defect taxonomy with explicit coding rules, observed non-monotonic convergence, and audit-scope expansion.
  arXiv  Detail & Related papers  (2026-05-12T15:39:04Z)
• Evaluating AI Meeting Summaries with a Reusable Cross-Domain Pipeline [2.4832413743954618]
  We present a reusable evaluation pipeline for generative AI applications, instantiated for AI meeting summaries.<n>The system separates reusable orchestration from task-specific semantics across five stages: source intake, structured reference construction, candidate generation, structured scoring, and reporting.<n>We benchmark the offline loop on a typed dataset of 114 meetings spanning city_council, private_data, and whitehouse_press_briefings.
  arXiv  Detail & Related papers  (2026-04-23T07:02:11Z)
• Evaluating Prompt Injection Defenses for Educational LLM Tutors: Security-Usability-Latency Trade-offs [51.56484100374058]
  guardrail design entails explicit trade-offs among adversarial robustness, benign-task usability, and response latency.<n>We evaluate a domain-specific multi-layer safeguard pipeline combining deterministic pattern filters, structural validation, contextual sandboxing, and session-level behavioral checks.<n>Results expose operational trade-offs: NeMo reaches 0 percent bypass at 16.22 percent FPR and roughly 1.5s latency, while Prompt Guard yields 38.48 percent bypass with 3.60 percent FPR.
  arXiv  Detail & Related papers  (2026-03-29T18:52:01Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Scaling Agentic Verifier for Competitive Coding [66.11758166379092]
  Large language models (LLMs) have demonstrated strong coding capabilities but still struggle to solve competitive programming problems correctly in a single attempt.<n>Execution-based re-ranking offers a promising test-time scaling strategy, yet existing methods are constrained by either difficult test case generation or inefficient random input sampling.<n>We propose Agentic Verifier, an execution-based agent that actively reasons about program behaviors and searches for highly discriminative test inputs.
  arXiv  Detail & Related papers  (2026-02-04T06:30:40Z)
• INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection [40.157413213893165]
  Administrative phone tasks drain roughly 1 trillion USD annually from U.S. healthcare.<n>We introduce INSURE-Dial, the first public benchmark for developing and assessing compliance-aware voice agents.
  arXiv  Detail & Related papers  (2026-01-28T17:27:49Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking [57.42241521034744]
  We propose the concept of certified error control of candidate set pruning for relevance ranking. Our method successfully prunes the first-stage retrieved candidate sets to improve the second-stage reranking speed.
  arXiv  Detail & Related papers  (2022-05-19T16:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.