Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents

• URL: http://arxiv.org/abs/2607.18039v1
• Date: Mon, 20 Jul 2026 15:07:58 GMT
• Title: Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents
• Abstract summary: BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded evidence for controlled LLM decisions.<n>Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and forgetting state inside a fixed LangGraph DAGcitelanggraph2024.<n>The paper contributes three reusable deployment patterns: textbfhybrid RAG evidence construction, textbfevidence-grounded issue/action decision, and textbftrace-driven RAG and
• Score: 6.9430026014199635
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.

Related papers

• ASTRA - Agentic System for Ticket Resolution and Analysis [3.0724026668543174]
  ASTRA is an agentic system for ticket resolution in which a central orchestrator coordinates three specialist information-gathering agents.<n>TicketSimilarityAgent retrieves relevant historical precedents through dense retrieval and LLM reranking.<n>LogAgent distills hundreds of thousands of log lines into structured, quote-grounded findings.<n> DomainKnowledgeAgent retrieves relevant technical knowledge via the Model Context Protocol.
  arXiv  Detail & Related papers  (2026-08-28T18:50:17Z)
• SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification [71.9783246097687]
  SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence.<n>For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation.<n>The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models.
  arXiv  Detail & Related papers  (2026-08-04T13:16:15Z)
• From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents [0.24554686192257424]
  Productization adds requirements for source boundaries, entity routing, answer contracts, and reproducible traces.<n>We present a harness-engineering approach that reconstructs this pattern into a traceable, auditable LLM-agent architecture.<n>We instantiate it on a public-data slice of five Korean corporate groups and evaluate three research questions.
  arXiv  Detail & Related papers  (2026-07-09T01:08:33Z)
• MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG [50.46218163777298]
  Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions.<n>We propose MMAgent-R$2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism.
  arXiv  Detail & Related papers  (2026-07-08T13:15:39Z)
• What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs [52.50210189669399]
  Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs)<n>We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation.<n>We show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods.
  arXiv  Detail & Related papers  (2026-06-01T03:18:01Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• PAVE: Premise-Aware Validation and Editing for Retrieval-Augmented LLMs [8.082352336629816]
  PAVE is an inference-time validation layer for evidence-grounded question answering.<n>It decomposes retrieved context into question-conditioned atomic facts, drafts an answer, scores how well that draft is supported by the extracted premises, and revises low-support outputs before finalization.
  arXiv  Detail & Related papers  (2026-03-21T06:23:21Z)
• Orchestrating Specialized Agents for Trustworthy Enterprise RAG [8.772844442593975]
  One-pass retrieval-and-write pipelines often yield shallow summaries.<n>We introduce ADORE, an agentic framework that replaces linear retrieval with iterative, user-steered investigation.<n>Our contributions are threefold: Memory-locked synthesis, Evidence-coverage-guided execution, and section-packed long-context grounding.
  arXiv  Detail & Related papers  (2026-01-26T08:48:41Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• ORCHID: Orchestrated Retrieval-Augmented Classification with Human-in-the-Loop Intelligent Decision-Making for High-Risk Property [6.643427585499247]
  ORCHID is a modular agentic system for HRP classification.<n>It pairs retrieval-augmented generation (RAG) with human oversight to produce policy-based outputs that can be audited.<n>The demonstration shows single item submission, grounded citations, SME feedback capture, and exportable audit artifacts.
  arXiv  Detail & Related papers  (2025-11-07T03:48:05Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.<n>In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources.<n>We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.