Helping Customers in Distress: An LLM-powered Agent that Converses, Probes, and Routes

• URL: http://arxiv.org/abs/2605.16268v1
• Date: Tue, 31 Mar 2026 15:07:20 GMT
• Title: Helping Customers in Distress: An LLM-powered Agent that Converses, Probes, and Routes
• Abstract summary: Banks receive millions of reports of fraud, scams, and disputed transactions every year.<n>To address this, we develop a customer-facing AI powered triaging agent.<n>It conducts multi-turn conversations, ask relevant questions, and classify cases for accurate, policy-guided routing.
• Score: 0.8738446148742723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Banks receive millions of reports of fraud, scams, and disputed transactions every year, making it challenging to accurately direct customers to the appropriate specialist teams for assistance. The existing manual process driven by humans is slow and stressful for both customers and staff. To address this, we develop a customer-facing AI powered triaging agent that leverages large language models (LLMs) to conduct multi-turn conversations, ask relevant questions, and classify cases for accurate, policy-guided routing, making it embedded in the customer journey. To evaluate and continuously improve the agent, synthetic digital twins of real customers were simulated, generating realistic, labelled dialogues based on historical data to test a wide range of real-world scenarios. This work details the triage agent's modelling approach, integration with policy, safety guardrails and reasoning frameworks, the use of the synthetic agent for scalable evaluation, and findings on the AI system's accuracy, robustness, and compliance. Results show that the agent successfully improves triaging of historical cases, achieving a 30.6% increase in classification accuracy, with high satisfaction levels reported by our subject-matter experts, highlighting how targeted probing can lead to more effective triage in banking operations at scale.

Related papers

• Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework [2.29541210878158]
  We present a unified framework that bridges offline development with online impact for customer support AI agents at Nubank.<n>A central insight is that evaluation-pipeline quality directly determines iteration velocity.<n>On most use cases, AI satisfaction reaches within a few percentage points of expert human agents.
  arXiv  Detail & Related papers  (2026-06-07T22:44:00Z)
• Large-Scale ChatBot Validation Through Customer Digital Twin Simulations [2.163639266428698]
  We introduce a methodology for creating high-fidelity synthetic customer agents (SCAs) as digital twins.<n>We develop an SCA-based validation framework combining automated LLM-as-a-Judge evaluation, human expert testing, and adversarial probing.
  arXiv  Detail & Related papers  (2026-05-15T15:40:11Z)
• ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning [88.87305027895657]
  This work investigates Reinforcement Learning-based optimization for shopping agents in real-world scenarios.<n>Agents must simultaneously satisfy multiple interdependent objectives spanning objective metrics (product correctness), subjective qualities (persuasiveness), outcome rewards (final response quality), and process rewards (tool efficiency)<n>Experiments demonstrate that our RL-trained agent, ChatShopBuddy, consistently outperforms larger models relying on generic reasoning.
  arXiv  Detail & Related papers  (2026-03-06T09:18:51Z)
• From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistants [0.0]
  Building reliable conversational AI assistants for customer-facing industries remains challenging due to noisy conversational data, fragmented knowledge, and the requirement for accurate human hand-off.<n>This paper presents an end-to-end framework for constructing and evaluating a conversational AI assistant directly from historical call transcripts.
  arXiv  Detail & Related papers  (2026-01-26T07:44:47Z)
• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress [71.02263260394261]
  Large language models (LLMs) still encounter challenges in multi-turn decision-making tasks.<n>We build process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.<n>AgentPRM captures both the interdependence between sequential decisions and their contribution to the final goal.
  arXiv  Detail & Related papers  (2025-11-11T14:57:54Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• How can we assess human-agent interactions? Case studies in software agent design [52.953425368394306]
  We make two major steps towards the rigorous assessment of human-agent interactions.<n>We propose PULSE, a framework for more efficient human-centric evaluation of agent designs.<n>We deploy the framework on a large-scale web platform built around the open-source software agent OpenHands.
  arXiv  Detail & Related papers  (2025-10-10T19:04:28Z)
• MADS: Multi-Agent Dialogue Simulation for Diverse Persuasion Data Generation [10.585352489359684]
  We propose MADS (Multi-Agent Dialogue Simulation), a scalable framework for generating persuasive multi-turn dialogues via agent self-play.<n> MADS employs three coordinated agents: User Agents designed to simulate diverse persona-driven behaviors by leveraging personality signifiers such as Zodiac Signs and MBTI types.<n>We further validate its effectiveness through users' Chain-of-Attitude (CoA) modeling and dedicated LLMs' persuasion assessment.
  arXiv  Detail & Related papers  (2025-09-30T06:55:39Z)
• Co-Investigator AI: The Rise of Agentic AI for Smarter, Trustworthy AML Compliance Narratives [2.7295959384567356]
  Co-Investigator AI is an agentic framework optimized to produce Suspicious Activity Reports (SARs) significantly faster and with greater accuracy than traditional methods.<n>We demonstrate its ability to streamline SAR drafting, align narratives with regulatory expectations, and enable compliance teams to focus on higher-order analytical work.
  arXiv  Detail & Related papers  (2025-09-10T08:16:04Z)
• How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $τ$-bench [58.114899897566964]
  In a multi-turn conversational environment, large language models (LLMs) often struggle with consistent reasoning and adherence to domain-specific policies.<n>We propose the Input-Reformulation Multi-Agent (IRMA) framework, which automatically reformulates user queries augmented with relevant domain rules.<n>IRMA significantly outperforms ReAct, Function Calling, and Self-Reflection by 16.1%, 12.7%, and 19.1%, respectively.
  arXiv  Detail & Related papers  (2025-08-28T15:57:33Z)
• CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments [90.29937153770835]
  We introduce CRMArena, a benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments.<n>We show that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting, and less than 55% even with function-calling abilities.<n>Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments.
  arXiv  Detail & Related papers  (2024-11-04T17:30:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.