Related papers: Comprehensive Framework for Evaluating Conversational AI Chatbots

Related papers

Herculean: An Agentic Benchmark for Financial Intelligence [114.33879935277217]
We introduce Herculean, the first skilled benchmark for agentic financial intelligence spanning four representative skill sets.<n>Across frontier agents, we find agents perform relatively well on Trading and Market Insights, but struggle substantially on Hedging and Auditing.<n>Overall, our results point to a key gap in current agents in turning financial reasoning into dependable workflow execution.
arXiv Detail & Related papers (2026-05-14T04:30:49Z)
What if AI systems weren't chatbots? [8.421028777254135]
We show how treating AI primarily as conversational assistants has extensive structural downsides.<n>We examine broader societal effects, including labor displacement, concentration of economic power, and increased environmental costs.
arXiv Detail & Related papers (2026-05-08T15:40:18Z)
Reconsidering Conversational Norms in LLM Chatbots for Sustainable AI [2.198430261120653]
We argue that interaction level behavior appears to be an underexamined factor shaping the environmental impact of LLM based systems.<n>We show how extended conversational patterns increase token production and raise the computational cost of inference.<n>We argue that everyday user habits contribute to cumulative operational demand in ways that are rarely quantified.
arXiv Detail & Related papers (2025-12-16T18:38:11Z)
Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents [63.03252293761656]
This paper systematically reviews the technologies, applications, and evaluation methods of industry agents based on large language models (LLMs)<n>We examine the three key technological pillars that support the advancement of agent capabilities: Memory, Planning, and Tool Use.<n>We provide an overview of the application of industry agents in real-world domains such as digital engineering, scientific discovery, embodied intelligence, collaborative business execution, and complex system simulation.
arXiv Detail & Related papers (2025-10-20T12:46:55Z)
Taxonomy of User Needs and Actions [51.86289485979439]
The growing ubiquity of conversational AI highlights the need for frameworks that capture users' instrumental goals and the situated, adaptive, and social practices through which they achieve them.<n>To address this gap, we introduce the Taxonomy of User Needs and Actions (TUNA), an empirically grounded framework developed through iterative qualitative analysis of 1193 human-AI conversations.<n> TUNA organizes user actions into a three-level hierarchy encompassing behaviors associated with information seeking, synthesis, procedural guidance, content creation, social interaction, and meta-conversation.
arXiv Detail & Related papers (2025-10-07T17:04:42Z)
Exploring Big Five Personality and AI Capability Effects in LLM-Simulated Negotiation Dialogues [16.07828032939124]
This paper presents an evaluation framework for agentic AI systems in mission-critical negotiation contexts.<n>Using Sotopia as a simulation testbed, we present two experiments that systematically evaluated how personality traits and AI agent characteristics influence social negotiation outcomes.
arXiv Detail & Related papers (2025-06-19T00:14:56Z)
The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
arXiv Detail & Related papers (2025-06-09T18:37:14Z)
A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions [51.96890647837277]
Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users. This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence.
arXiv Detail & Related papers (2025-04-07T21:01:25Z)
AgentOrca: A Dual-System Framework to Evaluate Language Agents on Operational Routine and Constraint Adherence [54.317522790545304]
We present AgentOrca, a dual-system framework for evaluating language agents' compliance with operational constraints and routines. Our framework encodes action constraints and routines through both natural language prompts for agents and corresponding executable code serving as ground truth for automated verification. Our findings reveal notable performance gaps among state-of-the-art models, with large reasoning models like o1 demonstrating superior compliance while others show significantly lower performance.
arXiv Detail & Related papers (2025-03-11T17:53:02Z)
Conversation Routines: A Prompt Engineering Framework for Task-Oriented Dialog Systems [0.21756081703275998]
This study introduces Conversation Routines (CR), a structured prompt engineering framework for developing task-oriented dialog systems using Large Language Models (LLMs)<n>The proposed CR framework enables the development of Conversation Agentic Systems (CAS) through natural language specifications.<n>We demonstrate the framework's effectiveness through two proof-of-concept implementations: a Train Booking System and an Interactive Ticket Copilot.
arXiv Detail & Related papers (2025-01-20T17:19:02Z)
Modular Conversational Agents for Surveys and Interviews [6.019313905775819]
This paper introduces a modular approach and its resulting parameterized process for designing AI agents. We demonstrate the adaptability, generalizability, and efficacy of our modular approach through three empirical studies. The results suggest that the AI agent increases completion rates and response quality.
arXiv Detail & Related papers (2024-12-22T15:00:16Z)
Augmenting Compliance-Guaranteed Customer Service Chatbots: Context-Aware Knowledge Expansion with Large Language Models [12.934175064411845]
Retrieval-based chatbots leverage human-verified Q&A knowledge to deliver accurate, verifiable responses.<n>To effectively handle diverse customer inquiries, augmenting the knowledge base with "similar questions" that retain semantic meaning is a cost-effective strategy.<n>We propose context-aware approaches to enable comprehensive semantic exploration and enhanced alignment with source question-answer relationships.
arXiv Detail & Related papers (2024-10-16T10:48:14Z)
CA-BERT: Leveraging Context Awareness for Enhanced Multi-Turn Chat Interaction [2.3178408584843906]
This paper introduces Context-Aware BERT (CA-BERT), a transformer-based model specifically fine-tuned to address this challenge. We describe the development of CA-BERT, which adapts the robust architecture of BERT with a novel training regimen focused on a specialized dataset of chat dialogues. The model is evaluated on its ability to classify context necessity, demonstrating superior performance over baseline BERT models in terms of accuracy and efficiency.
arXiv Detail & Related papers (2024-09-05T06:27:59Z)
Expanding the Set of Pragmatic Considerations in Conversational AI [0.26206189324400636]
We discuss several pragmatic limitations of current conversational AI systems. We label our complaints as "Turing Test Triggers" (TTTs) We develop a taxonomy of pragmatic considerations intended to identify what pragmatic competencies a conversational AI system requires.
arXiv Detail & Related papers (2023-10-27T19:21:50Z)
IMTLab: An Open-Source Platform for Building, Evaluating, and Diagnosing Interactive Machine Translation Systems [94.39110258587887]
We present IMTLab, an open-source end-to-end interactive machine translation (IMT) system platform. IMTLab treats the whole interactive translation process as a task-oriented dialogue with a human-in-the-loop setting.
arXiv Detail & Related papers (2023-10-17T11:29:04Z)
A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects [100.75759050696355]
We provide a comprehensive overview of the prominent problems and advanced designs for conversational agent's proactivity in different types of dialogues. We discuss challenges that meet the real-world application needs but require a greater research focus in the future.
arXiv Detail & Related papers (2023-05-04T11:38:49Z)
Approximating Online Human Evaluation of Social Chatbots with Prompting [11.657633779338724]
Existing evaluation metrics aim to automate offline user evaluation and approximate human judgment of pre-curated dialogs. We propose an approach to approximate online human evaluation leveraging large language models (LLMs) from the GPT family. We introduce a new Dialog system Evaluation framework based on Prompting (DEP), which enables a fully automatic evaluation pipeline.
arXiv Detail & Related papers (2023-04-11T14:45:01Z)
Intent Recognition in Conversational Recommender Systems [0.0]
We introduce a pipeline to contextualize the input utterances in conversations. We then take the next step towards leveraging reverse feature engineering to link the contextualized input and learning model to support intent recognition.
arXiv Detail & Related papers (2022-12-06T11:02:42Z)
Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
arXiv Detail & Related papers (2022-09-07T10:09:12Z)
Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy [83.61404191470126]
We propose a new solution named I-Pro that can learn Proactive policy in the Interactive setting. Specifically, we learn the trade-off via a learned goal weight, which consists of four factors. The experimental results demonstrate I-Pro significantly outperforms baselines in terms of effectiveness and interpretability.
arXiv Detail & Related papers (2022-04-07T14:11:31Z)
Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue [48.75221685739286]
We present a novel modular dialogue system framework that seamlessly integrates factual information and social content into persuasive dialogue. Our framework is generalizable to any dialogue tasks that have mixed social and task contents.
arXiv Detail & Related papers (2022-03-15T05:38:34Z)
You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
arXiv Detail & Related papers (2020-04-11T12:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.