Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants

• URL: http://arxiv.org/abs/2502.00537v1
• Date: Sat, 01 Feb 2025 19:23:21 GMT
• Title: Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants
• Authors: Md Mehrab Tanjim, Xiang Chen, Victor S. Bursztyn, Uttaran Bhattacharya, Tung Mai, Vaishnavi Muppala, Akash Maharaj, Saayan Mitra, Eunyee Koh, Yunyao Li, Ken Russell,
• Abstract summary: We propose an NLU-NLG framework for ambiguity detection and resolution through reformulating query automatically. We develop a taxonomy based on real user conversational logs and draw insights from it to design rules and extract features for a classifier. This has been deployed in the real world application, namely Adobe Experience Platform AI Assistant.
• Score: 22.24244100928786
• License:
• Abstract: Multi-turn conversations with an Enterprise AI Assistant can be challenging due to conversational dependencies in questions, leading to ambiguities and errors. To address this, we propose an NLU-NLG framework for ambiguity detection and resolution through reformulating query automatically and introduce a new task called "Ambiguity-guided Query Rewrite." To detect ambiguities, we develop a taxonomy based on real user conversational logs and draw insights from it to design rules and extract features for a classifier which yields superior performance in detecting ambiguous queries, outperforming LLM-based baselines. Furthermore, coupling the query rewrite module with our ambiguity detecting classifier shows that this end-to-end framework can effectively mitigate ambiguities without risking unnecessary insertions of unwanted phrases for clear queries, leading to an improvement in the overall performance of the AI Assistant. Due to its significance, this has been deployed in the real world application, namely Adobe Experience Platform AI Assistant.

Related papers

• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions. Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes. We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• Improving Retrieval in Sponsored Search by Leveraging Query Context Signals [6.152499434499752]
  We propose an approach to enhance query understanding by augmenting queries with rich contextual signals. We use web search titles and snippets to ground queries in real-world information and utilize GPT-4 to generate query rewrites and explanations. Our context-aware approach substantially outperforms context-free models.
  arXiv  Detail & Related papers  (2024-07-19T14:28:53Z)
• AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries [56.82807063333088]
  We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-open programs. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness) In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch.
  arXiv  Detail & Related papers  (2024-06-27T10:43:04Z)
• AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents [74.17623527375241]
  We introduce a novel framework, called AutoGuide, which automatically generates context-aware guidelines from offline experiences. As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current decision-making process. Our evaluation demonstrates that AutoGuide significantly outperforms competitive baselines in complex benchmark domains.
  arXiv  Detail & Related papers  (2024-03-13T22:06:03Z)
• DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton [44.26173742405563]
  This paper introduces the retrieval-augmented large language model with Definite Finite Automaton (DFA-RAG) DFA-RAG is a framework designed to enhance the capabilities of conversational agents using large language models (LLMs)
  arXiv  Detail & Related papers  (2024-02-06T21:14:45Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• Building Interpretable and Reliable Open Information Retriever for New Domains Overnight [67.03842581848299]
  Information retrieval is a critical component for many down-stream tasks such as open-domain question answering (QA) We propose an information retrieval pipeline that uses entity/event linking model and query decomposition model to focus more accurately on different information units of the query. We show that, while being more interpretable and reliable, our proposed pipeline significantly improves passage coverages and denotation accuracies across five IR and QA benchmarks.
  arXiv  Detail & Related papers  (2023-08-09T07:47:17Z)
• ZeQR: Zero-shot Query Reformulation for Conversational Search [11.644235288057123]
  We introduce a novel Zero-shot Query Reformulation (or Query Rewriting) framework that reformulates queries based on previous dialogue contexts without requiring supervision from conversational search data. Specifically, our framework utilizes language models designed for machine reading comprehension tasks to explicitly resolve two common ambiguities: coreference and omission, in raw queries. It also provides greater explainability and effectively enhances query intent understanding because ambiguities are explicitly and proactively resolved.
  arXiv  Detail & Related papers  (2023-07-18T16:05:25Z)
• On the Impact of Speech Recognition Errors in Passage Retrieval for Spoken Question Answering [13.013751306590303]
  We study the robustness of lexical and dense retrievers against questions with synthetic ASR noise. We create a new dataset with questions voiced by human users and use their transcriptions to show that the retrieval performance can further degrade when dealing with natural ASR noise instead of synthetic ASR noise.
  arXiv  Detail & Related papers  (2022-09-26T18:29:36Z)
• Personalized Query Rewriting in Conversational AI Agents [7.086654234990377]
  We propose a query rewriting approach by leveraging users' historically successful interactions as a form of memory. We present a neural retrieval model and a pointer-generator network with hierarchical attention and show that they perform significantly better at the query rewriting task with the aforementioned user memories than without.
  arXiv  Detail & Related papers  (2020-11-09T20:45:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.