LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems

• URL: http://arxiv.org/abs/2407.12016v1
• Date: Thu, 27 Jun 2024 06:54:53 GMT
• Title: LLM-based Frameworks for API Argument Filling in Task-Oriented Conversational Systems
• Authors: Jisoo Mok, Mohammad Kachuee, Shuyang Dai, Shayan Ray, Tara Taghavi, Sungroh Yoon,
• Abstract summary: We study the application of Large Language Models (LLMs) for the problem of API argument filling task. Our experimental results demonstrate that when paired with proposed techniques, the argument filling performance of LLMs noticeably improves.
• Score: 39.54340884416936
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Task-orientated conversational agents interact with users and assist them via leveraging external APIs. A typical task-oriented conversational system can be broken down into three phases: external API selection, argument filling, and response generation. The focus of our work is the task of argument filling, which is in charge of accurately providing arguments required by the selected API. Upon comprehending the dialogue history and the pre-defined API schema, the argument filling task is expected to provide the external API with the necessary information to generate a desirable agent action. In this paper, we study the application of Large Language Models (LLMs) for the problem of API argument filling task. Our initial investigation reveals that LLMs require an additional grounding process to successfully perform argument filling, inspiring us to design training and prompting frameworks to ground their responses. Our experimental results demonstrate that when paired with proposed techniques, the argument filling performance of LLMs noticeably improves, paving a new way toward building an automated argument filling framework.

Related papers

• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents [7.166156709980112]
  We introduce textscShortcutsBench, a large-scale benchmark for the comprehensive evaluation of API-based agents. textscShortcutsBench includes a wealth of real APIs from Apple Inc.'s operating systems. Our evaluation reveals significant limitations in handling complex queries related to API selection, parameter filling, and requesting necessary information from systems and users.
  arXiv  Detail & Related papers  (2024-06-28T08:45:02Z)
• The Task-oriented Queries Benchmark (ToQB) [0.0]
  A standard benchmark for task-oriented queries is not yet available. Existing benchmarks in the relevant NLP fields have primarily focused on task-oriented dialogues.
  arXiv  Detail & Related papers  (2024-06-05T05:05:41Z)
• LLM+Reasoning+Planning for supporting incomplete user queries in presence of APIs [0.09374652839580183]
  In practice, natural language task requests (user queries) are often incomplete, i.e., they may not contain all the information required by the APIs. We leverage logical reasoning and classical AI planning along with an LLM for accurately answering user queries. Our approach achieves over 95% success rate in most cases on a dataset containing complete and incomplete single goal and multi-goal queries.
  arXiv  Detail & Related papers  (2024-05-21T01:16:34Z)
• Contextual API Completion for Unseen Repositories Using LLMs [6.518508607788089]
  We introduce a novel technique to mitigate hallucinations by leveraging global and local contextual information within a code repository for API completion tasks. Our approach is tailored to refine code completion tasks, with a focus on optimizing local API completions. Our tool, LANCE, surpasses Copilot by 143% and 142% for API token completion and conversational API completion, respectively.
  arXiv  Detail & Related papers  (2024-05-07T18:22:28Z)
• Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models [56.93074140619464]
  We propose RiC (Reasoning in Conversation), a method that focuses on solving subjective tasks through dialogue simulation. The motivation of RiC is to mine useful contextual information by simulating dialogues instead of supplying chain-of-thought style rationales. We evaluate both API-based and open-source LLMs including GPT-4, ChatGPT, and OpenChat across twelve tasks.
  arXiv  Detail & Related papers  (2024-02-27T05:37:10Z)
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations [70.7884839812069]
  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue.
  arXiv  Detail & Related papers  (2023-11-09T18:45:16Z)
• OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
  We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs. Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
  arXiv  Detail & Related papers  (2023-05-24T10:08:04Z)
• Frugal Prompting for Dialog Models [17.048111072193933]
  This study examines different approaches for building dialog systems using large language models (LLMs) As part of prompt tuning, we experiment with various ways of providing instructions, exemplars, current query and additional context. The research also analyzes the representations of dialog history that have the optimal usable-information density.
  arXiv  Detail & Related papers  (2023-05-24T09:06:49Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.