Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering

• URL: http://arxiv.org/abs/2402.14320v6
• Date: Sun, 29 Sep 2024 02:41:12 GMT
• Title: Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering
• Authors: Chang Zong, Yuchen Yan, Weiming Lu, Jian Shao, Eliot Huang, Heng Chang, Yueting Zhuang,
• Abstract summary: Triad is a unified framework that utilizes an LLM-based agent with three roles for KBQA tasks. Our framework is executed in four phases, involving the collaboration of the agent's multiple roles.
• Score: 42.277248862366164
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress with LLM-based agents has shown promising results across various tasks. However, their use in answering questions from knowledge bases remains largely unexplored. Implementing a KBQA system using traditional methods is challenging due to the shortage of task-specific training data and the complexity of creating task-focused model structures. In this paper, we present Triad, a unified framework that utilizes an LLM-based agent with three roles for KBQA tasks. The agent is assigned three roles to tackle different KBQA subtasks: agent as a generalist for mastering various subtasks, as a decision maker for the selection of candidates, and as an advisor for answering questions with knowledge. Our KBQA framework is executed in four phases, involving the collaboration of the agent's multiple roles. We evaluated the performance of our framework using three benchmark datasets, and the results show that our framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks, yielding F1 scores of 11.8% and 20.7%, respectively.

Related papers

• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• Beyond Chains: Bridging Large Language Models and Knowledge Bases in Complex Question Answering [28.898509577087516]
  Knowledge Base Question Answering (KBQA) aims to answer natural language questions using structured knowledge from KBs.<n>We propose PDRR: a four-stage framework consisting of Predict, Decompose, Retrieve, and Reason.<n>Our method first predicts the question type and decomposes the question into structured triples. Then retrieves relevant information from KBs and guides the LLM as an agent to reason over and complete the triples.
  arXiv  Detail & Related papers  (2025-05-20T09:01:52Z)
• HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems [1.1930434318557155]
  We introduce HALO, a multi-agent collaboration framework based on a hierarchical reasoning architecture.<n>Specifically, we incorporate a high-level planning agent for task decomposition, mid-level role-design agents for subtask-specific agent instantiation, and low-level inference agents for subtask execution.<n>As the majority of users lack expertise in prompt engineering, we leverage an Adaptive Prompt Refinement module to transform raw queries into task-specific prompts.
  arXiv  Detail & Related papers  (2025-05-17T04:14:03Z)
• Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks [11.125564622217892]
  Many methods for improving Large Language Model (LLM) agents for sequential decision-making tasks depend on task-specific knowledge engineering. We investigate how LLM agents can automatically improve their performance by learning in-context from their own successful experiences on similar tasks.
  arXiv  Detail & Related papers  (2025-05-01T00:48:12Z)
• Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts [67.67746334493302]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across numerous tasks, yet they often rely on external context to handle complex tasks. We propose a tri-encoder sequential retriever that models this process as a Markov Decision Process (MDP) We show that our method consistently and significantly outperforms baselines, underscoring the importance of explicitly modeling inter-example dependencies.
  arXiv  Detail & Related papers  (2025-04-15T17:35:56Z)
• MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents [59.825725526176655]
  Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents. Existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. We introduce MultiAgentBench, a benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios.
  arXiv  Detail & Related papers  (2025-03-03T05:18:50Z)
• Software Mention Recognition with a Three-Stage Framework Based on BERTology Models at SOMD 2024 [0.0]
  This paper describes our systems for the sub-task I in the Software Mention Detection in Scholarly Publications shared-task. Our best performing system addresses the named entity recognition problem through a three-stage framework. Our framework based on the XLM-R-based model achieves a weighted F1-score of 67.80%, delivering our team the 3rd rank in Sub-task I for the Software Mention Recognition task.
  arXiv  Detail & Related papers  (2024-04-23T17:06:24Z)
• Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models [7.399563588835834]
  Interactive-KBQA is a framework designed to generate logical forms through direct interaction with knowledge bases (KBs) Our method achieves competitive results on the WebQuestionsSP, ComplexWebQuestions, KQA Pro, and MetaQA datasets.
  arXiv  Detail & Related papers  (2024-02-23T06:32:18Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• Agents meet OKR: An Object and Key Results Driven Agent System with Hierarchical Self-Collaboration and Self-Evaluation [25.308341461293857]
  OKR-Agent is designed to enhance the capabilities of Large Language Models (LLMs) in task-solving. Our framework includes two novel modules: hierarchical Objects and Key Results generation and multi-level evaluation.
  arXiv  Detail & Related papers  (2023-11-28T06:16:30Z)
• Make a Choice! Knowledge Base Question Answering with In-Context Learning [1.7827767384590838]
  Question answering over knowledge bases (KBQA) aims to answer factoid questions with a given knowledge base (KB) Due to the large scale of KB, annotated data is impossible to cover all fact schemas in KB. We present McL-KBQA, a framework that incorporates the few-shot ability of LLM into the KBQA method via ICL-based multiple choice.
  arXiv  Detail & Related papers  (2023-05-23T11:56:03Z)
• Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL [107.58821842920393]
  We quantify the agent's behavior difference and build its relationship with the policy performance via bf Role Diversity We find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity. The decomposed factors can significantly impact policy optimization on three popular directions.
  arXiv  Detail & Related papers  (2022-06-01T04:58:52Z)
• SYGMA: System for Generalizable Modular Question Answering OverKnowledge Bases [57.89642289610301]
  We present SYGMA, a modular approach facilitating general-izability across multiple knowledge bases and multiple rea-soning types. We demonstrate effectiveness of our system by evaluating on datasets belonging to two distinct knowledge bases,DBpedia and Wikidata.
  arXiv  Detail & Related papers  (2021-09-28T01:57:56Z)
• A Survey on Complex Question Answering over Knowledge Base: Recent Advances and Challenges [71.4531144086568]
  Question Answering (QA) over Knowledge Base (KB) aims to automatically answer natural language questions. Researchers have shifted their attention from simple questions to complex questions, which require more KB triples and constraint inference.
  arXiv  Detail & Related papers  (2020-07-26T07:13:32Z)
• An Iterative Multi-Knowledge Transfer Network for Aspect-Based Sentiment Analysis [73.7488524683061]
  We propose a novel Iterative Multi-Knowledge Transfer Network (IMKTN) for end-to-end ABSA. Our IMKTN transfers the task-specific knowledge from any two of the three subtasks to another one at the token level by utilizing a well-designed routing algorithm. Experimental results on three benchmark datasets demonstrate the effectiveness and superiority of our approach.
  arXiv  Detail & Related papers  (2020-04-04T13:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.