Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization

• URL: http://arxiv.org/abs/2409.14907v1
• Date: Mon, 23 Sep 2024 11:01:31 GMT
• Title: Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarization
• Authors: Aseem Srivastava, Smriti Joshi, Tanmoy Chakraborty, Md Shad Akhtar,
• Abstract summary: Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks. Mental health experts first plan to apply domain knowledge in writing summaries. We introduce a novel planning engine to orchestrate structuring knowledge alignment.
• Score: 30.27904166070852
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In mental health counseling, condensing dialogues into concise and relevant summaries (aka counseling notes) holds pivotal significance. Large Language Models (LLMs) exhibit remarkable capabilities in various generative tasks; however, their adaptation to domain-specific intricacies remains challenging, especially within mental health contexts. Unlike standard LLMs, mental health experts first plan to apply domain knowledge in writing summaries. Our work enhances LLMs' ability by introducing a novel planning engine to orchestrate structuring knowledge alignment. To achieve high-order planning, we divide knowledge encapsulation into two major phases: (i) holding dialogue structure and (ii) incorporating domain-specific knowledge. We employ a planning engine on Llama-2, resulting in a novel framework, PIECE. Our proposed system employs knowledge filtering-cum-scaffolding to encapsulate domain knowledge. Additionally, PIECE leverages sheaf convolution learning to enhance its understanding of the dialogue's structural nuances. We compare PIECE with 14 baseline methods and observe a significant improvement across ROUGE and Bleurt scores. Further, expert evaluation and analyses validate the generation quality to be effective, sometimes even surpassing the gold standard. We further benchmark PIECE with other LLMs and report improvement, including Llama-2 (+2.72%), Mistral (+2.04%), and Zephyr (+1.59%), to justify the generalizability of the planning engine.

Related papers

• Structure-aware Domain Knowledge Injection for Large Language Models [37.089378357827826]
  This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus requirement to a mere 0.3%, while achieving an impressive 50% of traditional knowledge injection performance. Our method demonstrates the potential of comparable improvement against the state-of-the-art MMedLM2 on MMedBench, while significantly reducing the training costs to 5%.
  arXiv  Detail & Related papers  (2024-07-23T12:38:48Z)
• Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems [55.24980128638365]
  We show the necessity of external knowledge and goal guidance which contribute significantly to the recommendation accuracy and language quality. We propose a novel ChatCRS framework to decompose the complex CRS task into several sub-tasks. Experimental results on two multi-goal CRS datasets reveal that ChatCRS sets new state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2024-05-03T05:42:57Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations [50.81844184210381]
  We propose a general paradigm that augments large language models with DOmain-specific KnowledgE to enhance their performance on practical applications, namely DOKE. This paradigm relies on a domain knowledge extractor, working in three steps: 1) preparing effective knowledge for the task; 2) selecting the knowledge for each specific sample; and 3) expressing the knowledge in an LLM-understandable way.
  arXiv  Detail & Related papers  (2023-11-16T07:09:38Z)
• Systematic Assessment of Factual Knowledge in Large Language Models [48.75961313441549]
  This paper proposes a framework to assess the factual knowledge of large language models (LLMs) by leveraging knowledge graphs (KGs) Our framework automatically generates a set of questions and expected answers from the facts stored in a given KG, and then evaluates the accuracy of LLMs in answering these questions.
  arXiv  Detail & Related papers  (2023-10-18T00:20:50Z)
• Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution [48.86322922826514]
  This paper defines a new task of Knowledge-aware Language Model Attribution (KaLMA) First, we extend attribution source from unstructured texts to Knowledge Graph (KG), whose rich structures benefit both the attribution performance and working scenarios. Second, we propose a new Conscious Incompetence" setting considering the incomplete knowledge repository. Third, we propose a comprehensive automatic evaluation metric encompassing text quality, citation quality, and text citation alignment.
  arXiv  Detail & Related papers  (2023-10-09T11:45:59Z)
• KoLA: Carefully Benchmarking World Knowledge of Large Language Models [87.96683299084788]
  We construct a Knowledge-oriented LLM Assessment benchmark (KoLA) We mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. We use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, to evaluate the capacity to handle unseen data and evolving knowledge.
  arXiv  Detail & Related papers  (2023-06-15T17:20:46Z)
• Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding [24.315130086787374]
  We propose a knowledge-prompting-based PLM framework KP-PLM. This framework can be flexibly combined with existing mainstream PLMs. To further leverage the factual knowledge from these prompts, we propose two novel knowledge-aware self-supervised tasks.
  arXiv  Detail & Related papers  (2022-10-16T13:36:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.