"Is ChatGPT a Better Explainer than My Professor?": Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline

• URL: http://arxiv.org/abs/2406.18512v1
• Date: Wed, 26 Jun 2024 17:33:51 GMT
• Title: "Is ChatGPT a Better Explainer than My Professor?": Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline
• Authors: Grace Li, Milad Alshomary, Smaranda Muresan,
• Abstract summary: Explanations form the foundation of knowledge sharing and build upon communication principles, social dynamics, and learning theories. Our research leverages previous work on explanatory acts, a framework for understanding the different strategies that explainers and explainees employ in a conversation to both explain, understand, and engage with the other party. With the rise of generative AI in the past year, we hope to better understand the capabilities of Large Language Models (LLMs) and how they can augment expert explainer's capabilities in conversational settings.
• Score: 23.81489190082685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explanations form the foundation of knowledge sharing and build upon communication principles, social dynamics, and learning theories. We focus specifically on conversational approaches for explanations because the context is highly adaptive and interactive. Our research leverages previous work on explanatory acts, a framework for understanding the different strategies that explainers and explainees employ in a conversation to both explain, understand, and engage with the other party. We use the 5-Levels dataset was constructed from the WIRED YouTube series by Wachsmuth et al., and later annotated by Booshehri et al. with explanatory acts. These annotations provide a framework for understanding how explainers and explainees structure their response when crafting a response. With the rise of generative AI in the past year, we hope to better understand the capabilities of Large Language Models (LLMs) and how they can augment expert explainer's capabilities in conversational settings. To achieve this goal, the 5-Levels dataset (We use Booshehri et al.'s 2023 annotated dataset with explanatory acts.) allows us to audit the ability of LLMs in engaging in explanation dialogues. To evaluate the effectiveness of LLMs in generating explainer responses, we compared 3 different strategies, we asked human annotators to evaluate 3 different strategies: human explainer response, GPT4 standard response, GPT4 response with Explanation Moves.

Related papers

• Modeling the Quality of Dialogical Explanations [21.429245729478918]
  We study explanation dialogues in terms of the interactions between the explainer and explainee. We analyze the interaction flows, comparing them to those appearing in expert dialogues. We encode the interaction flows using two language models that can handle long inputs.
  arXiv  Detail & Related papers  (2024-03-01T16:49:55Z)
• Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models [54.21695754082441]
  We propose a framework to teach Large Language Models (LLMs) to generate explainable stock predictions. A reflective agent learns how to explain past stock movements through self-reasoning, while the PPO trainer trains the model to generate the most likely explanations. Our framework can outperform both traditional deep-learning and LLM methods in prediction accuracy and Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-02-06T03:18:58Z)
• Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning [50.00090601424348]
  Large language models (LLMs) have shown remarkable capabilities in various natural language understanding tasks. We propose EASE, an Explanation-Aware Soft Ensemble framework to empower in-context learning with LLMs.
  arXiv  Detail & Related papers  (2023-11-13T06:13:38Z)
• May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability [19.1651512576945]
  We investigate if free-form conversations can enhance users' comprehension of static explanations. We measure the effect of the conversation on participants' ability to choose from three machine learning models. Our findings highlight the importance of customized model explanations in the format of free-form conversations.
  arXiv  Detail & Related papers  (2023-09-25T09:00:38Z)
• Complementary Explanations for Effective In-Context Learning [77.83124315634386]
  Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts. This work aims to better understand the mechanisms by which explanations are used for in-context learning.
  arXiv  Detail & Related papers  (2022-11-25T04:40:47Z)
• Explanations from Large Language Models Make Small Reasoners Better [61.991772773700006]
  We show that our method can consistently and significantly outperform finetuning baselines across different settings. As a side benefit, human evaluation shows that our method can generate high-quality explanations to justify its predictions.
  arXiv  Detail & Related papers  (2022-10-13T04:50:02Z)
• "Mama Always Had a Way of Explaining Things So I Could Understand'': A Dialogue Corpus for Learning to Construct Explanations [26.540485804067536]
  We introduce a first corpus of dialogical explanations to enable NLP research on how humans explain. The corpus consists of 65 transcribed English dialogues from the Wired video series emph5 Levels, explaining 13 topics to five explainees of different proficiency.
  arXiv  Detail & Related papers  (2022-09-06T14:00:22Z)
• Human Interpretation of Saliency-based Explanation Over Text [65.29015910991261]
  We study saliency-based explanations over textual data. We find that people often mis-interpret the explanations. We propose a method to adjust saliencies based on model estimates of over- and under-perception.
  arXiv  Detail & Related papers  (2022-01-27T15:20:32Z)
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering [159.9645181522436]
  GreaseLM is a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. We show that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
  arXiv  Detail & Related papers  (2022-01-21T19:00:05Z)
• Explanation as a process: user-centric construction of multi-level and multi-modal explanations [0.34410212782758043]
  We present a process-based approach that combines multi-level and multi-modal explanations. We use Inductive Logic Programming, an interpretable machine learning approach, to learn a comprehensible model.
  arXiv  Detail & Related papers  (2021-10-07T19:26:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.