Empirical Analysis of Dialogue Relation Extraction with Large Language Models

• URL: http://arxiv.org/abs/2404.17802v1
• Date: Sat, 27 Apr 2024 06:55:41 GMT
• Title: Empirical Analysis of Dialogue Relation Extraction with Large Language Models
• Authors: Guozheng Li, Zijie Xu, Ziyu Shang, Jiajun Liu, Ke Ji, Yikai Guo,
• Abstract summary: We investigate the capabilities of different large language models (LLMs) in Dialogue relation extraction. We find that LLMs significantly alleviate two issues in existing DRE methods. LLMs deliver competitive or superior performances under both full-shot and few-shot settings.
• Score: 7.820897176683249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dialogue relation extraction (DRE) aims to extract relations between two arguments within a dialogue, which is more challenging than standard RE due to the higher person pronoun frequency and lower information density in dialogues. However, existing DRE methods still suffer from two serious issues: (1) hard to capture long and sparse multi-turn information, and (2) struggle to extract golden relations based on partial dialogues, which motivates us to discover more effective methods that can alleviate the above issues. We notice that the rise of large language models (LLMs) has sparked considerable interest in evaluating their performance across diverse tasks. To this end, we initially investigate the capabilities of different LLMs in DRE, considering both proprietary models and open-source models. Interestingly, we discover that LLMs significantly alleviate two issues in existing DRE methods. Generally, we have following findings: (1) scaling up model size substantially boosts the overall DRE performance and achieves exceptional results, tackling the difficulty of capturing long and sparse multi-turn information; (2) LLMs encounter with much smaller performance drop from entire dialogue setting to partial dialogue setting compared to existing methods; (3) LLMs deliver competitive or superior performances under both full-shot and few-shot settings compared to current state-of-the-art; (4) LLMs show modest performances on inverse relations but much stronger improvements on general relations, and they can handle dialogues of various lengths especially for longer sequences.

Related papers

• MT2-CSD: A New Dataset and Multi-Semantic Knowledge Fusion Method for Conversational Stance Detection [5.892386683874131]
  We introduce MT2-CSD, a comprehensive dataset for multi-target, multi-turn conversational stance detection.<n>To address these challenges, we propose the Large Language model enhanced Conversational Attention Network (LLM-CRAN)
  arXiv  Detail & Related papers  (2025-06-26T06:59:30Z)
• DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation [21.229539297323488]
  Large Language Models (LLMs) excel at many tasks but struggle with ambiguous scenarios where multiple valid responses exist.<n>Small Language Models (SLMs) demonstrate robustness in such scenarios but are susceptible to misleading or adversarial inputs.<n>We introduce SLIDE (Small and Large Integrated for Dialogue Evaluation), a method integrating SLMs and LLMs via adaptive weighting.
  arXiv  Detail & Related papers  (2025-06-04T23:41:31Z)
• Towards Robust Dialogue Breakdown Detection: Addressing Disruptors in Large Language Models with Self-Guided Reasoning [30.13634341221476]
  Large language models (LLMs) are rapidly changing various domains. This paper addresses the challenge of detecting and mitigating dialogue breakdowns within LLM-driven systems. We propose an approach that combines specialized fine-tuning with advanced prompting strategies.
  arXiv  Detail & Related papers  (2025-04-26T07:51:05Z)
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation [81.18701211912779]
  We introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. Our method has achieved state-of-the-art performance on two common datasets.
  arXiv  Detail & Related papers  (2024-12-24T16:38:04Z)
• The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio [118.75449542080746]
  This paper presents the first systematic investigation of hallucinations in large multimodal models (LMMs) Our study reveals two key contributors to hallucinations: overreliance on unimodal priors and spurious inter-modality correlations. Our findings highlight key vulnerabilities, including imbalances in modality integration and biases from training data, underscoring the need for balanced cross-modal learning.
  arXiv  Detail & Related papers  (2024-10-16T17:59:02Z)
• FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understanding [32.197113821638936]
  We propose a novel integrated Long-Context Large Language Model (FltLM) FltLM incorporates a context filter with a soft mask mechanism, identifying and dynamically excluding irrelevant content to concentrate on pertinent information. Experimental results demonstrate that FltLM significantly outperforms supervised fine-tuning and retrieval-based methods in complex QA scenarios.
  arXiv  Detail & Related papers  (2024-10-09T13:47:50Z)
• Are LLMs Good Annotators for Discourse-level Event Relation Extraction? [15.365993658296016]
  Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning.
  arXiv  Detail & Related papers  (2024-07-28T19:27:06Z)
• SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation [23.203761925540736]
  We propose a novel framework SLIDE (Small and Large Integrated for Dialogue Evaluation) Our approach achieves state-of-the-art performance in both the classification and evaluation tasks, and additionally the SLIDE exhibits better correlation with human evaluators.
  arXiv  Detail & Related papers  (2024-05-24T20:32:49Z)
• Evaluating Very Long-Term Conversational Memory of LLM Agents [95.84027826745609]
  We introduce a machine-human pipeline to generate high-quality, very long-term dialogues. We equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency.
  arXiv  Detail & Related papers  (2024-02-27T18:42:31Z)
• Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models [51.75805497456226]
  This work focuses on the factual consistency issue with the help of the dialogue summarization task. Our evaluation shows that, on average, 26.8% of the summaries generated by LLMs contain factual inconsistency. To stimulate and enhance the dialogue comprehension ability of LLMs, we propose a fine-tuning paradigm with auto-constructed multi-task data.
  arXiv  Detail & Related papers  (2023-11-13T09:32:12Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generation [53.87485260058957]
  We study video-grounded dialogue generation, where a response is generated based on the dialogue context and the associated video. The primary challenges of this task lie in (1) the difficulty of integrating video data into pre-trained language models (PLMs) We propose a multi-agent reinforcement learning method to collaboratively perform reasoning on different modalities.
  arXiv  Detail & Related papers  (2022-10-22T14:45:29Z)
• A Mixture-of-Expert Approach to RL-based Dialogue Management [56.08449336469477]
  We use reinforcement learning to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maximizes overall user satisfaction. Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with aly complex action space even for a medium-size vocabulary. We develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) a number of specialized LMs (or experts) capable of generating utterances corresponding to a
  arXiv  Detail & Related papers  (2022-05-31T19:00:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.