Soft Measures for Extracting Causal Collective Intelligence

• URL: http://arxiv.org/abs/2409.18911v1
• Date: Fri, 27 Sep 2024 16:54:36 GMT
• Title: Soft Measures for Extracting Causal Collective Intelligence
• Authors: Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone,
• Abstract summary: Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments.
• Score: 2.1948331804353223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.

Related papers

• THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models [0.0]
  Hallucination, the generation of factually incorrect content, is a growing challenge in Large Language Models. This paper introduces THaMES, an integrated framework and library addressing this gap. THaMES offers an end-to-end solution for evaluating and mitigating hallucinations in LLMs.
  arXiv  Detail & Related papers  (2024-09-17T16:55:25Z)
• A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models [63.949883238901414]
  We present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.
  arXiv  Detail & Related papers  (2024-08-29T17:46:18Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making. We present a process-based benchmark MR-Ben that demands a meta-reasoning skill. Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation [20.178644251662316]
  We introduce the hierarchical graph of thoughts (HGOT) to enhance the retrieval of pertinent passages during in-context learning. The framework employs the divide-and-conquer strategy to break down complex queries into manageable sub-queries. It refines self-consistency majority voting for answer selection, which incorporates the recently proposed citation recall and precision metrics.
  arXiv  Detail & Related papers  (2024-02-14T18:41:19Z)
• A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition [56.20144064187554]
  This paper develops a novel cross-modal feature fusion method for the Conversational emotion recognition (CER) task. By setting a matching weight and calculating attention scores between modal features row by row, LMAM contains fewer parameters than the self-attention method. We show that LMAM can be embedded into any existing state-of-the-art DL-based CER methods and help boost their performance in a plug-and-play manner.
  arXiv  Detail & Related papers  (2023-06-16T16:02:44Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• The m-connecting imset and factorization for ADMG models [10.839217026568784]
  Acyclic directed mixed graph (ADMG) models characterize margins of DAG models. ADMG models have not seen wide-spread use due to their complexity and a shortage of statistical tools for their analysis. We introduce the m-connecting imset which provides an alternative representation for the independence models induced by ADMGs.
  arXiv  Detail & Related papers  (2022-07-18T22:29:15Z)
• Time Series Forecasting Using Fuzzy Cognitive Maps: A Survey [0.0]
  Fuzzy Cognitive Maps (FCM) have shown remarkable results as a tool to model and analyze the dynamics of complex systems. FCMs are a mixture of fuzzy logic, neural network, and expert system aspects, which act as a powerful tool for simulating and studying the dynamic behavior of complex systems.
  arXiv  Detail & Related papers  (2022-01-07T02:11:53Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Evolutionary Algorithms for Fuzzy Cognitive Maps [0.0]
  Fuzzy Cognitive Maps (FCMs) is a complex systems modeling technique which has lately risen in popularity. The present study reviews the genetic algorithms used for training FCMs, as well as gives a general overview of the FCM learning algorithms.
  arXiv  Detail & Related papers  (2020-12-19T15:17:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.