Leveraging a Cognitive Model to Measure Subjective Similarity of Human and GPT-4 Written Content

• URL: http://arxiv.org/abs/2409.00269v2
• Date: Thu, 10 Oct 2024 14:51:43 GMT
• Title: Leveraging a Cognitive Model to Measure Subjective Similarity of Human and GPT-4 Written Content
• Authors: Tyler Malloy, Maria José Ferreira, Fei Fang, Cleotilde Gonzalez,
• Abstract summary: Cosine similarity between two documents can be computed using token embeddings formed by Large Language Models (LLMs) such as GPT-4. This similarity metric is beneficial in that it takes into account individual biases and constraints in a manner that is grounded in the cognitive mechanisms of decision making. This dataset is used to demonstrate the benefits of leveraging a cognitive model to measure the subjective similarity of human participants in an educational setting.
• Score: 26.409490082213445
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cosine similarity between two documents can be computed using token embeddings formed by Large Language Models (LLMs) such as GPT-4, and used to categorize those documents across a range of uses. However, these similarities are ultimately dependent on the corpora used to train these LLMs, and may not reflect subjective similarity of individuals or how their biases and constraints impact similarity metrics. This lack of cognitively-aware personalization of similarity metrics can be particularly problematic in educational and recommendation settings where there is a limited number of individual judgements of category or preference, and biases can be particularly relevant. To address this, we rely on an integration of an Instance-Based Learning (IBL) cognitive model with LLM embeddings to develop the Instance-Based Individualized Similarity (IBIS) metric. This similarity metric is beneficial in that it takes into account individual biases and constraints in a manner that is grounded in the cognitive mechanisms of decision making. To evaluate the IBIS metric, we also introduce a dataset of human categorizations of emails as being either dangerous (phishing) or safe (ham). This dataset is used to demonstrate the benefits of leveraging a cognitive model to measure the subjective similarity of human participants in an educational setting.

Related papers

• Causal Fair Metric: Bridging Causality, Individual Fairness, and Adversarial Robustness [7.246701762489971]
  Adversarial perturbation, used to identify vulnerabilities in models, and individual fairness, aiming for equitable treatment of similar individuals, both depend on metrics to generate comparable input data instances. Previous attempts to define such joint metrics often lack general assumptions about data or structural causal models and were unable to reflect counterfactual proximity. This paper introduces a causal fair metric formulated based on causal structures encompassing sensitive attributes and protected causal perturbation.
  arXiv  Detail & Related papers  (2023-10-30T09:53:42Z)
• Gender Biases in Automatic Evaluation Metrics for Image Captioning [87.15170977240643]
  We conduct a systematic study of gender biases in model-based evaluation metrics for image captioning tasks. We demonstrate the negative consequences of using these biased metrics, including the inability to differentiate between biased and unbiased generations. We present a simple and effective way to mitigate the metric bias without hurting the correlations with human judgments.
  arXiv  Detail & Related papers  (2023-05-24T04:27:40Z)
• Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learning [75.46664770669949]
  We present relation-aware contrastive self-supervised learning (ReCo) to integrate instance relations. Our ReCo consistently gains remarkable performance improvements.
  arXiv  Detail & Related papers  (2022-11-02T03:25:28Z)
• Learning Personalized Item-to-Item Recommendation Metric via Implicit Feedback [24.37151414523712]
  This paper studies the item-to-item recommendation problem in recommender systems from a new perspective of metric learning via implicit feedback. We develop and investigate a personalizable deep metric model that captures both the internal contents of items and how they were interacted with by users.
  arXiv  Detail & Related papers  (2022-03-18T18:08:57Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Refining Self-Supervised Learning in Imaging: Beyond Linear Metric [25.96406219707398]
  We introduce in this paper a new statistical perspective, exploiting the Jaccard similarity metric, as a measure-based metric. Specifically, our proposed metric may be interpreted as a dependence measure between two adapted projections learned from the so-called latent representations. To the best of our knowledge, this effectively non-linearly fused information embedded in the Jaccard similarity, is novel to self-supervision learning with promising results.
  arXiv  Detail & Related papers  (2022-02-25T19:25:05Z)
• Measuring Disparate Outcomes of Content Recommendation Algorithms with Distributional Inequality Metrics [5.74271110290378]
  We evaluate a set of metrics originating from economics, distributional inequality metrics, and their ability to measure disparities in content exposure in the Twitter algorithmic timeline. We show that we can use these metrics to identify content suggestion algorithms that contribute more strongly to skewed outcomes between users.
  arXiv  Detail & Related papers  (2022-02-03T14:41:39Z)
• Adaptive Hierarchical Similarity Metric Learning with Noisy Labels [138.41576366096137]
  We propose an Adaptive Hierarchical Similarity Metric Learning method. It considers two noise-insensitive information, textiti.e., class-wise divergence and sample-wise consistency. Our method achieves state-of-the-art performance compared with current deep metric learning approaches.
  arXiv  Detail & Related papers  (2021-10-29T02:12:18Z)
• Deep Relational Metric Learning [84.95793654872399]
  This paper presents a deep relational metric learning framework for image clustering and retrieval. We learn an ensemble of features that characterizes an image from different aspects to model both interclass and intraclass distributions. Experiments on the widely-used CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate that our framework improves existing deep metric learning methods and achieves very competitive results.
  arXiv  Detail & Related papers  (2021-08-23T09:31:18Z)
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning [141.2562447971]
  We propose to solve a few-shot (or low-shot) visual reasoning problem, by resorting to analogical reasoning. We extract structural relationships between elements in both domains, and enforce them to be as similar as possible with analogical learning. We validate our method on RAVEN dataset, on which it outperforms state-of-the-art method, with larger gains when the training data is scarce.
  arXiv  Detail & Related papers  (2020-07-23T14:00:34Z)
• Learning similarity measures from data [1.4766350834632755]
  Defining similarity measures is a requirement for some machine learning methods. Data sets are typically gathered as part of constructing a CBR or machine learning system. Our objective is to investigate how to apply machine learning to effectively learn a similarity measure.
  arXiv  Detail & Related papers  (2020-01-15T13:29:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.