Long-form analogies generated by chatGPT lack human-like psycholinguistic properties

• URL: http://arxiv.org/abs/2306.04537v1
• Date: Wed, 7 Jun 2023 15:42:31 GMT
• Title: Long-form analogies generated by chatGPT lack human-like psycholinguistic properties
• Authors: S. M. Seals and Valerie L. Shalin
• Abstract summary: We apply psycholinguistic methods to evaluate individual sentences from long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT.
• Score: 0.5884031187931463
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Psycholinguistic analyses provide a means of evaluating large language model (LLM) output and making systematic comparisons to human-generated text. These methods can be used to characterize the psycholinguistic properties of LLM output and illustrate areas where LLMs fall short in comparison to human-generated text. In this work, we apply psycholinguistic methods to evaluate individual sentences from long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT. We perform a supervised classification analysis using 78 features extracted from Coh-metrix that analyze text cohesion, language, and readability (Graesser et. al., 2004). Results illustrate high performance for classifying student-generated and chatGPT-generated analogies. To evaluate which features contribute most to model performance, we use a hierarchical clustering approach. Results from this analysis illustrate several linguistic differences between the two sources.

Related papers

• An Interpretable Deep Learning Approach for Morphological Script Type Analysis [15.142597136864618]
  We propose an interpretable deep learning-based approach to morphological script type analysis. More precisely, we adapt a deep instance segmentation method to learn comparable character prototypes. We demonstrate our approach by applying it to the Textualis Formata script type and its two subtypes formalized by A. Derolez: Northern and Southern Textualis
  arXiv  Detail & Related papers  (2024-08-20T19:15:06Z)
• Comprehending Lexical and Affective Ontologies in the Demographically Diverse Spatial Social Media Discourse [0.0]
  This study aims to comprehend linguistic and socio-demographic features, encompassing English language styles, conveyed sentiments, and lexical diversity within social media data. Our analysis entails the extraction and examination of various statistical, grammatical, and sentimental features from two groups. Our investigation unveils substantial disparities in certain linguistic attributes between the two groups, yielding a macro F1 score of approximately 0.85.
  arXiv  Detail & Related papers  (2023-11-12T04:23:33Z)
• StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding [72.38872974837462]
  We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus. textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
  arXiv  Detail & Related papers  (2023-10-19T16:29:23Z)
• ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base [51.777618249271725]
  ANALOGYKB is a million-scale analogy knowledge base derived from existing knowledge graphs (KGs) It identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs)
  arXiv  Detail & Related papers  (2023-05-10T09:03:01Z)
• ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models [1.4546044532817048]
  ANALOGICAL is a new benchmark to intrinsically evaluate large language models. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy.
  arXiv  Detail & Related papers  (2023-05-08T21:12:20Z)
• Quantifying Synthesis and Fusion and their Impact on Machine Translation [79.61874492642691]
  In Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level. For computing literature, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish,
  arXiv  Detail & Related papers  (2022-05-06T17:04:58Z)
• Leveraging Pre-trained Language Model for Speech Sentiment Analysis [58.78839114092951]
  We explore the use of pre-trained language models to learn sentiment information of written texts for speech sentiment analysis. We propose a pseudo label-based semi-supervised training strategy using a language model on an end-to-end speech sentiment approach.
  arXiv  Detail & Related papers  (2021-06-11T20:15:21Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• Evaluating text coherence based on the graph of the consistency of phrases to identify symptoms of schizophrenia [0.0]
  State-of-the-art methods of the detection of schizophrenia symptoms based on the estimation of text coherence have been analyzed. The method based on the graph of the consistency of phrases has been proposed to evaluate the semantic coherence and the cohesion of a text.
  arXiv  Detail & Related papers  (2020-05-06T08:38:20Z)
• Evaluating Transformer-Based Multilingual Text Classification [55.53547556060537]
  We argue that NLP tools perform unequally across languages with different syntactic and morphological structures. We calculate word order and morphological similarity indices to aid our empirical study.
  arXiv  Detail & Related papers  (2020-04-29T03:34:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.