The BS-meter: A ChatGPT-Trained Instrument to Detect Sloppy Language-Games

• URL: http://arxiv.org/abs/2411.15129v2
• Date: Tue, 10 Jun 2025 07:11:59 GMT
• Title: The BS-meter: A ChatGPT-Trained Instrument to Detect Sloppy Language-Games
• Authors: Alessandro Trevisan, Harry Giddens, Sarah Dillon, Alan F. Blackwell,
• Abstract summary: We show that a statistical model of sloppy bullshit can reliably relate the Frankfurtian artificial bullshit of ChatGPT to the political and workplace functions of bullshit as observed in natural human language.
• Score: 41.94295877935867
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: What can we learn about language from studying how it is used by ChatGPT and other large language model (LLM)-based chatbots? In this paper, we analyse the distinctive character of language generated by ChatGPT, in relation to questions raised by natural language processing pioneer, and student of Wittgenstein, Margaret Masterman. Following frequent complaints that LLM-based chatbots produce "slop," or even "bullshit," in the sense of Frankfurt's popular monograph On Bullshit, we conduct an empirical study to contrast the language of 1,000 scientific publications with typical text generated by ChatGPT. We then explore whether the same language features can be detected in two well-known contexts of social dysfunction: George Orwell's critique of political speech, and David Graeber's characterisation of bullshit jobs. Using simple hypothesis-testing methods, we demonstrate that a statistical model of sloppy bullshit can reliably relate the Frankfurtian artificial bullshit of ChatGPT to the political and workplace functions of bullshit as observed in natural human language.

Related papers

• ChatGPT for President! Presupposed content in politicians versus GPT-generated texts [0.0]
  This study examines ChatGPT-4's capability to replicate linguistic strategies used in political discourse. Using a corpus-based pragmatic analysis, this study assesses how well ChatGPT can mimic these persuasive strategies.
  arXiv  Detail & Related papers  (2025-03-03T07:48:04Z)
• OffensiveLang: A Community Based Implicit Offensive Language Dataset [5.813922783967869]
  Hate speech or offensive languages exist in both explicit and implicit forms. OffensiveLang is a community based implicit offensive language dataset. We present a prompt-based approach that effectively generates implicit offensive languages.
  arXiv  Detail & Related papers  (2024-03-04T20:34:58Z)
• Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model [23.60677380868016]
  Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills. Here, we conduct the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages. We find that ChatGPT massively underperforms purpose-built systems, particularly in English.
  arXiv  Detail & Related papers  (2023-10-23T17:21:03Z)
• Primacy Effect of ChatGPT [69.49920102917598]
  We study the primacy effect of ChatGPT: the tendency of selecting the labels at earlier positions as the answer. We hope that our experiments and analyses provide additional insights into building more reliable ChatGPT-based solutions.
  arXiv  Detail & Related papers  (2023-10-20T00:37:28Z)
• Chatbot-supported Thesis Writing: An Autoethnographic Report [0.0]
  ChatGPT might be applied to formats that require learners to generate text, such as bachelor theses or student research papers. ChatGPT is to be valued as a beneficial tool in thesis writing. However, writing a conclusive thesis still requires the learner's meaningful engagement.
  arXiv  Detail & Related papers  (2023-10-14T09:09:26Z)
• What has ChatGPT read? The origins of archaeological citations used by a generative artificial intelligence application [0.0]
  This paper tested what archaeological literature appears to have been included in ChatGPT's training phase. While ChatGPT offered seemingly pertinent references, a large percentage proved to be fictitious. It can be shown that all references provided by ChatGPT that were found to be genuine have also been cited on Wikipedia pages.
  arXiv  Detail & Related papers  (2023-08-07T05:06:35Z)
• Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text [48.36706154871577]
  We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts) It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts. We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
  arXiv  Detail & Related papers  (2023-07-21T06:38:37Z)
• Emergence of a phonological bias in ChatGPT [0.0]
  I demonstrate that ChatGPT displays phonological biases that are a hallmark of human language processing. ChatGPT has a tendency to use consonants over vowels to identify words. This is observed across languages that differ in their relative distribution of consonants and vowels such as English and Spanish.
  arXiv  Detail & Related papers  (2023-05-25T10:57:43Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [56.85319224208865]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account. We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed. Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Ethical ChatGPT: Concerns, Challenges, and Commandments [5.641321839562139]
  This paper highlights specific ethical concerns on ChatGPT and articulates key challenges when ChatGPT is used in various applications. Practical commandments of ChatGPT are also proposed that can serve as checklist guidelines for those applying ChatGPT in their applications.
  arXiv  Detail & Related papers  (2023-05-18T02:04:13Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification [0.0]
  ChatGPT has shown strong capabilities in natural language generation tasks, which naturally leads researchers to explore where its abilities end. We compare ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on datasets, manually annotated with genres. Results show that ChatGPT outperforms the fine-tuned model when applied to the dataset which was not seen before by either of the models.
  arXiv  Detail & Related papers  (2023-03-07T14:59:33Z)
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity [79.12003701981092]
  We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-02-08T12:35:34Z)
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
  Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community. It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
  arXiv  Detail & Related papers  (2023-02-08T09:44:51Z)
• ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text [2.0378492681344493]
  We study whether a machine learning model can be effectively trained to accurately distinguish between original human and seemingly human (that is, ChatGPT-generated) text. We employ an explainable artificial intelligence framework to gain insight into the reasoning behind the model trained to differentiate between ChatGPT-generated and human-generated text. Our study focuses on short online reviews, conducting two experiments comparing human-generated and ChatGPT-generated text.
  arXiv  Detail & Related papers  (2023-01-30T08:06:08Z)
• Language Models as Inductive Reasoners [125.99461874008703]
  We propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts. We create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. We provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts.
  arXiv  Detail & Related papers  (2022-12-21T11:12:14Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• COLD: A Benchmark for Chinese Offensive Language Detection [54.60909500459201]
  We use COLDataset, a Chinese offensive language dataset with 37k annotated sentences. We also propose textscCOLDetector to study output offensiveness of popular Chinese language models. Our resources and analyses are intended to help detoxify the Chinese online communities and evaluate the safety performance of generative language models.
  arXiv  Detail & Related papers  (2022-01-16T11:47:23Z)
• The Rediscovery Hypothesis: Language Models Need to Meet Linguistics [8.293055016429863]
  We study whether linguistic knowledge is a necessary condition for good performance of modern language models. We show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures. This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objective with linguistic information.
  arXiv  Detail & Related papers  (2021-03-02T15:57:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.