AI in the Gray: Exploring Moderation Policies in Dialogic Large Language
Models vs. Human Answers in Controversial Topics
- URL: http://arxiv.org/abs/2308.14608v1
- Date: Mon, 28 Aug 2023 14:23:04 GMT
- Title: AI in the Gray: Exploring Moderation Policies in Dialogic Large Language
Models vs. Human Answers in Controversial Topics
- Authors: Vahid Ghafouri, Vibhor Agarwal, Yong Zhang, Nishanth Sastry, Jose
Such, Guillermo Suarez-Tangil
- Abstract summary: Controversial topics, such as "religion", "gender identity", "freedom of speech", and "equality", can be a source of conflict.
By exposing ChatGPT to such debatable questions, we aim to understand its level of awareness and if existing models are subject to socio-political and/or economic biases.
- Score: 7.444967232238375
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The introduction of ChatGPT and the subsequent improvement of Large Language
Models (LLMs) have prompted more and more individuals to turn to the use of
ChatBots, both for information and assistance with decision-making. However,
the information the user is after is often not formulated by these ChatBots
objectively enough to be provided with a definite, globally accepted answer.
Controversial topics, such as "religion", "gender identity", "freedom of
speech", and "equality", among others, can be a source of conflict as partisan
or biased answers can reinforce preconceived notions or promote disinformation.
By exposing ChatGPT to such debatable questions, we aim to understand its level
of awareness and if existing models are subject to socio-political and/or
economic biases. We also aim to explore how AI-generated answers compare to
human ones. For exploring this, we use a dataset of a social media platform
created for the purpose of debating human-generated claims on polemic subjects
among users, dubbed Kialo.
Our results show that while previous versions of ChatGPT have had important
issues with controversial topics, more recent versions of ChatGPT
(gpt-3.5-turbo) are no longer manifesting significant explicit biases in
several knowledge areas. In particular, it is well-moderated regarding economic
aspects. However, it still maintains degrees of implicit libertarian leaning
toward right-winged ideals which suggest the need for increased moderation from
the socio-political point of view. In terms of domain knowledge on
controversial topics, with the exception of the "Philosophical" category,
ChatGPT is performing well in keeping up with the collective human level of
knowledge. Finally, we see that sources of Bing AI have slightly more tendency
to the center when compared to human answers. All the analyses we make are
generalizable to other types of biases and domains.
Related papers
- In Generative AI we Trust: Can Chatbots Effectively Verify Political
Information? [39.58317527488534]
This article presents a comparative analysis of the ability of two large language model (LLM)-based chatbots, ChatGPT and Bing Chat, to detect veracity of political information.
We use AI auditing methodology to investigate how chatbots evaluate true, false, and borderline statements on five topics: COVID-19, Russian aggression against Ukraine, the Holocaust, climate change, and LGBTQ+ related debates.
The results show high performance of ChatGPT for the baseline veracity evaluation task, with 72 percent of the cases evaluated correctly on average across languages without pre-training.
arXiv Detail & Related papers (2023-12-20T15:17:03Z) - Critical Role of Artificially Intelligent Conversational Chatbot [0.0]
We explore scenarios involving ChatGPT's ethical implications within academic contexts.
We propose architectural solutions aimed at preventing inappropriate use and promoting responsible AI interactions.
arXiv Detail & Related papers (2023-10-31T14:08:07Z) - Primacy Effect of ChatGPT [69.49920102917598]
We study the primacy effect of ChatGPT: the tendency of selecting the labels at earlier positions as the answer.
We hope that our experiments and analyses provide additional insights into building more reliable ChatGPT-based solutions.
arXiv Detail & Related papers (2023-10-20T00:37:28Z) - Public Attitudes Toward ChatGPT on Twitter: Sentiments, Topics, and
Occupations [1.6466986427682635]
We investigated public attitudes towards ChatGPT by applying natural language processing techniques such as sentiment analysis.
Our sentiment analysis result indicates that the overall sentiment was largely neutral to positive, and negative sentiments were decreasing over time.
Our topic model reveals that the most popular topics discussed were Education, Bard, Search Engines, OpenAI, Marketing, and Cybersecurity.
arXiv Detail & Related papers (2023-06-22T15:10:18Z) - Adding guardrails to advanced chatbots [5.203329540700177]
Launch of ChatGPT in November 2022 has ushered in a new era of AI.
There are already concerns that humans may be replaced by chatbots for a variety of jobs.
These biases may cause significant harm and/or inequity toward different subpopulations.
arXiv Detail & Related papers (2023-06-13T02:23:04Z) - ChatGPT for Us: Preserving Data Privacy in ChatGPT via Dialogue Text
Ambiguation to Expand Mental Health Care Delivery [52.73936514734762]
ChatGPT has gained popularity for its ability to generate human-like dialogue.
Data-sensitive domains face challenges in using ChatGPT due to privacy and data-ownership concerns.
We propose a text ambiguation framework that preserves user privacy.
arXiv Detail & Related papers (2023-05-19T02:09:52Z) - AI, write an essay for me: A large-scale comparison of human-written
versus ChatGPT-generated essays [66.36541161082856]
ChatGPT and similar generative AI models have attracted hundreds of millions of users.
This study compares human-written versus ChatGPT-generated argumentative student essays.
arXiv Detail & Related papers (2023-04-24T12:58:28Z) - To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection.
We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains.
Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
arXiv Detail & Related papers (2023-04-04T03:04:28Z) - Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot.
Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community.
It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
arXiv Detail & Related papers (2023-02-08T09:44:51Z) - A Categorical Archive of ChatGPT Failures [47.64219291655723]
ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation.
It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries.
However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
arXiv Detail & Related papers (2023-02-06T04:21:59Z) - The political ideology of conversational AI: Converging evidence on
ChatGPT's pro-environmental, left-libertarian orientation [0.0]
OpenAI introduced ChatGPT, a state-of-the-art dialogue model that can converse with its human counterparts.
This paper focuses on one of democratic society's most important decision-making processes: political elections.
We uncover ChatGPT's pro-environmental, left-libertarian ideology.
arXiv Detail & Related papers (2023-01-05T07:13:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.