Related papers: ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

URL: http://arxiv.org/abs/2307.01135v1
Date: Mon, 3 Jul 2023 16:15:34 GMT
Title: ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience
Authors: Ruiyun Xu (Rayna), Yue Feng (Katherine), and Hailiang Chen
Abstract summary: We investigate the differences in user behavior when employing search engines and ChatGPT tools for information-seeking tasks. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools.
Score: 0.0
License: http://creativecommons.org/licenses/by/4.0/
Abstract: The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.

Related papers

Adoption and Impact of ChatGPT in Computer Science Education: A Case Study on a Database Administration Course [0.46040036610482665]
This contribution presents an exploratory and correlational study conducted with 37 students who used ChatGPT as a support tool to learn database administration. The usage and perceived utility of ChatGPT were moderate, but positive correlations between student grade and ChatGPT usage were found.
arXiv Detail & Related papers (2024-05-26T20:51:28Z)
Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
arXiv Detail & Related papers (2023-11-11T11:01:13Z)
Are Chatbots Reliable Text Annotators? Sometimes [0.0]
ChatGPT is a closed-source product which has major drawbacks with regards to transparency, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer an alternative without these drawbacks.
arXiv Detail & Related papers (2023-11-09T22:28:14Z)
"ChatGPT, a Friend or Foe for Education?" Analyzing the User's Perspectives on the Latest AI Chatbot Via Reddit [0.0]
This study has analyzed 247 Reddit top posts related to the educational use of ChatGPT. Results show that the majority of the users took a neutral viewpoint. There was more positive perception than negative regarding the usefulness of ChatGPT in education.
arXiv Detail & Related papers (2023-09-27T23:59:44Z)
Inappropriate Benefits and Identification of ChatGPT Misuse in Programming Tests: A Controlled Experiment [0.0]
Students can ask ChatGPT to complete a programming task, generating a solution from other people's work without proper acknowledgment of the source(s) We performed a controlled experiment measuring the inappropriate benefits of using ChatGPT in terms of completion time and programming performance.
arXiv Detail & Related papers (2023-08-11T06:42:29Z)
AI and Education: An Investigation into the Use of ChatGPT for Systems Thinking [0.09618478226711495]
The study assesses the accuracy, helpfulness, and reliability of ChatGPT's responses across different versions of the tool. Despite some limitations, this study suggests that ChatGPT can be a valuable tool for teaching and learning systems thinking.
arXiv Detail & Related papers (2023-07-26T14:12:16Z)
Transformative Effects of ChatGPT on Modern Education: Emerging Era of AI Chatbots [36.760677949631514]
ChatGPT was released to provide coherent and useful replies based on analysis of large volumes of data. Our preliminary evaluation concludes that ChatGPT performed differently in each subject area including finance, coding and maths. There are clear drawbacks in its use, such as the possibility of producing inaccurate or false data. Academic regulations and evaluation practices need to be updated, should ChatGPT be used as a tool in education.
arXiv Detail & Related papers (2023-05-25T17:35:57Z)
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
arXiv Detail & Related papers (2023-04-12T05:08:52Z)
To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
arXiv Detail & Related papers (2023-04-04T03:04:28Z)
ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models [49.52083248451775]
Large language models (LLMs) have made significant progress in NLP. We specifically focus on ChatGPT, a widely used and easily accessible LLM. We conduct a series of experiments on 11 datasets to evaluate ChatGPT's commonsense abilities.
arXiv Detail & Related papers (2023-03-29T03:05:43Z)
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT [103.57103957631067]
ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. We evaluate ChatGPT's understanding ability by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question answering tasks.
arXiv Detail & Related papers (2023-02-19T12:29:33Z)
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community. It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
arXiv Detail & Related papers (2023-02-08T09:44:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.