Prototyping Theories with ChatGPT: Experiment with the Technology Acceptance Model

• URL: http://arxiv.org/abs/2307.05488v1
• Date: Sun, 4 Jun 2023 23:55:53 GMT
• Title: Prototyping Theories with ChatGPT: Experiment with the Technology Acceptance Model
• Abstract summary: The objective of the studies is to assess ChatGPT's ability to comprehend theoretical concepts and differentiate between constructs. The results of the experiments indicate that ChatGPT can generate responses aligned with the constructs of the Acceptance Technology Model (TAM) The experiments reveal potential biases in the generated samples, particularly regarding gender and usage experiences.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This research paper presents the findings of two experimental studies that explore the use of ChatGPT as a tool for theory prototyping. The objective of the studies is to assess ChatGPT's ability to comprehend theoretical concepts and differentiate between constructs. During the experiments, duplicated responses were identified in both Study 1 and Study 2, with duplicate response rates of 26.25% and 40% respectively. The results of the experiments indicate that ChatGPT can generate responses aligned with the constructs of the Technology Acceptance Model (TAM). The loading and reliability coefficients demonstrate the validity of the models, with Study 1 achieving an R-squared value of 82% and Study 2 achieving 71%. In Study 2, two items with negative wording exhibited low loadings and were subsequently removed from the model. Both studies exhibit reasonable discriminant validity despite high correlations among the TAM constructs. The experiments reveal potential biases in the generated samples, particularly regarding gender and usage experiences. These biases may impact the responses of constructs and should be considered when interpreting ChatGPT's conceptual capabilities. In sum, ChatGPT shows promise as a tool for theory prototyping, generating relevant responses aligned with theoretical constructs. However, further investigation is needed to address limitations such as duplicated responses, variations in prompts, and the generalizability of findings to different contexts.

Related papers

• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation [4.702729080310267]
  We build a dataset of 11,488 idea pairs grounded in objective outcomes from PapersWithCode.<n>We train models to discover latent reasoning paths, achieving 71.35% acc. with interpretable justifications.<n>Our results demonstrate that compute-efficient small language models can serve as effective, objective verifiers.
  arXiv  Detail & Related papers  (2026-04-06T19:03:11Z)
• Observationally Informed Adaptive Causal Experimental Design [55.998153710215654]
  We propose Active Residual Learning, a new paradigm that leverages the observational model as a foundational prior.<n>This approach shifts the experimental focus from learning target causal quantities from scratch to efficiently estimating the residuals required to correct observational bias.<n> Experiments on synthetic and semi-synthetic benchmarks demonstrate that R-Design significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-03-04T06:52:37Z)
• Exploring Gaps in the APS: Direct Minimal Pair Analysis in LLM Syntactic Assessments [9.161468569386708]
  This paper argues that the direct minimal pair approach offers greater diagnostic transparency.<n>We show GPT-2 succeeds across all four tested conditions, indicating robust knowledge of filler-gap licensing principles.
  arXiv  Detail & Related papers  (2025-10-07T15:03:09Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• ChatGPT-4 in the Turing Test: A Critical Analysis [0.0]
  This paper critically examines the recent publication "ChatGPT-4 in the Turing Test" by Restrepo Echavarr'ia (2025) The analysis reveals that the criticisms based on rigid criteria and limited experimental data are not fully justified.
  arXiv  Detail & Related papers  (2025-03-09T10:43:17Z)
• Using ChatGPT to Score Essays and Short-Form Constructed Responses [0.0]
  Investigation focused on various prediction models, including linear regression, random forest, gradient boost, and boost. ChatGPT's performance was evaluated against human raters using quadratic weighted kappa (QWK) metrics. Study concludes that ChatGPT can complement human scoring but requires additional development to be reliable for high-stakes assessments.
  arXiv  Detail & Related papers  (2024-08-18T16:51:28Z)
• Pragmatic inference of scalar implicature by LLMs [0.0]
  This study investigates how Large Language Models (LLMs) engage in pragmatic inference of implicature, such as some. BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model. GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model.
  arXiv  Detail & Related papers  (2024-08-13T06:52:29Z)
• Graph Stochastic Neural Process for Inductive Few-shot Knowledge Graph Completion [63.68647582680998]
  We focus on a task called inductive few-shot knowledge graph completion (I-FKGC) Inspired by the idea of inductive reasoning, we cast I-FKGC as an inductive reasoning problem. We present a neural process-based hypothesis extractor that models the joint distribution of hypothesis, from which we can sample a hypothesis for predictions. In the second module, based on the hypothesis, we propose a graph attention-based predictor to test if the triple in the query set aligns with the extracted hypothesis.
  arXiv  Detail & Related papers  (2024-08-03T13:37:40Z)
• Prompting or Fine-tuning? Exploring Large Language Models for Causal Graph Validation [0.0]
  This study explores the capability of Large Language Models to evaluate causality in causal graphs. Our study compares two approaches: (1) prompting-based method for zero-shot and few-shot causal inference and, (2) fine-tuning language models for the causal relation prediction task.
  arXiv  Detail & Related papers  (2024-05-29T09:06:18Z)
• Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
  This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations. We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate. Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
  arXiv  Detail & Related papers  (2024-05-27T13:26:34Z)
• Predictive Churn with the Set of Good Models [61.00058053669447]
  This paper explores connections between two seemingly unrelated concepts of predictive inconsistency. The first, known as predictive multiplicity, occurs when models that perform similarly produce conflicting predictions for individual samples. The second concept, predictive churn, examines the differences in individual predictions before and after model updates.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• SOUL: Towards Sentiment and Opinion Understanding of Language [96.74878032417054]
  We propose a new task called Sentiment and Opinion Understanding of Language (SOUL) SOUL aims to evaluate sentiment understanding through two subtasks: Review (RC) and Justification Generation (JG)
  arXiv  Detail & Related papers  (2023-10-27T06:48:48Z)
• A Double Machine Learning Approach to Combining Experimental and Observational Data [59.29868677652324]
  We propose a double machine learning approach to combine experimental and observational studies. Our framework tests for violations of external validity and ignorability under milder assumptions.
  arXiv  Detail & Related papers  (2023-07-04T02:53:11Z)
• OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models [48.412284346337344]
  We conduct a thorough investigation into the reasoning capabilities of Large Language Models (LLMs) Our study entails finetuning three different sizes of Open Pretrained Transformers (OPT) We then evaluate all models on 57 out-of-domain tasks drawn from the SUPER-NATURALINSTRUCTIONS benchmark.
  arXiv  Detail & Related papers  (2023-05-19T20:58:22Z)
• An Empirical Study on the Language Modal in Visual Question Answering [31.692905677913068]
  Generalization beyond in-domain experience to out-of-distribution data is of paramount significance in the AI domain. This paper attempts to provide new insights into the influence of language modality on VQA performance.
  arXiv  Detail & Related papers  (2023-05-17T11:56:40Z)
• Insights into performance evaluation of com-pound-protein interaction prediction methods [0.0]
  Machine learning based prediction of compound-protein interactions (CPIs) is important for drug design, screening and repurposing studies. We have observed a number of fundamental issues in experiment design that lead to over optimistic estimates of model performance.
  arXiv  Detail & Related papers  (2022-01-28T20:07:19Z)
• Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference [5.283529004179579]
  Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) is the task of predicting the entailment relation between a pair of sentences. Models that understand entailment should encode both, the premise and the hypothesis. Experiments by Poliak et al. revealed a strong preference of these models towards patterns observed only in the hypothesis.
  arXiv  Detail & Related papers  (2021-01-19T01:08:06Z)
• Correct block-design experiments mitigate temporal correlation bias in EEG classification [68.85562949901077]
  We show that the main claim in [1] is drastically overstated and their other analyses are seriously flawed by wrong methodological choices. We investigate the influence of EEG temporal correlation on classification accuracy by testing the same models in two additional experimental settings.
  arXiv  Detail & Related papers  (2020-11-25T22:25:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.