GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students

• URL: http://arxiv.org/abs/2608.25815v1
• Date: Wed, 26 Aug 2026 13:58:56 GMT
• Title: GenAIT: Development and Validation of an Objective Generative AI Literacy Test for High School Students
• Abstract summary: There is growing international interest in generative AI (GenAI) literacy and its assessment among high school students, but objective assessment in this population remains underdeveloped.<n>This article reports the iterative development and validation of the GenAI Literacy Test (GenAIT), an 18-item multiple-choice test measuring high school students' conceptual knowledge about GenAI.
• Score: 0.5735035463793009
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: There is growing international interest in generative AI (GenAI) literacy and its assessment among high school students, but objective assessment in this population remains underdeveloped. This article reports the iterative development and validation of the GenAI Literacy Test (GenAIT), an 18-item multiple-choice test measuring high school students' conceptual knowledge about GenAI, with content spanning technical, practical, and human-impact domains. Expert review of relevance, clarity, and comprehensiveness provided evidence of content validity. In a large-scale survey of 7432 Estonian high school students, we evaluated the psychometric functioning of the Estonian-language GenAIT using confirmatory factor analysis, classical test theory, and item response theory. Results supported approximate unidimensionality, broadly adequate reliability for group-level research (marginal reliability = .72, KR-20 = .69), and good fit of a three-parameter logistic model (RMSEA = .013, TLI = .987, CFI = .990, SRMSR = .021). Measurement precision was sufficient for the majority of students but varied substantially across the latent trait, with lower precision for lower scoring students. GenAIT is therefore more suitable for group-level research than high-stakes individual classification. GenAIT scores were unrelated to perceived usefulness and perceived ease of use, and negatively associated with LLM use frequency, suggesting that frequent use and favorable perceptions of AI should not be treated as proxies for conceptual understanding.

Related papers

• Is Solving Better Than Evaluating GenAI Solutions? [45.714140455691485]
  This study compares evaluating GenAI-generated solutions with traditional problem solving in a junior-level algorithms course.<n>We found no statistically significant differences between groups in midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions.<n>Students received significantly higher homework scores when evaluating GenAI-generated solutions, but this localized advantage did not translate into downstream gains.
  arXiv  Detail & Related papers  (2026-07-30T02:08:29Z)
• Measuring How Students Rely on Generative AI in Academic Writing: Development and Multi-Source Validation of the Generative AI Reliance Types Scale (GenAI-RTS) [0.0]
  How students rely on generative AI (GenAI) has become a central question for learning, academic integrity, and educational equity.<n>This study developed and validated the GenAI Reliance Types Scale (GenAI-RTS), a 20-item instrument measuring four theoretically derived types of GenAI reliance.
  arXiv  Detail & Related papers  (2026-07-15T19:04:11Z)
• AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot [45.90272517740283]
  We report the first large-scale field deployment of AI-assisted peer review.<n>Every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system.
  arXiv  Detail & Related papers  (2026-04-15T14:51:07Z)
• Exploring Student Perception on Gen AI Adoption in Higher Education: A Descriptive Study [1.978884131103313]
  This study examines how students perceive, use, and evaluate Generative Artificial Intelligence (GenAI)<n>Data were collected through a questionnaire administered to 436 postgraduate Computer Science students at the University of Hertfordshire.
  arXiv  Detail & Related papers  (2026-03-29T17:19:39Z)
• AI Literacy, Safety Awareness, and STEM Career Aspirations of Australian Secondary Students: Evaluating the Impact of Workshop Interventions [38.350232667249095]
  Deepfakes and other forms of synthetic media pose growing safety risks for adolescents.<n>This study evaluates the impact of Day of AI Australia's workshop-based intervention on Australian secondary students.
  arXiv  Detail & Related papers  (2026-01-30T02:55:53Z)
• Predicting ChatGPT Use in Assignments: Implications for AI-Aware Assessment Design [0.0]
  The study reveals that frequent use of ChatGPT for learning new concepts correlates with potential overreliance.<n>We propose discipline-specific guidelines and reimagined assessment strategies to balance innovation with academic rigor.
  arXiv  Detail & Related papers  (2025-08-16T11:09:38Z)
• Evaluating and Mitigating Bias in AI-Based Medical Text Generation [35.24191727599811]
  AI systems may reflect and amplify human bias, and reduce the quality of their performance in historically under-served populations.<n>In this study, we investigate the fairness problem in text generation within the medical field.<n>We propose an algorithm that selectively optimize those underperformed groups to reduce bias.
  arXiv  Detail & Related papers  (2025-04-24T06:10:40Z)
• Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AI [41.96102438774773]
  This work presents the findings from a university-level competition, which challenged participants to design prompts for eliciting biased outputs from GenAI tools. We quantitatively and qualitatively analyze the competition submissions and identify a diverse set of biases in GenAI and strategies employed by participants to induce bias in GenAI.
  arXiv  Detail & Related papers  (2024-10-20T18:44:45Z)
• A Meta-analysis of College Students' Intention to Use Generative Artificial Intelligence [5.13644976086965]
  This study conducted a meta-analysis of 27 empirical studies under an integrated theoretical framework. Main variables are strongly correlated with students' behavioural intention to use GenAI. Gender, notably, only moderated attitudes on students' behavioural intention to use GenAI.
  arXiv  Detail & Related papers  (2024-08-25T15:46:57Z)
• Understanding Student and Academic Staff Perceptions of AI Use in Assessment and Feedback [0.0]
  The rise of Artificial Intelligence (AI) and Generative Artificial Intelligence (GenAI) in higher education necessitates assessment reform. This study addresses a critical gap by exploring student and academic staff experiences with AI and GenAI tools. An online survey collected data from 35 academic staff and 282 students across two universities in Vietnam and one in Singapore.
  arXiv  Detail & Related papers  (2024-06-22T10:25:01Z)
• ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models [53.00812898384698]
  We argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. We propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.
  arXiv  Detail & Related papers  (2024-05-28T22:45:28Z)
• Why is plausibility surprisingly problematic as an XAI criterion? [38.0428570713717]
  Plausibility is usually quantified by metrics of feature localization or feature correlation.<n>Our examination shows that plausibility is invalid to measure explainability, and human explanations are not the ground truth for XAI.<n>Due to the invalidity of measurements and the unethical issues, this paper argues that the community should stop using plausibility as a criterion for the evaluation and optimization of XAI algorithms.
  arXiv  Detail & Related papers  (2023-03-30T20:59:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.