AI Literacy as a Key Driver of User Experience in AI-Powered Assessment: Insights from Socratic Mind

• URL: http://arxiv.org/abs/2507.21654v1
• Date: Tue, 29 Jul 2025 10:11:24 GMT
• Title: AI Literacy as a Key Driver of User Experience in AI-Powered Assessment: Insights from Socratic Mind
• Authors: Meryem Yilmaz Soylu, Jeonghyun Lee, Jui-Tse Hung, Christopher Zhang Cui, David A. Joyner,
• Abstract summary: This study examines how students' AI literacy and prior exposure to AI technologies shape their perceptions of Socratic Mind.<n>Data from 309 undergraduates in Computer Science and Business courses were collected.
• Score: 2.0272430076690027
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As Artificial Intelligence (AI) tools become increasingly embedded in higher education, understanding how students interact with these systems is essential to supporting effective learning. This study examines how students' AI literacy and prior exposure to AI technologies shape their perceptions of Socratic Mind, an interactive AI-powered formative assessment tool. Drawing on Self-Determination Theory and user experience research, we analyze relationships among AI literacy, perceived usability, satisfaction, engagement, and perceived learning effectiveness. Data from 309 undergraduates in Computer Science and Business courses were collected through validated surveys. Partial least squares structural equation modeling showed that AI literacy - especially self-efficacy, conceptual understanding, and application skills - significantly predicts usability, satisfaction, and engagement. Usability and satisfaction, in turn, strongly predict perceived learning effectiveness, while prior AI exposure showed no significant effect. These findings highlight that AI literacy, rather than exposure alone, shapes student experiences. Designers should integrate adaptive guidance and user-centered features to support diverse literacy levels, fostering inclusive, motivating, and effective AI-based learning environments.

Related papers

• AI Literacy for Community Colleges: Instructors' Perspectives on Scenario-Based and Interactive Approaches to Teaching AI [0.500208619516796]
  This research category full paper investigates how community college instructors evaluate interactive, no-code AI literacy resources designed for non-STEM learners.<n>As artificial intelligence becomes increasingly integrated into everyday technologies, AI literacy has emerged as a critical skill across disciplines.<n>We developed AI User, an interactive online curriculum that introduces core AI concepts through scenario - based activities set in real - world contexts.
  arXiv  Detail & Related papers  (2025-11-07T15:51:53Z)
• Deploying AI for Signal Processing education: Selected challenges and intriguing opportunities [44.18936398140735]
  Article explores the use of AI tools to facilitate and enhance education.<n> Primers are provided on several core technical issues that arise when using AI in educational settings.<n>The article serves as a resource for researchers and educators seeking to advance AI's role in engineering education.
  arXiv  Detail & Related papers  (2025-09-10T19:19:26Z)
• Bridging Minds and Machines: Toward an Integration of AI and Cognitive Science [48.38628297686686]
  Cognitive Science has profoundly shaped disciplines such as Artificial Intelligence (AI), Philosophy, Psychology, Neuroscience, Linguistics, and Culture.<n>Many breakthroughs in AI trace their roots to cognitive theories, while AI itself has become an indispensable tool for advancing cognitive research.<n>We argue that the future of AI within Cognitive Science lies not only in improving performance but also in constructing systems that deepen our understanding of the human mind.
  arXiv  Detail & Related papers  (2025-08-28T11:26:17Z)
• Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study [10.71612026319996]
  This study analyzed 315 student-AI conversations during a brief, quiz-based scenario across various STEM courses.<n>Students exhibited overall low reliance on AI and many of them could not effectively use AI for learning.<n>Certain behavioral metrics strongly predicted AI reliance, highlighting potential behavioral mechanisms to explain AI adoption.
  arXiv  Detail & Related papers  (2025-08-27T20:00:27Z)
• AI as IA: The use and abuse of artificial intelligence (AI) for human enhancement through intellectual augmentation (IA) [45.88028371034407]
  This paper offers an overview of the prospects and ethics of using AI to achieve human enhancement.<n>We discuss potential ethical problems, namely inadequate performance, safety, coercion and manipulation, privacy, cognitive liberty, authenticity, and fairness.<n>We conclude that while there are very significant technical hurdles to real human enhancement through AI, and significant ethical problems, there are also significant benefits that may realistically be achieved.
  arXiv  Detail & Related papers  (2025-08-18T09:39:11Z)
• Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence [59.07578850674114]
  Sound deductive reasoning is an indisputably desirable aspect of general intelligence.<n>It is well-documented that even the most advanced frontier systems regularly and consistently falter on easily-solvable reasoning tasks.<n>We argue that their unsound behavior is a consequence of the statistical learning approach powering their development.
  arXiv  Detail & Related papers  (2025-06-30T14:37:50Z)
• Evaluating AI-Powered Learning Assistants in Engineering Higher Education: Student Engagement, Ethical Challenges, and Policy Implications [0.2812395851874055]
  This study evaluates the use of the Educational AI Hub, an AI-powered learning framework, in undergraduate civil and environmental engineering courses at a large R1 public university.<n>Students appreciated the AI assistant for its convenience and comfort, with nearly half reporting greater ease in using the AI tool.<n>While most students viewed AI use as ethically acceptable, many expressed uncertainties about institutional policies and apprehension about potential academic misconduct.
  arXiv  Detail & Related papers  (2025-06-06T03:02:49Z)
• Distinguishing Fact from Fiction: Student Traits, Attitudes, and AI Hallucination Detection in Business School Assessment [2.3359837623080613]
  We examine how academic skills, cognitive traits, and AI scepticism influence students' ability to detect factually incorrect AI-generated responses (hallucinations) in a high-stakes assessment at a UK business school.<n>We find that only 20% successfully identified the hallucination, with strong academic performance, interpretive skills thinking, writing proficiency, and AI scepticism emerging as key predictors.
  arXiv  Detail & Related papers  (2025-05-28T18:39:57Z)
• AI in Education: Rationale, Principles, and Instructional Implications [0.0]
  Generative AI, like ChatGPT, can create human-like content, prompting questions about its educational role.<n>The study emphasizes deliberate strategies to ensure AI complements, not replaces, genuine cognitive effort.
  arXiv  Detail & Related papers  (2024-12-02T14:08:07Z)
• Aligning Generalisation Between Humans and Machines [74.120848518198]
  AI technology can support humans in scientific discovery and forming decisions, but may also disrupt democracies and target individuals.<n>The responsible use of AI and its participation in human-AI teams increasingly shows the need for AI alignment.<n>A crucial yet often overlooked aspect of these interactions is the different ways in which humans and machines generalise.
  arXiv  Detail & Related papers  (2024-11-23T18:36:07Z)
• Let people fail! Exploring the influence of explainable virtual and robotic agents in learning-by-doing tasks [45.23431596135002]
  This study compares the effects of classic vs. partner-aware explanations on human behavior and performance during a learning-by-doing task. Results indicated that partner-aware explanations influenced participants differently based on the type of artificial agents involved.
  arXiv  Detail & Related papers  (2024-11-15T13:22:04Z)
• Imagining and building wise machines: The centrality of AI metacognition [78.76893632793497]
  We examine what is known about human wisdom and sketch a vision of its AI counterpart.<n>We argue that AI systems particularly struggle with metacognition.<n>We discuss how wise AI might be benchmarked, trained, and implemented.
  arXiv  Detail & Related papers  (2024-11-04T18:10:10Z)
• Untangling Critical Interaction with AI in Students Written Assessment [2.8078480738404]
  Key challenge exists in ensuring that humans are equipped with the required critical thinking and AI literacy skills. This paper provides a first step toward conceptualizing the notion of critical learner interaction with AI. Using both theoretical models and empirical data, our preliminary findings suggest a general lack of Deep interaction with AI during the writing process.
  arXiv  Detail & Related papers  (2024-04-10T12:12:50Z)
• Toward enriched Cognitive Learning with XAI [44.99833362998488]
  We introduce an intelligent system (CL-XAI) for Cognitive Learning which is supported by artificial intelligence (AI) tools. The use of CL-XAI is illustrated with a game-inspired virtual use case where learners tackle problems to enhance problem-solving skills.
  arXiv  Detail & Related papers  (2023-12-19T16:13:47Z)
• MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies [6.368014180870025]
  The questionnaire should be modular (i.e., including different facets that can be used independently of each other) to be flexibly applicable in professional life. We derived 60 items to represent different facets of AI Literacy according to Ng and colleagues conceptualisation of AI literacy. Additional 12 items to represent psychological competencies such as problem solving, learning, and emotion regulation in regard to AI.
  arXiv  Detail & Related papers  (2023-02-18T12:35:55Z)
• Building Bridges: Generative Artworks to Explore AI Ethics [56.058588908294446]
  In recent years, there has been an increased emphasis on understanding and mitigating adverse impacts of artificial intelligence (AI) technologies on society. A significant challenge in the design of ethical AI systems is that there are multiple stakeholders in the AI pipeline, each with their own set of constraints and interests. This position paper outlines some potential ways in which generative artworks can play this role by serving as accessible and powerful educational tools.
  arXiv  Detail & Related papers  (2021-06-25T22:31:55Z)
• An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
  This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
  arXiv  Detail & Related papers  (2021-05-07T12:01:31Z)
• Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
  The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal. In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
  arXiv  Detail & Related papers  (2021-01-19T12:23:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.