AI Literacy Assessment Revisited: A Task-Oriented Approach Aligned with Real-world Occupations

• URL: http://arxiv.org/abs/2511.05475v1
• Date: Fri, 07 Nov 2025 18:38:15 GMT
• Title: AI Literacy Assessment Revisited: A Task-Oriented Approach Aligned with Real-world Occupations
• Authors: Christopher Bogart, Aparna Warrier, Arav Agarwal, Ross Higashi, Yufan Zhang, Jesse Flot, Jaromir Savelka, Heather Burte, Majd Sakr,
• Abstract summary: We describe the development of a novel AI literacy assessment instrument, and accompanying formative assessments, in the context of a US Navy robotics training program.<n>We argue that when training people for AI-related work, educators should consider evaluating them with instruments that emphasize highly contextualized practical skills.
• Score: 1.7684337194725703
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As artificial intelligence (AI) systems become ubiquitous in professional contexts, there is an urgent need to equip workers, often with backgrounds outside of STEM, with the skills to use these tools effectively as well as responsibly, that is, to be AI literate. However, prevailing definitions and therefore assessments of AI literacy often emphasize foundational technical knowledge, such as programming, mathematics, and statistics, over practical knowledge such as interpreting model outputs, selecting tools, or identifying ethical concerns. This leaves a noticeable gap in assessing someone's AI literacy for real-world job use. We propose a work-task-oriented assessment model for AI literacy which is grounded in the competencies required for effective use of AI tools in professional settings. We describe the development of a novel AI literacy assessment instrument, and accompanying formative assessments, in the context of a US Navy robotics training program. The program included training in robotics and AI literacy, as well as a competition with practical tasks and a multiple choice scenario task meant to simulate use of AI in a job setting. We found that, as a measure of applied AI literacy, the competition's scenario task outperformed the tests we adopted from past research or developed ourselves. We argue that when training people for AI-related work, educators should consider evaluating them with instruments that emphasize highly contextualized practical skills rather than abstract technical knowledge, especially when preparing workers without technical backgrounds for AI-integrated roles.

Related papers

• AI as Cognitive Amplifier: Rethinking Human Judgment in the Age of Generative AI [0.65268245109828]
  I propose a three-level model of AI engagement.<n>I argue that the transition between levels requires not technical training but development of domain expertise and metacognitive skills.
  arXiv  Detail & Related papers  (2025-10-30T11:55:34Z)
• Look: AI at Work! - Analysing Key Aspects of AI-support at the Work Place [39.146761527401424]
  We look at the areas of AI that the applications are concerned with.<n>This includes the importance of high-quality data for training learning-based systems.<n>In terms of the psychological factors we derive research questions to investigate in the development of AI supported work systems.
  arXiv  Detail & Related papers  (2025-09-02T12:51:23Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems.<n>We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure.<n>Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• Generative AI Literacy: Twelve Defining Competencies [48.90506360377104]
  This paper introduces a competency-based model for generative artificial intelligence (AI) literacy covering essential skills and knowledge areas necessary to interact with generative AI.<n>The competencies range from foundational AI literacy to prompt engineering and programming skills, including ethical and legal considerations.<n>These twelve competencies offer a framework for individuals, policymakers, government officials, and educators looking to navigate and take advantage of the potential of generative AI responsibly.
  arXiv  Detail & Related papers  (2024-11-29T14:55:15Z)
• Imagining and building wise machines: The centrality of AI metacognition [78.76893632793497]
  We examine what is known about human wisdom and sketch a vision of its AI counterpart.<n>We argue that AI systems particularly struggle with metacognition.<n>We discuss how wise AI might be benchmarked, trained, and implemented.
  arXiv  Detail & Related papers  (2024-11-04T18:10:10Z)
• AI Literacy for All: Adjustable Interdisciplinary Socio-technical Curriculum [0.8879149917735942]
  This paper presents a curriculum, "AI Literacy for All," to promote an interdisciplinary understanding of AI. The paper presents four pillars of AI literacy: understanding the scope and technical dimensions of AI, learning how to interact with Gen-AI in an informed and responsible way, the socio-technical issues of ethical and responsible AI, and the social and future implications of AI.
  arXiv  Detail & Related papers  (2024-09-02T13:13:53Z)
• A Multi-Year Grey Literature Review on AI-assisted Test Automation [46.97326049485643]
  Test Automation (TA) techniques are crucial for quality assurance in software engineering but face limitations.<n>Given the prevalent usage of AI in industry, sources of truth are held in grey literature as well as the minds of professionals.<n>This study surveys grey literature to explore how AI is adopted in TA, focusing on the problems it solves, its solutions, and the available tools.
  arXiv  Detail & Related papers  (2024-08-12T15:26:36Z)
• Towards the Terminator Economy: Assessing Job Exposure to AI through LLMs [10.844598404826355]
  One-third of U.S. employment is highly exposed to AI, primarily in high-skill jobs requiring a graduate or postgraduate level of education.<n>Even in high-skill occupations, AI exhibits high variability in task substitution, suggesting that AI and humans complement each other within the same occupation.<n>All results, models, and code are freely available online to allow the community to reproduce our results, compare outcomes, and use our work as a benchmark to monitor AI's progress over time.
  arXiv  Detail & Related papers  (2024-07-27T08:14:18Z)
• MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies [6.368014180870025]
  The questionnaire should be modular (i.e., including different facets that can be used independently of each other) to be flexibly applicable in professional life. We derived 60 items to represent different facets of AI Literacy according to Ng and colleagues conceptualisation of AI literacy. Additional 12 items to represent psychological competencies such as problem solving, learning, and emotion regulation in regard to AI.
  arXiv  Detail & Related papers  (2023-02-18T12:35:55Z)
• Automated Machine Learning: A Case Study on Non-Intrusive Appliance Load Monitoring [81.06807079998117]
  We propose a novel approach to enable Automated Machine Learning (AutoML) for Non-Intrusive Appliance Load Monitoring (NIALM)<n>NIALM offers a cost-effective alternative to smart meters for measuring the energy consumption of electric devices and appliances.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)
• Cybertrust: From Explainable to Actionable and Interpretable AI (AI2) [58.981120701284816]
  Actionable and Interpretable AI (AI2) will incorporate explicit quantifications and visualizations of user confidence in AI recommendations. It will allow examining and testing of AI system predictions to establish a basis for trust in the systems' decision making.
  arXiv  Detail & Related papers  (2022-01-26T18:53:09Z)
• Measuring Ethics in AI with AI: A Methodology and Dataset Construction [1.6861004263551447]
  We propose to use such newfound capabilities of AI technologies to augment our AI measuring capabilities. We do so by training a model to classify publications related to ethical issues and concerns. We highlight the implications of AI metrics, in particular their contribution towards developing trustful and fair AI-based tools and technologies.
  arXiv  Detail & Related papers  (2021-07-26T00:26:12Z)
• Building Bridges: Generative Artworks to Explore AI Ethics [56.058588908294446]
  In recent years, there has been an increased emphasis on understanding and mitigating adverse impacts of artificial intelligence (AI) technologies on society. A significant challenge in the design of ethical AI systems is that there are multiple stakeholders in the AI pipeline, each with their own set of constraints and interests. This position paper outlines some potential ways in which generative artworks can play this role by serving as accessible and powerful educational tools.
  arXiv  Detail & Related papers  (2021-06-25T22:31:55Z)
• An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
  This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
  arXiv  Detail & Related papers  (2021-05-07T12:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.