Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use

• URL: http://arxiv.org/abs/2609.15624v1
• Date: Mon, 14 Sep 2026 14:16:18 GMT
• Title: Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use
• Abstract summary: Researchers assessing competent generative-AI use at work must choose among self-reports, objective tests, and measures of oversight and reliance.<n>We conducted a structured, seeded review of 24 focal empirical publications.<n>We found no validated individual-level instrument in the focal corpus that tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Researchers assessing competent generative-AI use at work must choose among self-reports, objective tests, and measures of oversight and reliance. We conducted a structured, seeded review of 24 focal empirical publications, starting from the 2024 COSMIN-based review and adding a targeted update through 17 August 2026. We grouped the measures into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. In an exploratory meta-analysis, we pooled three direct subjective-objective correlations from one research program (REML r = .055; Hartung-Knapp 95% CI [-.047, .156]; combined reported N = 2,765). We could not resolve a discrepancy between the largest study's reported correlation and p-value, leaving its weight uncertain. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gave r = .079 (95% CI [-.025, .181]). This sensitivity analysis concerns a broader comparison. From this small evidence base, we cannot establish a population correlation, validate workplace cutoffs, or justify substituting self-ratings for performance scores. We identified tests of foundation knowledge (AICOS-S and GLAT) and measures of verification, reliance, trust, and dependency. We found no validated individual-level instrument in the focal corpus that tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure; some cover subsets. We propose a four-layer workplace battery with non-compensatory decision rules, but have not tested its thresholds or whether it improves on other assessment approaches.

Related papers

• RISED: A Pre-Deployment Safety Evaluation Framework for Clinical AI Decision-Support Systems [0.0]
  We propose a five-dimension pre-deployment evaluation covering Reliability, Inclusivity, Sensitivity, Equity, and Deployability.<n> RISED is released as an open-source Python package that supplies the quantitative verdicts existing clinical AI reporting standards require.
  arXiv  Detail & Related papers  (2026-05-13T02:17:13Z)
• MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills [7.84691221924879]
  This study developed and preliminarily evaluated a domain-specific audit framework for medical research agent skills.<n>We evaluated 75 skills across five medical research categories (15 per category)<n>The mean consensus quality score was 72.4 (SD = 13.0); 57.3% of skills fell below the Limited Release threshold.
  arXiv  Detail & Related papers  (2026-04-22T11:01:48Z)
• Eligibility-Aware Evidence Synthesis: An Agentic Framework for Clinical Trial Meta-Analysis [49.83108591873481]
  EligMeta is an agentic framework that integrates automated trial discovery with eligibility-aware meta-analysis.<n>The framework structures eligibility criteria and computes similarity-based study weights reflecting population alignment between target trials.
  arXiv  Detail & Related papers  (2026-04-03T03:18:50Z)
• Can Large Language Models Detect Methodological Flaws? Evidence from Gesture Recognition for UAV-Based Rescue Operation Based on Deep Learning [0.0]
  We analyze a gesture-recognition paper reporting near-perfect accuracy on a small, human-centered dataset.<n>We first show that the evaluation protocol is consistent with subject-level data leakage due to non-independent training and test splits.<n>We then assess whether this flaw can be detected independently by six state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-03-23T11:39:16Z)
• Nonstandard Errors in AI Agents [6.890249567932368]
  We study whether state-of-the-art AI coding agents, given the same data and research question, produce the same empirical results.<n>We find that AI agents exhibit sizable textitnonstandard errors (NSEs), that is, uncertainty from agent-to-agent variation in analytical choices.<n>These findings have implications for the growing use of AI in automated policy evaluation and empirical research.
  arXiv  Detail & Related papers  (2026-03-17T16:21:22Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems [19.880569341968023]
  Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety.<n>We propose a retrieval-augmented multi-agent framework designed to automate the generation of instance-specific evaluation rubrics.
  arXiv  Detail & Related papers  (2026-01-21T16:40:41Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look [52.114284476700874]
  This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed. We find that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits.
  arXiv  Detail & Related papers  (2024-11-13T01:12:35Z)
• Assaying Out-Of-Distribution Generalization in Transfer Learning [103.57862972967273]
  We take a unified view of previous work, highlighting message discrepancies that we address empirically. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting.
  arXiv  Detail & Related papers  (2022-07-19T12:52:33Z)
• Reinforcement Learning with Heterogeneous Data: Estimation and Inference [84.72174994749305]
  We introduce the K-Heterogeneous Markov Decision Process (K-Hetero MDP) to address sequential decision problems with population heterogeneity. We propose the Auto-Clustered Policy Evaluation (ACPE) for estimating the value of a given policy, and the Auto-Clustered Policy Iteration (ACPI) for estimating the optimal policy in a given policy class. We present simulations to support our theoretical findings, and we conduct an empirical study on the standard MIMIC-III dataset.
  arXiv  Detail & Related papers  (2022-01-31T20:58:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.