Bridging Psychometric and Content Development Practices with AI: A Community-Based Workflow for Augmenting Hawaiian Language Assessments

• URL: http://arxiv.org/abs/2512.17140v1
• Date: Fri, 19 Dec 2025 00:21:48 GMT
• Title: Bridging Psychometric and Content Development Practices with AI: A Community-Based Workflow for Augmenting Hawaiian Language Assessments
• Abstract summary: The paper presents the design and evaluation of a community-based artificial intelligence (AI) workflow for the Kaiapuni Assessment of Educational Outcomes (K'EO) program.<n>K'EO is the only native language assessment used for federal accountability in the United States.<n>The project explored whether document-grounded language models could ethically and effectively augment human analysis of item performance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents the design and evaluation of a community-based artificial intelligence (AI) workflow developed for the Kaiapuni Assessment of Educational Outcomes (KĀ'EO) program, the only native language assessment used for federal accountability in the United States. The project explored whether document-grounded language models could ethically and effectively augment human analysis of item performance while preserving the cultural and linguistic integrity of the Hawaiian language. Operating under the KĀ'EO AI Policy Framework, the workflow used NotebookLM for cross-document synthesis of psychometric data and Claude 3.5 Sonnet for developer-facing interpretation, with human oversight at every stage. Fifty-eight flagged items across Hawaiian Language Arts, Mathematics, and Science were reviewed during Round 2 of the AI Lab, producing six interpretive briefs that identified systemic design issues such as linguistic ambiguity, Depth-of-Knowledge (DOK) misalignment, and structural overload. The findings demonstrate that AI can serve as an ethically bounded amplifier of human expertise, accelerating analysis while simultaneously prioritizing fairness, human expertise, and cultural authority. This work offers a replicable model for responsible AI integration in Indigenous-language educational measurement.

Related papers

• A Human-in-the-Loop Framework for AI-Assisted Scoring in Large-Scale Writing Assessment [0.0]
  This study presents the design and validation of an AI-assisted scoring framework for written responses in a large-scale national assessment.<n>The proposed approach focuses on short written texts of approximately 150-200 words and incorporates a human-in-the-loop strategy to preserve assessment quality while reducing manual workload.
  arXiv  Detail & Related papers  (2026-09-04T13:47:29Z)
• JieZi: A Large-Scale Expert-Audited Dataset and Benchmark for Ancient Chinese Character Exegesis [43.26094253985753]
  Ancient Chinese Character Exegesis (ACCE) is a vision-language question answering task that models the scholarly exegesis process.<n>ACCE is organized into four progressive levels: basic character identification, glyph-form analysis, meaning exegesis, and diachronic evolution analysis.<n>JieZi-Dataset is the first large-scale, expert-audited VQA training dataset for ACCE, comprising over 500K QA pairs.<n>JieZi-Bench is an evaluation benchmark aligned with the exegesis process, constructed and verified by human experts.
  arXiv  Detail & Related papers  (2026-08-12T07:30:00Z)
• AI Technologies in Language Access: Attitudes Towards AI and the Human Value of Language Access Managers [41.99844472131922]
  This paper reports on the attitudes and perceptions of language access managers towards the AI and the human value in the AI age.<n>The results indicate that language access managers show conditional optimism towards the inevitable AI implementations.
  arXiv  Detail & Related papers  (2026-05-19T01:07:21Z)
• Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education [0.0]
  The rapid adoption of large language models in AI-powered language education has created an urgent need for evaluations that assess pedagogical effectiveness.<n>We introduce L2-Bench, a novel evaluation benchmark grounded in a validated "language learning experience designer" construct.<n>Our methodology integrates pedagogical theory, sociotechnical AI evaluation methods, and operationalizes a hierarchical taxonomy to structure an expert-curated dataset.
  arXiv  Detail & Related papers  (2026-03-20T16:13:03Z)
• Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Stud [2.659655189346942]
  This paper presents a methodology for evaluating AIETs in language models.<n>We selected four AIETs: Model Cards, ALTAI, FactSheets, and Harms Modeling.<n>The evaluation considered the developers' perspective on the AIETs' use and quality in helping to identify ethical considerations about their model.
  arXiv  Detail & Related papers  (2025-12-16T02:43:37Z)
• AIssistant: An Agentic Approach for Human--AI Collaborative Scientific Work on Reviews and Perspectives in Machine Learning [2.464267718050055]
  We present here the first experiments with AIssistant for perspective and review research papers in machine learning.<n>Our system integrates modular tools and agents for literature, section-wise experimentation, citation management, and automatic paper text generation.<n>Despite its effectiveness, we identify key limitations, including hallucinated citations, difficulty adapting to dynamic paper structures, and incomplete integration of multimodal content.
  arXiv  Detail & Related papers  (2025-09-14T15:50:31Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• The next question after Turing's question: Introducing the Grow-AI test [51.56484100374058]
  This study aims to extend the framework for assessing artificial intelligence, called GROW-AI.<n>GROW-AI is designed to answer the question "Can machines grow up?" -- a natural successor to the Turing Test.<n>The originality of the work lies in the conceptual transposition of the process of "growing" from the human world to that of artificial intelligence.
  arXiv  Detail & Related papers  (2025-08-22T10:19:42Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Enhancing AI-Driven Education: Integrating Cognitive Frameworks, Linguistic Feedback Analysis, and Ethical Considerations for Improved Content Generation [0.0]
  This paper synthesizes insights from four related studies to propose a comprehensive framework for enhancing AI-driven educational tools.<n>We integrate cognitive assessment frameworks, linguistic analysis of AI-generated feedback, and ethical design principles to guide the development of effective and responsible AI tools.
  arXiv  Detail & Related papers  (2025-05-01T06:36:21Z)
• Survey on Vision-Language-Action Models [0.2636873872510828]
  This work does not represent original research, but highlights how AI can help automate literature reviews.<n>Future research will focus on developing a structured framework for AI-assisted literature reviews.
  arXiv  Detail & Related papers  (2025-02-07T11:56:46Z)
• Harnessing AI for efficient analysis of complex policy documents: a case study of Executive Order 14110 [44.99833362998488]
  Policy documents, such as legislation, regulations, and executive orders, are crucial in shaping society. This study aims to evaluate the potential of AI in streamlining policy analysis and to identify the strengths and limitations of current AI approaches.
  arXiv  Detail & Related papers  (2024-06-10T11:19:28Z)
• Explainable Authorship Identification in Cultural Heritage Applications: Analysis of a New Perspective [48.031678295495574]
  We explore the applicability of existing general-purpose eXplainable Artificial Intelligence (XAI) techniques to AId. In particular, we assess the relative merits of three different types of XAI techniques on three different AId tasks. Our analysis shows that, while these techniques make important first steps towards explainable Authorship Identification, more work remains to be done.
  arXiv  Detail & Related papers  (2023-11-03T20:51:15Z)
• Supporting Human-AI Collaboration in Auditing LLMs with LLMs [33.56822240549913]
  Large language models have been shown to be biased and behave irresponsibly. It is crucial to audit these language models rigorously. Existing auditing tools leverage either or both humans and AI to find failures.
  arXiv  Detail & Related papers  (2023-04-19T21:59:04Z)
• How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics [47.20761880464552]
  generative dialogue modeling is widely seen as a language modeling task. The task demands an agent to have a complex natural language understanding of its input text to carry a meaningful interaction with an user. The automatic metrics used evaluate the quality of the generated text as a proxy to the holistic interaction of the agent.
  arXiv  Detail & Related papers  (2020-08-24T13:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.