Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government

• URL: http://arxiv.org/abs/2606.20963v1
• Date: Thu, 18 Jun 2026 22:00:25 GMT
• Title: Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government
• Abstract summary: We introduce the Generative Responsible Data Evaluation (GRAIDES) as a lightweight open-source data model for centralising AI observability across popular vendors.<n> Practical blueprints for code, architecture and statistical evaluation are shared as guidance about how to approach generative system assurance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Trust in the application of generative Artificial Intelligence (AI) relies on well-governed measurable evidence of performance and safety. In practice, however, evaluation data is often fragmented across systems, inconsistently structured and difficult to compare. We introduce the Generative Responsible AI Data Evaluation Schema (GRAIDES) as a lightweight open-source data model for centralising AI observability across popular vendors. Practical blueprints for code, architecture and statistical evaluation are shared as guidance about how to approach generative system assurance at the organisational level. Illustrative case study results are reported from Westminster City Council's AI catalogue with a focus on measuring human-model alignment including detecting systematic disagreement between evaluators. By framing evaluations as a data modelling problem, GRAIDES provides a practical pathway toward more consistent and reproducible benchmarking, tuning and assurance activities for generative AI systems.

Related papers

• FLARE-AI: Flaw Reporting for AI [51.64618567247884]
  Flaw reporting for deployed AI systems is fundamental to identifying system failures and improving AI safety.<n>Yet the AI reporting ecosystem is fragmented: researchers who identify flaws often do not know what to report, and groups who receive reports rarely share them with other relevant stakeholders.<n>We audit 12 reporting systems published by AI developers, cybersecurity groups, and AI flaw aggregators.<n>We introduce FLARE-AI, an open-source AI flaw reporting system designed for interoperability with existing systems.
  arXiv  Detail & Related papers  (2026-06-30T12:27:09Z)
• AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration [49.050688065974605]
  Membership Inference Attacks (MIAs) serve as a fundamental auditing tool for evaluating training data leakage in machine learning models.<n>We propose AutoMIA, an agentic framework that reformulates membership inference as an automated process of self-exploration and strategy evolution.
  arXiv  Detail & Related papers  (2026-04-01T15:17:45Z)
• UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities [70.79422099851506]
  We present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems.<n>UDE standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface.
  arXiv  Detail & Related papers  (2026-03-24T13:01:31Z)
• Position: Science of AI Evaluation Requires Item-level Benchmark Data [42.82377343454172]
  We argue that item-level AI benchmark data is essential for establishing a rigorous science of AI evaluation.<n>To catalyze community-wide adoption, we introduce OpenEval, a growing repository of item-level benchmark data.
  arXiv  Detail & Related papers  (2026-02-27T04:31:30Z)
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• An AI Implementation Science Study to Improve Trustworthy Data in a Large Healthcare System [1.6881002551798014]
  This study presents an AI implementation case study within Shriners Childrens (SCs), a large multisite pediatric system.<n>We introduce a Python-based data quality assessment tool compatible with SCs infrastructure, extending OHDsi's R/Java-based Data Quality Dashboard.<n>We also compare systematic and case-specific AI implementation strategies for Craniofacial Microsomia (CFM) using the FHIR standard.
  arXiv  Detail & Related papers  (2025-12-01T14:21:16Z)
• Creative Adversarial Testing (CAT): A Novel Framework for Evaluating Goal-Oriented Agentic AI Systems [0.0]
  Creative Adversarial Testing (CAT) is a novel approach designed to capture and analyze the complex relationship between Agentic AI tasks and the system's intended objectives.<n>We validate the CAT framework through extensive simulation using synthetic interaction data modeled after Alexa+ audio services.<n>Our results demonstrate that the CAT framework provides unprecedented insights into goal-task alignment, enabling more effective optimization and development of Agentic AI systems.
  arXiv  Detail & Related papers  (2025-09-26T23:52:20Z)
• Variance-Bounded Evaluation of Entity-Centric AI Systems Without Ground Truth: Theory and Measurement [0.0]
  We introduce VB-Score, a variance-bounded evaluation framework for entity-centric AI systems.<n> VB-Score enumerates plausible interpretations through constraint relaxation and Monte Carlo sampling.<n>It then evaluates system outputs by their expected success across interpretations, penalized by variance to assess robustness of the system.
  arXiv  Detail & Related papers  (2025-09-26T07:54:38Z)
• Demographic Benchmarking: Bridging Socio-Technical Gaps in Bias Detection [0.0]
  This paper describes how the ITACA AI auditing platform tackles demographic benchmarking when auditing AI recommender systems.<n>The framework serves us as auditors as it allows us to not just measure but establish acceptability ranges for specific performance indicators.<n>Our approach integrates socio-demographic insights directly into AI systems, reducing bias and improving overall performance.
  arXiv  Detail & Related papers  (2025-01-27T12:14:49Z)
• OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain [62.89809156574998]
  We introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain.<n>Our benchmark is characterized by its multi-dimensional evaluation framework.<n>Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets.
  arXiv  Detail & Related papers  (2024-12-17T15:38:42Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.