Compliance Rating Scheme: A Data Provenance Framework for Generative AI Datasets

• URL: http://arxiv.org/abs/2512.21775v1
• Date: Thu, 25 Dec 2025 20:13:46 GMT
• Title: Compliance Rating Scheme: A Data Provenance Framework for Generative AI Datasets
• Authors: Matyas Bohacek, Ignacio Vilanova Echavarri,
• Abstract summary: We introduce the Compliance Rating Scheme (CRS), a framework designed to evaluate dataset compliance with critical transparency, accountability, and security principles.<n>We also release an open-source Python library built around data provenance technology to implement this framework.
• Score: 2.707154152696381
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Generative Artificial Intelligence (GAI) has experienced exponential growth in recent years, partly facilitated by the abundance of large-scale open-source datasets. These datasets are often built using unrestricted and opaque data collection practices. While most literature focuses on the development and applications of GAI models, the ethical and legal considerations surrounding the creation of these datasets are often neglected. In addition, as datasets are shared, edited, and further reproduced online, information about their origin, legitimacy, and safety often gets lost. To address this gap, we introduce the Compliance Rating Scheme (CRS), a framework designed to evaluate dataset compliance with critical transparency, accountability, and security principles. We also release an open-source Python library built around data provenance technology to implement this framework, allowing for seamless integration into existing dataset-processing and AI training pipelines. The library is simultaneously reactive and proactive, as in addition to evaluating the CRS of existing datasets, it equally informs responsible scraping and construction of new datasets.

Related papers

• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation [38.14943360647566]
  Data contamination poses a significant challenge to the fairness of LLM evaluations in natural language processing tasks.<n>We propose textbfCoreEval, a strategy for automatically updating data with real-world knowledge.
  arXiv  Detail & Related papers  (2025-11-24T08:44:29Z)
• Generative Data Refinement: Just Ask for Better Data [19.774236070314963]
  Training datasets now grow faster than the rate at which new data is indexed on the web.<n>Much more data exists as user-generated content that is not publicly indexed, but incorporating such data comes with considerable risks.<n>We introduce a framework, Generative Data Refinement (GDR), for using pretrained generative models to transform a dataset with undesirable content into a refined dataset.
  arXiv  Detail & Related papers  (2025-09-10T14:49:12Z)
• Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems [18.62773754004561]
  We show that using public question and answer (Q&A) datasets to assess retrieval performance can lead to non-optimal systems design.<n>We propose solutions based on the characterization of RAG datasets through labels and through label-targeted data generation.
  arXiv  Detail & Related papers  (2024-11-29T13:57:07Z)
• Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning [3.623224034411137]
  offline multi-agent reinforcement learning (MARL) is an exciting direction of research that uses static datasets to find optimal control policies for multi-agent systems. Though the field is by definition data-driven, efforts have thus far neglected data in their drive to achieve state-of-the-art results. We show how the majority of works generate their own datasets without consistent methodology and provide sparse information about the characteristics of these datasets.
  arXiv  Detail & Related papers  (2024-09-18T14:13:24Z)
• DataGen: Unified Synthetic Dataset Generation via Large Language Models [88.16197692794707]
  DataGen is a comprehensive framework designed to produce diverse, accurate, and highly controllable datasets.<n>To augment data diversity, DataGen incorporates an attribute-guided generation module and a group checking feature.<n>Extensive experiments demonstrate the superior quality of data generated by DataGen.
  arXiv  Detail & Related papers  (2024-06-27T07:56:44Z)
• On Responsible Machine Learning Datasets with Fairness, Privacy, and Regulatory Norms [56.119374302685934]
  There have been severe concerns over the trustworthiness of AI technologies. Machine and deep learning algorithms depend heavily on the data used during their development. We propose a framework to evaluate the datasets through a responsible rubric.
  arXiv  Detail & Related papers  (2023-10-24T14:01:53Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• DataPerf: Benchmarks for Data-Centric AI Development [81.03754002516862]
  DataPerf is a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We provide an open, online platform with multiple rounds of challenges to support this iterative development. The benchmarks, online evaluation platform, and baseline implementations are open source.
  arXiv  Detail & Related papers  (2022-07-20T17:47:54Z)
• The Problem of Zombie Datasets:A Framework For Deprecating Datasets [55.878249096379804]
  We examine the public afterlives of several prominent datasets, including ImageNet, 80 Million Tiny Images, MS-Celeb-1M, Duke MTMC, Brainwash, and HRT Transgender. We propose a dataset deprecation framework that includes considerations of risk, mitigation of impact, appeal mechanisms, timeline, post-deprecation protocol, and publication checks.
  arXiv  Detail & Related papers  (2021-10-18T20:13:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.