MetaGAI: A Large-Scale and High-Quality Benchmark for Generative AI Model and Data Card Generation

• URL: http://arxiv.org/abs/2604.23539v1
• Date: Sun, 26 Apr 2026 05:24:03 GMT
• Title: MetaGAI: A Large-Scale and High-Quality Benchmark for Generative AI Model and Data Card Generation
• Abstract summary: We introduce MetaGAI, a comprehensive benchmark comprising 2,541 verified document triplets constructed through semantic triangulation of academic papers, GitHub repositories, and Hugging Face artifacts.<n>Unlike prior single-source datasets, MetaGAI employs a multi-agent framework with specialized Retriever, Generator, and Editor agents, validated through four-dimensional human-in-the-loop assessment.
• Score: 7.0330781653125145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid proliferation of Generative AI necessitates rigorous documentation standards for transparency and governance. However, manual creation of Model and Data Cards is not scalable, while automated approaches lack large-scale, high-fidelity benchmarks for systematic evaluation. We introduce MetaGAI, a comprehensive benchmark comprising 2,541 verified document triplets constructed through semantic triangulation of academic papers, GitHub repositories, and Hugging Face artifacts. Unlike prior single-source datasets, MetaGAI employs a multi-agent framework with specialized Retriever, Generator, and Editor agents, validated through four-dimensional human-in-the-loop assessment, including human evaluation of editor-refined ground truth. We establish a robust evaluation protocol combining automated metrics with validated LLM-as-a-Judge frameworks. Extensive analysis reveals that sparse Mixture-of-Experts architectures achieve superior cost-quality efficiency, while a fundamental trade-off exists between faithfulness and completeness. MetaGAI provides a foundational testbed for benchmarking, training, and analyzing automated Model and Data Card generation methods at scale. Our data and code are available at: https://github.com/haoxuan-unt2024/MetaGAI-Benchmark.

Related papers

• The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systems [3.7149543017244238]
  This study empirically evaluates Retrieval-Augmented Generation (RAG) dependent architectures for the generation of files for GitHub repositories.<n>Results indicate a critical architectural trade-off: the Single-Agent pipeline achieves lexical quality comparable to MAS while reducing token consumption by 86% and operating at twice the speed.
  arXiv  Detail & Related papers  (2026-06-29T16:29:31Z)
• VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation [65.0648741395158]
  VGA-Bench is a unified benchmark for evaluation of video generation quality and aesthetic quality.<n>We design 1,016 diverse prompts and generate a large-scale dataset of over 60,000 videos using 12 video generation models.<n>Our models achieve reliable alignment with human judgments, offering both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-04-11T09:44:39Z)
• AdaQE-CG: Adaptive Query Expansion for Web-Scale Generative AI Model and Data Card Generation [6.289039465679207]
  We propose AdaQE-CG, an Adaptive Query Expansion for Card Generation framework that combines dynamic information extraction with cross-card knowledge transfer.<n>AdaQE-CG substantially outperforms existing approaches, exceeds human-authored data cards, and approaches human-level quality for model cards.<n>We introduce MetaGAI-Bench, the first large-scale, expert-annotated benchmark for evaluating GAI documentation.
  arXiv  Detail & Related papers  (2026-03-16T04:02:56Z)
• TokaMark: A Comprehensive Benchmark for MAST Tokamak Plasma Models [56.94569090844015]
  TokaMark is a structured benchmark to evaluate AI models on real experimental data collected from the Mega Ampere Spherical Tokamak (MAST)<n>TokaMark aims to accelerate progress in data-driven AI-based plasma modeling, contributing to the broader goal of achieving sustainable and stable fusion energy.
  arXiv  Detail & Related papers  (2026-02-05T16:49:44Z)
• InfoSynth: Information-Guided Benchmark Synthesis for LLMs [69.80981631587501]
  Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation.<n>Traditional benchmark creation relies on manual human effort, a process that is both expensive and time-consuming.<n>This work introduces Info Synth, a novel framework for automatically generating and evaluating reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-01-02T05:26:27Z)
• Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation [4.044540605397838]
  Auto-BenchmarkCard is a workflow for generating validated descriptions of AI benchmarks.<n> Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains.
  arXiv  Detail & Related papers  (2025-12-10T12:09:44Z)
• RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering [50.42577862494645]
  We present RAG-IGBench, a benchmark designed to evaluate the task of Interleaved Generation based on Retrieval-Augmented Generation (RAG-IG) in open-domain question answering.<n>RAG-IG integrates multimodal large language models (MLLMs) with retrieval mechanisms, enabling the models to access external image-text information for generating coherent multimodal content.
  arXiv  Detail & Related papers  (2025-10-11T03:06:39Z)
• ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links [57.514511353084565]
  We introduce a new domain-agnostic framework for selecting a best-performing approach and annotating cross-document links.<n>We apply our framework in two distinct domains -- peer review and news.<n>The resulting novel datasets lay foundation for numerous cross-document tasks like media framing and peer review.
  arXiv  Detail & Related papers  (2025-09-01T11:32:24Z)
• ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation [51.297873393639456]
  ArtifactsBench is a framework for automated visual code generation evaluation.<n>Our framework renders each generated artifact and captures its dynamic behavior through temporal screenshots.<n>We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading Large Language Models.
  arXiv  Detail & Related papers  (2025-07-07T12:53:00Z)
• AutoGEEval++: A Multi-Level and Multi-Geospatial-Modality Automated Evaluation Framework for Large Language Models in Geospatial Code Generation on Google Earth Engine [9.161127232765063]
  This study presents AutoGEEval++, and the first automated assessment system for large language models (LLMs) generating geospatial code on Google Earth Engine (GEE)<n>Built on the GEE Python API, AutoGEEval++ features a benchmark dataset-AutoGEEval++-Bench with 6,365 test cases across 26 data types and three task categories: unit, combo, and theme tests.<n>Using AutoGEEval++, we evaluate 24 state-of-the-art LLMs, including general-purpose, reasoning-enhanced, code-centric, and geoscience-specific models.
  arXiv  Detail & Related papers  (2025-06-12T05:42:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.