Benchmarking Generative AI Models for Deep Learning Test Input Generation

• URL: http://arxiv.org/abs/2412.17652v1
• Date: Mon, 23 Dec 2024 15:30:42 GMT
• Title: Benchmarking Generative AI Models for Deep Learning Test Input Generation
• Authors: Maryam, Matteo Biagiola, Andrea Stocco, Vincenzo Riccio,
• Abstract summary: Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets.<n>Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images.<n>We benchmark and combine different GenAI models with TIGs, assessing their effectiveness, efficiency, and quality of the generated test images.
• Score: 6.674615464230326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets. Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images, although these advancements also imply increased complexity and resource demands for training. In this work, we benchmark and combine different GenAI models with TIGs, assessing their effectiveness, efficiency, and quality of the generated test images, in terms of domain validity and label preservation. We conduct an empirical study involving three different GenAI architectures (VAEs, GANs, Diffusion Models), five classification tasks of increasing complexity, and 364 human evaluations. Our results show that simpler architectures, such as VAEs, are sufficient for less complex datasets like MNIST. However, when dealing with feature-rich datasets, such as ImageNet, more sophisticated architectures like Diffusion Models achieve superior performance by generating a higher number of valid, misclassification-inducing inputs.

Related papers

• AI-GenBench: A New Ongoing Benchmark for AI-Generated Image Detection [9.540547388707987]
  Ai-GenBench is a novel benchmark designed to address the need for robust detection of AI-generated images in real-world scenarios. By establishing clear evaluation rules and controlled augmentation strategies, Ai-GenBench enables meaningful comparison of detection methods and scalable solutions.
  arXiv  Detail & Related papers  (2025-04-29T15:41:13Z)
• M3-AGIQA: Multimodal, Multi-Round, Multi-Aspect AI-Generated Image Quality Assessment [65.3860007085689]
  M3-AGIQA is a comprehensive framework for AGI quality assessment. It includes a structured multi-round evaluation mechanism, where intermediate image descriptions are generated. Experiments conducted on multiple benchmark datasets demonstrate that M3-AGIQA achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-02-21T03:05:45Z)
• Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation [69.62857948698436]
  Recent advances in large language models (LLMs) have improved their performance on coding benchmarks. However, improvement is plateauing due to the exhaustion of readily available high-quality data. We propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity.
  arXiv  Detail & Related papers  (2025-02-20T18:32:19Z)
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step [77.86514804787622]
  Chain-of-Thought (CoT) reasoning has been extensively explored in large models to tackle complex understanding tasks. We provide the first comprehensive investigation of the potential of CoT reasoning to enhance autoregressive image generation. We propose the Potential Assessment Reward Model (PARM) and PARM++, specialized for autoregressive image generation.
  arXiv  Detail & Related papers  (2025-01-23T18:59:43Z)
• Detecting the Undetectable: Combining Kolmogorov-Arnold Networks and MLP for AI-Generated Image Detection [0.0]
  This paper presents a novel detection framework adept at robustly identifying images produced by cutting-edge generative AI models. We propose a classification system that integrates semantic image embeddings with a traditional Multilayer Perceptron (MLP)
  arXiv  Detail & Related papers  (2024-08-18T06:00:36Z)
• Improving Interpretability and Robustness for the Detection of AI-Generated Images [6.116075037154215]
  We analyze existing state-of-the-art AIGI detection methods based on frozen CLIP embeddings. We show how to interpret them, shedding light on how images produced by various AI generators differ from real ones.
  arXiv  Detail & Related papers  (2024-06-21T10:33:09Z)
• RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detection [11.265512559447986]
  We introduce RU-AI, a new large-scale multimodal dataset for robust and efficient detection of machine-generated content in text, image and voice.<n>Our dataset is constructed on the basis of three large publicly available datasets: Flickr8K, COCO and Places205.<n>The results reveal that existing models still struggle to achieve accurate and robust classification after training on our dataset.
  arXiv  Detail & Related papers  (2024-06-07T12:58:14Z)
• Large Multi-modality Model Assisted AI-Generated Image Quality Assessment [53.182136445844904]
  We introduce a large Multi-modality model Assisted AI-Generated Image Quality Assessment (MA-AGIQA) model. It uses semantically informed guidance to sense semantic information and extract semantic vectors through carefully designed text prompts. It achieves state-of-the-art performance, and demonstrates its superior generalization capabilities on assessing the quality of AI-generated images.
  arXiv  Detail & Related papers  (2024-04-27T02:40:36Z)
• 3DG: A Framework for Using Generative AI for Handling Sparse Learner Performance Data From Intelligent Tutoring Systems [22.70004627901319]
  We introduce the 3DG framework (3-Dimensional tensor for Densification and Generation), a novel approach combining tensor factorization with advanced generative models. The framework effectively generated scalable, personalized simulations of learning performance.
  arXiv  Detail & Related papers  (2024-01-29T22:34:01Z)
• AI-Generated Images as Data Source: The Dawn of Synthetic Era [61.879821573066216]
  generative AI has unlocked the potential to create synthetic images that closely resemble real-world photographs. This paper explores the innovative concept of harnessing these AI-generated images as new data sources. In contrast to real data, AI-generated data exhibit remarkable advantages, including unmatched abundance and scalability.
  arXiv  Detail & Related papers  (2023-10-03T06:55:19Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• Domain Generalization via Ensemble Stacking for Face Presentation Attack Detection [4.61143637299349]
  Face Presentation Attack Detection (PAD) plays a pivotal role in securing face recognition systems against spoofing attacks. This work proposes a comprehensive solution that combines synthetic data generation and deep ensemble learning. Experimental results on four datasets demonstrate low half total error rates (HTERs) on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-05T16:44:36Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.