Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images

• URL: http://arxiv.org/abs/2404.13784v1
• Date: Sun, 21 Apr 2024 21:30:17 GMT
• Title: Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images
• Authors: Ali Naseh, Katherine Thai, Mohit Iyyer, Amir Houmansadr,
• Abstract summary: This paper studies the possibility of employing multi-modal models with enhanced visual understanding to mimic the outputs of platforms like DALL-E 3 and Midjourney. We create prompts that generate images similar to those available in marketplaces and from premium stock image providers, yet at a markedly lower expense. Our findings, supported by both automated metrics and human assessment, reveal that comparable visual content can be produced for a fraction of the prevailing market prices.
• Score: 45.302905684461905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the digital imagery landscape rapidly evolving, image stocks and AI-generated image marketplaces have become central to visual media. Traditional stock images now exist alongside innovative platforms that trade in prompts for AI-generated visuals, driven by sophisticated APIs like DALL-E 3 and Midjourney. This paper studies the possibility of employing multi-modal models with enhanced visual understanding to mimic the outputs of these platforms, introducing an original attack strategy. Our method leverages fine-tuned CLIP models, a multi-label classifier, and the descriptive capabilities of GPT-4V to create prompts that generate images similar to those available in marketplaces and from premium stock image providers, yet at a markedly lower expense. In presenting this strategy, we aim to spotlight a new class of economic and security considerations within the realm of digital imagery. Our findings, supported by both automated metrics and human assessment, reveal that comparable visual content can be produced for a fraction of the prevailing market prices ($0.23 - $0.27 per image), emphasizing the need for awareness and strategic discussions about the integrity of digital media in an increasingly AI-integrated landscape. Our work also contributes to the field by assembling a dataset consisting of approximately 19 million prompt-image pairs generated by the popular Midjourney platform, which we plan to release publicly.

Related papers

• CTR-Driven Advertising Image Generation with Multimodal Large Language Models [53.40005544344148]
  We explore the use of Multimodal Large Language Models (MLLMs) for generating advertising images by optimizing for Click-Through Rate (CTR) as the primary objective. To further improve the CTR of generated images, we propose a novel reward model to fine-tune pre-trained MLLMs through Reinforcement Learning (RL) Our method achieves state-of-the-art performance in both online and offline metrics.
  arXiv  Detail & Related papers  (2025-02-05T09:06:02Z)
• Generative AI for Vision: A Comprehensive Study of Frameworks and Applications [0.0]
  Generative AI is transforming image synthesis, enabling the creation of high-quality, diverse, and photorealistic visuals. This work presents a structured classification of image generation techniques based on the nature of the input. We highlight key frameworks including DALL-E, ControlNet, and DeepSeek Janus-Pro, and address challenges such as computational costs, data biases, and output alignment with user intent.
  arXiv  Detail & Related papers  (2025-01-29T22:42:05Z)
• PAID: A Framework of Product-Centric Advertising Image Design [31.08944590096747]
  We propose a novel framework called Product-Centric Advertising Image Design (PAID) It consists of four sequential stages to highlight product foregrounds and taglines while achieving overall image aesthetics. To support the PAID framework, we create corresponding datasets with over 50,000 labeled images.
  arXiv  Detail & Related papers  (2025-01-24T08:21:35Z)
• Grounding Descriptions in Images informs Zero-Shot Visual Recognition [47.66166611138081]
  We propose GRAIN, a new pretraining strategy aimed at aligning representations at both fine and coarse levels simultaneously. We demonstrate the enhanced zero-shot performance of our model compared to current state-of-the art methods.
  arXiv  Detail & Related papers  (2024-12-05T18:52:00Z)
• ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images [1.534667887016089]
  This paper presents an innovative approach called ENCLIP, for enhancing the performance of the Contrastive Language-Image Pretraining (CLIP) model. It focuses on addressing the challenges posed by limited data availability and low-quality images.
  arXiv  Detail & Related papers  (2024-11-25T05:15:38Z)
• Chaining text-to-image and large language model: A novel approach for generating personalized e-commerce banners [8.508453886143677]
  We demonstrate the use of text-to-image models for generating personalized web banners for online shoppers. The novelty in this approach lies in converting users' interaction data to meaningful prompts without human intervention. Our results show that the proposed approach can create high-quality personalized banners for users.
  arXiv  Detail & Related papers  (2024-02-28T07:56:04Z)
• Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual Concept Understanding [36.01657852250117]
  Let's Go Shopping dataset is a large-scale public dataset with 15 million image-caption pairs from publicly available e-commerce websites. Our experiments show that the classifiers trained on existing benchmark datasets do not readily generalize to e-commerce data. LGS enables image-captioning models to generate richer captions and helps text-to-image generation models achieve e-commerce style transfer.
  arXiv  Detail & Related papers  (2024-01-09T14:24:29Z)
• Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning [93.6645991946674]
  We introduce panoramic panoptic segmentation, as the most holistic scene understanding. A complete surrounding understanding provides a maximum of information to a mobile agent. We propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain.
  arXiv  Detail & Related papers  (2022-06-21T20:07:15Z)
• There is a Time and Place for Reasoning Beyond the Image [63.96498435923328]
  Images often more significant than only the pixels to human eyes, as we can infer, associate, and reason with contextual information from other sources to establish a more complete picture. We introduce TARA: a dataset with 16k images with their associated news, time and location automatically extracted from New York Times (NYT), and an additional 61k examples as distant supervision from WIT. We show that there exists a 70% gap between a state-of-the-art joint model and human performance, which is slightly filled by our proposed model that uses segment-wise reasoning, motivating higher-level vision-language joint models that
  arXiv  Detail & Related papers  (2022-03-01T21:52:08Z)
• Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings [63.79979145520512]
  We explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. We propose a novel Multi-Modality Multi-Head Attention (M3H-Att) to capture the intricate cross-media interactions. Our model significantly outperforms the previous state of the art based on traditional attention networks.
  arXiv  Detail & Related papers  (2020-11-03T08:44:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.