PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation

• URL: http://arxiv.org/abs/2607.24353v1
• Date: Mon, 27 Jul 2026 12:31:32 GMT
• Title: PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation
• Abstract summary: We propose PRISM, a Prompt Refinement optimisation framework via Image-grounded Self-rewarding Mechanism.<n>PRISM closes the prompt-image-feedback loop by interpreting generated images with structured visual diagnosis.<n>Experiments show that PRISM improves holistic image quality and fine-grained semantic alignment.
• Score: 6.601249008729316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image generation models can synthesize high-quality images from natural language descriptions, but their performance remains highly sensitive to prompt formulation. Existing prompt optimization methods mainly rely on text-side rewriting, prompt expansion, or external reward signals, offering limited image-grounded diagnosis and weak support for learning reusable optimisation policies. In this paper, we propose PRISM, a Prompt Refinement framework via Image-grounded Self-rewarding Mechanism. PRISM closes the prompt-image-feedback loop by interpreting generated images with structured visual diagnosis and scoring them along semantic consistency, aesthetic quality, and human preference alignment. It first initializes a unified VLM through multi-task supervised fine-tuning, and then improves the prompt policy via self-rewarding optimization with a hybrid ideal-point and Chebyshev reward. Extensive experiments show that PRISM improves holistic image quality and fine-grained semantic alignment, while providing interpretable feedback for targeted prompt refinement. The code is available at https://anonymous.4open.science/r/PRISM-FF81.

Related papers

• PROVE: Training-Free Prompt Recovery using Verifiable Evidence [5.5813258856679555]
  We introduce PROVE (Prompt Recovery with Verified Evidence), a training-free, black-box prompt inversion attack.<n>It reconstructs prompts by composing verifiable scene descriptions rather than optimizing token sequences.<n>The resulting prompts are fully auditable, with every recovered claim grounded in explicit image evidence.
  arXiv  Detail & Related papers  (2026-08-13T18:08:00Z)
• HydraPrompt: An Adaptive and Asymmetric Framework of Vision-Language Models for Synthetic Image Detection [52.11418741192251]
  We propose an asymmetric prompting framework that adjusts the category centers by aligning with fine-grained image cues.<n>Specifically, we introduce a single set of prompts to capture the consistent representative patterns, which serves as a unified anchor for real content.<n>While (2) for fake category, we construct sample-adaptive prompts that specialize in capturing diverse cues from different samples, enabling adaptive modeling of forgery image variations.
  arXiv  Detail & Related papers  (2026-05-26T01:20:18Z)
• CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning [65.10059440725041]
  Visual-Language Models (VLMs) have achieved remarkable progress in image captioning, visual question answering, and visual reasoning.<n>They remain prone to vision-language misalignment, often producing overly generic or hallucinated descriptions.<n>Existing approaches address this via instruction tuning-requiring costly, large-scale annotated datasets or via complex test-time frameworks for caption refinement.<n>In this work, we revisit image-text alignment through the lens of cycle consistency.
  arXiv  Detail & Related papers  (2026-03-18T20:57:31Z)
• PEO: Training-Free Aesthetic Quality Enhancement in Pre-Trained Text-to-Image Diffusion Models with Prompt Embedding Optimization [42.698386517788606]
  This paper introduces a novel approach to aesthetic quality improvement in pre-trained text-to-image diffusion models when given a simple prompt.<n>Our method, dubbed Prompt Embedding Optimization (PEO), leverages a pre-trained text-to-image diffusion model as a backbone.<n>We achieve this by a tripartite objective function that improves the aesthetic fidelity of the generated image, ensures adherence to the optimized text embedding, and minimal divergence from the initial prompt.
  arXiv  Detail & Related papers  (2025-10-02T22:12:36Z)
• VisualPrompter: Prompt Optimization with Visual Feedback for Text-to-Image Synthesis [15.392482488365955]
  VisualPrompter is a training-free prompt engineering framework that refines user inputs to model-preferred sentences.<n>Our framework achieves new state-of-the-art performance on multiple benchmarks for text-image alignment evaluation.
  arXiv  Detail & Related papers  (2025-06-29T08:24:39Z)
• AlignGen: Boosting Personalized Image Generation with Cross-Modality Prior Alignment [74.47138661595584]
  We propose AlignGen, a Cross-Modality Prior Alignment mechanism for personalized image generation.<n>We show that AlignGen outperforms existing zero-shot methods and even surpasses popular test-time optimization approaches.
  arXiv  Detail & Related papers  (2025-05-28T02:57:55Z)
• FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting [18.708185548091716]
  FRAP is a simple, yet effective approach based on adaptively adjusting the per-token prompt weights.<n>We show FRAP generates images with significantly higher prompt-image alignment to prompts from complex datasets.<n>We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image alignment.
  arXiv  Detail & Related papers  (2024-08-21T15:30:35Z)
• Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers [57.62831463679979]
  We present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion.<n>We find that focusing on the CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the similarity between ground truth image and the image generated by the inverted prompts.
  arXiv  Detail & Related papers  (2024-08-12T21:35:59Z)
• Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis [3.783530340696776]
  This study proposes a Multi-Agent framework to optimize input prompts for text-to-image generation models. A professional prompts database serves as a benchmark to guide the instruction modifier towards generating high-caliber prompts. Preliminary ablation studies highlight the effectiveness of various system components and suggest areas for future improvements.
  arXiv  Detail & Related papers  (2024-06-13T00:33:29Z)
• Dynamic Prompt Optimizing for Text-to-Image Generation [63.775458908172176]
  We introduce the textbfPrompt textbfAuto-textbfEditing (PAE) method to improve text-to-image generative models. We employ an online reinforcement learning strategy to explore the weights and injection time steps of each word, leading to the dynamic fine-control prompts.
  arXiv  Detail & Related papers  (2024-04-05T13:44:39Z)
• Iterative Prompt Learning for Unsupervised Backlit Image Enhancement [86.90993077000789]
  We propose a novel unsupervised backlit image enhancement method, abbreviated as CLIP-LIT. We show that the open-world CLIP prior aids in distinguishing between backlit and well-lit images. Our method alternates between updating the prompt learning framework and enhancement network until visually pleasing results are achieved.
  arXiv  Detail & Related papers  (2023-03-30T17:37:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.