PRJ: Perception-Retrieval-Judgement for Generated Images

• URL: http://arxiv.org/abs/2506.03683v1
• Date: Wed, 04 Jun 2025 08:13:53 GMT
• Title: PRJ: Perception-Retrieval-Judgement for Generated Images
• Authors: Qiang Fu, Zonglei Jing, Zonghao Ying, Xiaoqian Li,
• Abstract summary: Perception-Retrieval-Judgement (PRJ) is a framework that models toxicity detection as a structured reasoning process.<n>PRJ follows a three-stage design: it first transforms an image into descriptive language (perception), then retrieves external knowledge related to harm categories and traits (retrieval), and finally evaluates toxicity based on legal or normative rules (judgement)<n> Experiments show that PRJ surpasses existing safety checkers in detection accuracy and robustness while uniquely supporting structured category-level toxicity interpretation.
• Score: 6.940819432582308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid progress of generative AI has enabled remarkable creative capabilities, yet it also raises urgent concerns regarding the safety of AI-generated visual content in real-world applications such as content moderation, platform governance, and digital media regulation. This includes unsafe material such as sexually explicit images, violent scenes, hate symbols, propaganda, and unauthorized imitations of copyrighted artworks. Existing image safety systems often rely on rigid category filters and produce binary outputs, lacking the capacity to interpret context or reason about nuanced, adversarially induced forms of harm. In addition, standard evaluation metrics (e.g., attack success rate) fail to capture the semantic severity and dynamic progression of toxicity. To address these limitations, we propose Perception-Retrieval-Judgement (PRJ), a cognitively inspired framework that models toxicity detection as a structured reasoning process. PRJ follows a three-stage design: it first transforms an image into descriptive language (perception), then retrieves external knowledge related to harm categories and traits (retrieval), and finally evaluates toxicity based on legal or normative rules (judgement). This language-centric structure enables the system to detect both explicit and implicit harms with improved interpretability and categorical granularity. In addition, we introduce a dynamic scoring mechanism based on a contextual toxicity risk matrix to quantify harmfulness across different semantic dimensions. Experiments show that PRJ surpasses existing safety checkers in detection accuracy and robustness while uniquely supporting structured category-level toxicity interpretation.

Related papers

• "Just a strange pic": Evaluating 'safety' in GenAI Image safety annotation tasks from diverse annotators' perspectives [28.275024260628484]
  This paper examines how annotators evaluate the safety of AI-generated images.<n>We find that annotators invoke moral, emotional, and contextual reasoning.<n>We argue for evaluation designs that scaffold moral reflection, differentiate types of harm, and make space for subjective, context-sensitive interpretations.
  arXiv  Detail & Related papers  (2025-07-21T19:53:29Z)
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs [83.11815479874447]
  We propose a novel jailbreak attack framework, inspired by cognitive decomposition and biases in human cognition.<n>We employ cognitive decomposition to reduce the complexity of malicious prompts and relevance bias to reorganize prompts.<n>We also introduce a ranking-based harmfulness evaluation metric that surpasses the traditional binary success-or-failure paradigm.
  arXiv  Detail & Related papers  (2025-05-03T05:28:11Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• CogMorph: Cognitive Morphing Attacks for Text-to-Image Models [65.38747950692752]
  This paper reveals a significant and previously unrecognized ethical risk inherent in text-to-image (T2I) generative models.<n>We introduce a novel method, termed the Cognitive Morphing Attack (CogMorph), which manipulates T2I models to generate images that retain the original core subjects but embeds toxic or harmful contextual elements.
  arXiv  Detail & Related papers  (2025-01-21T01:45:56Z)
• Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks [45.07581174558107]
  Retrieval-Augmented Generation (RAG) systems have emerged as a promising solution to mitigate hallucinations.<n>RAG systems are vulnerable to adversarial poisoning attacks, where malicious passages injected into retrieval databases can mislead the model into generating factually incorrect outputs.<n>This paper investigates both the retrieval and the generation components of RAG systems to understand how to enhance their robustness against such attacks.
  arXiv  Detail & Related papers  (2024-12-21T17:31:52Z)
• On the Fairness, Diversity and Reliability of Text-to-Image Generative Models [68.62012304574012]
  multimodal generative models have sparked critical discussions on their reliability, fairness and potential for misuse.<n>We propose an evaluation framework to assess model reliability by analyzing responses to global and local perturbations in the embedding space.<n>Our method lays the groundwork for detecting unreliable, bias-injected models and tracing the provenance of embedded biases.
  arXiv  Detail & Related papers  (2024-11-21T09:46:55Z)
• Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models [58.065255696601604]
  We use compositional property of diffusion models, which allows to leverage multiple prompts in a single image generation. We argue that it is essential to consider all possible approaches to image generation with diffusion models that can be employed by an adversary.
  arXiv  Detail & Related papers  (2024-04-21T16:35:16Z)
• Harm Amplification in Text-to-Image Models [5.397559484007124]
  Text-to-image (T2I) models have emerged as a significant advancement in generative AI. There exist safety concerns regarding their potential to produce harmful image outputs even when users input seemingly safe prompts. This phenomenon, where T2I models generate harmful representations that were not explicit in the input prompt, poses a potentially greater risk than adversarial prompts.
  arXiv  Detail & Related papers  (2024-02-01T23:12:57Z)
• Overcoming Failures of Imagination in AI Infused System Development and Deployment [71.9309995623067]
  NeurIPS 2020 requested that research paper submissions include impact statements on "potential nefarious uses and the consequences of failure" We argue that frameworks of harms must be context-aware and consider a wider range of potential stakeholders, system affordances, as well as viable proxies for assessing harms in the widest sense.
  arXiv  Detail & Related papers  (2020-11-26T18:09:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.