Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center

• URL: http://arxiv.org/abs/2506.22523v3
• Date: Wed, 02 Jul 2025 21:04:41 GMT
• Title: Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center
• Authors: James Wen, Sahil Nalawade, Zhiwei Liang, Catherine Bielick, Marisa Ferrara Boston, Alexander Chowdhury, Adele Collin, Luigi De Angelis, Jacob Ellen, Heather Frase, Rodrigo R. Gameiro, Juan Manuel Gutierrez, Pooja Kadam, Murat Keceli, Srikanth Krishnamurthy, Anne Kwok, Yanan Lance Lu, Heather Mattie, Liam G. McCoy, Katherine Miller, Allison C. Morgan, Marlene Louisa Moerig, Trang Nguyen, Alexander Owen-Post, Alex D. Ruiz, Sreekar Reddy Puchala, Soujanya Samineni, Takeshi Tohyama, Varun Ullanat, Carmine Valenza, Camilo Velez, Pengcheng Wang, Anna Wuest, Yuxiang Zhou, Yingde Zhu, Jason M. Johnson, Naomi Lenane, Jennifer Willcox, Francis J. Vitiello, Leo Anthony G. Celi, Renato Umeton,
• Abstract summary: Generative artificial intelligence (AI) deployment in academic medical settings raises copyright compliance concerns.<n>Dana-Farber Cancer Institute implemented GPT4DFCI, an internal generative AI tool utilizing OpenAI models.<n>Four teams attempted to extract copyrighted content from GPT4DFCI across four domains.
• Score: 49.85176045690678
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Background: Generative artificial intelligence (AI) deployment in academic medical settings raises copyright compliance concerns. Dana-Farber Cancer Institute implemented GPT4DFCI, an internal generative AI tool utilizing OpenAI models, that is approved for enterprise use in research and operations. Given (1) the exceptionally broad adoption of the tool in our organization, (2) our research mission, and (3) the shared responsibility model required to benefit from Customer Copyright Commitment in Azure OpenAI Service products, we deemed rigorous copyright compliance testing necessary. Case Description: We conducted a structured red teaming exercise in Nov. 2024, with 42 participants from academic, industry, and government institutions. Four teams attempted to extract copyrighted content from GPT4DFCI across four domains: literary works, news articles, scientific publications, and access-restricted clinical notes. Teams successfully extracted verbatim book dedications and near-exact passages through various strategies. News article extraction failed despite jailbreak attempts. Scientific article reproduction yielded only high-level summaries. Clinical note testing revealed appropriate privacy safeguards. Discussion: The successful extraction of literary content indicates potential copyrighted material presence in training data, necessitating inference-time filtering. Differential success rates across content types suggest varying protective mechanisms. The event led to implementation of a copyright-specific meta-prompt in GPT4DFCI; this mitigation has been in production since Jan. 2025. Conclusion: Systematic red teaming revealed specific vulnerabilities in generative AI copyright compliance, leading to concrete mitigation strategies. Academic medical institutions deploying generative AI should implement continuous testing protocols to ensure legal and ethical compliance.

Related papers

• Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review [0.0]
  18 academic manuscripts on the preprint website arXiv were found to contain hidden instructions designed to manipulate AI-assisted peer review.<n>Author responses varied: one planned to withdraw the affected paper, while another defended the practice as legitimate testing of reviewer compliance.<n>We examine the technique of prompt injection in large language models (LLMs), revealing four types of hidden prompts.
  arXiv  Detail & Related papers  (2025-07-08T17:11:13Z)
• BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text [6.682911432177815]
  In scientific research, limitations refer to the shortcomings, constraints, or weaknesses within a study.<n>Authors often a) underreport them in the paper text and b) use hedging strategies to satisfy editorial requirements.<n>This underreporting behavior, along with an explosion in the number of publications, has created a pressing need to automatically extract or generate such limitations.
  arXiv  Detail & Related papers  (2025-05-22T06:04:02Z)
• Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification [4.099848175176399]
  The application of AI tools to the legal field feels natural.<n>However, legal documents differ from the web-based text that underlies most AI systems.<n>We identify three areas of special relevance to practitioners: data curation, data annotation, and output verification.
  arXiv  Detail & Related papers  (2025-04-02T04:34:58Z)
• Evidencing Unauthorized Training Data from AI Generated Content using Information Isotopes [0.0]
  In a rush to stay competitive, some institutions may inadvertently or even deliberately include unauthorized data for AI training.<n>We introduce the concept of information isotopes and elucidate their properties in tracing training data within opaque AI systems.<n>We propose an information isotope tracing method designed to identify and provide evidence of unauthorized data usage.
  arXiv  Detail & Related papers  (2025-03-24T07:35:59Z)
• Generative AI Training and Copyright Law [0.1074267520911262]
  Training generative AI models requires extensive amounts of data.<n>A common practice is to collect such data through web scraping. Yet, much of what has been and is collected is copyright protected.<n>In the USA, AI developers rely on "fair use" and in Europe, the prevailing view is that the exception for "Text and Data Mining" (TDM) applies.
  arXiv  Detail & Related papers  (2025-02-21T08:45:14Z)
• Suspected Undeclared Use of Artificial Intelligence in the Academic Literature: An Analysis of the Academ-AI Dataset [0.0]
  Academ-AI documents examples of suspected undeclared AI usage in the academic literature. Undeclared AI seems to appear in journals with higher citation metrics and higher article processing charges.
  arXiv  Detail & Related papers  (2024-11-20T21:29:36Z)
• Evaluating Copyright Takedown Methods for Language Models [100.38129820325497]
  Language models (LMs) derive their capabilities from extensive training on diverse data, including potentially copyrighted material. This paper introduces the first evaluation of the feasibility and side effects of copyright takedowns for LMs. We examine several strategies, including adding system prompts, decoding-time filtering interventions, and unlearning approaches.
  arXiv  Detail & Related papers  (2024-06-26T18:09:46Z)
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools [32.78336381381673]
  We report on the first preregistered empirical evaluation of AI-driven legal research tools. We find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time. It provides evidence to inform the responsibilities of legal professionals in supervising and verifying AI outputs.
  arXiv  Detail & Related papers  (2024-05-30T17:56:05Z)
• Report of the 1st Workshop on Generative AI and Law [78.62063815165968]
  This report presents the takeaways of the inaugural Workshop on Generative AI and Law (GenLaw) A cross-disciplinary group of practitioners and scholars from computer science and law convened to discuss the technical, doctrinal, and policy challenges presented by law for Generative AI.
  arXiv  Detail & Related papers  (2023-11-11T04:13:37Z)
• Using Generative AI for Literature Searches and Scholarly Writing: Is the Integrity of the Scientific Discourse in Jeopardy? [0.0]
  We document the ongoing discussion in the science community with a review of news articles, editorials, and position statements by major scientific publishers. We provide a brief overview of potentially useful capabilities and sensible applications of ChatGPT and similar AI tools for purposes of scholarly writing.
  arXiv  Detail & Related papers  (2023-10-07T23:31:24Z)
• Identifying and Mitigating the Security Risks of Generative AI [179.2384121957896]
  This paper reports the findings of a workshop held at Google on the dual-use dilemma posed by GenAI. GenAI can be used just as well by attackers to generate new attacks and increase the velocity and efficacy of existing attacks. We discuss short-term and long-term goals for the community on this topic.
  arXiv  Detail & Related papers  (2023-08-28T18:51:09Z)
• FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare [73.78776682247187]
  Concerns have been raised about the technical, clinical, ethical and legal risks associated with medical AI. This work describes the FUTURE-AI guideline as the first international consensus framework for guiding the development and deployment of trustworthy AI tools in healthcare.
  arXiv  Detail & Related papers  (2023-08-11T10:49:05Z)
• A LLM Assisted Exploitation of AI-Guardian [57.572998144258705]
  We evaluate the robustness of AI-Guardian, a recent defense to adversarial examples published at IEEE S&P 2023. We write none of the code to attack this model, and instead prompt GPT-4 to implement all attack algorithms following our instructions and guidance. This process was surprisingly effective and efficient, with the language model at times producing code from ambiguous instructions faster than the author of this paper could have done.
  arXiv  Detail & Related papers  (2023-07-20T17:33:25Z)
• The State of AI Ethics Report (January 2021) [34.40432073381641]
  The State of AI Ethics aims to help anyone, from machine learning experts to human rights activists and policymakers, quickly digest and understand the field's ever-changing developments. The report distills the research and reporting surrounding various domains related to the ethics of AI, including: algorithmic injustice, discrimination, ethical AI, labor impacts, misinformation, privacy, risk and security, social media, and more. Unique to this report is "The Abuse and Misogynoir Playbook," written by Dr. Katlyn Tuner (Research Scientist, Space Enabled Research Group, MIT), Dr. Danielle Wood (Assistant Professor, Program
  arXiv  Detail & Related papers  (2021-05-19T10:59:17Z)
• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence [81.04070052740596]
  Legal Artificial Intelligence (LegalAI) focuses on applying the technology of artificial intelligence, especially natural language processing, to benefit tasks in the legal domain. This paper introduces the history, the current state, and the future directions of research in LegalAI.
  arXiv  Detail & Related papers  (2020-04-25T14:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.