Developing a Risk Identification Framework for Foundation Model Uses

• URL: http://arxiv.org/abs/2506.02066v1
• Date: Sun, 01 Jun 2025 23:37:41 GMT
• Title: Developing a Risk Identification Framework for Foundation Model Uses
• Authors: David Piorkowski, Michael Hind, John Richards, Jacquelyn Martino,
• Abstract summary: There is little guidance for practitioners on how to determine which risks are relevant for a given foundation model use.<n>We identify challenges for building a foundation model risk identification framework and adapt ideas from usage governance to synthesize four design requirements.
• Score: 7.013133148085937
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As foundation models grow in both popularity and capability, researchers have uncovered a variety of ways that the models can pose a risk to the model's owner, user, or others. Despite the efforts of measuring these risks via benchmarks and cataloging them in AI risk taxonomies, there is little guidance for practitioners on how to determine which risks are relevant for a given foundation model use. In this paper, we address this gap and develop requirements and an initial design for a risk identification framework. To do so, we look to prior literature to identify challenges for building a foundation model risk identification framework and adapt ideas from usage governance to synthesize four design requirements. We then demonstrate how a candidate framework can addresses these design requirements and provide a foundation model use example to show how the framework works in practice for a small subset of risks.

Related papers

• Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents [67.07177243654485]
  This survey collects and analyzes the different threats faced by large language models-based agents. We identify six key features of LLM-based agents, based on which we summarize the current research progress. We select four representative agents as case studies to analyze the risks they may face in practical use.
  arXiv  Detail & Related papers  (2024-11-14T15:40:04Z)
• A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation [0.3413711585591077]
  As generative AI systems, including large language models (LLMs) and diffusion models, advance rapidly, their growing adoption has led to new and complex security risks. This paper introduces a novel formal framework for categorizing and mitigating these emergent security risks. We identify previously under-explored risks, including latent space exploitation, multi-modal cross-attack vectors, and feedback-loop-induced model degradation.
  arXiv  Detail & Related papers  (2024-10-15T02:51:32Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents [22.94671478021277]
  Foundation model-enabled generative artificial intelligence facilitates the development and implementation of agents. There is a lack of systematic knowledge to guide practitioners in designing the agents. We present a pattern catalogue consisting of 18 architectural patterns with analyses of the context, forces, and trade-offs.
  arXiv  Detail & Related papers  (2024-05-16T23:24:48Z)
• Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning [61.2224355547598]
  Open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. Our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions.
  arXiv  Detail & Related papers  (2024-04-16T13:22:54Z)
• On the Societal Impact of Open Foundation Models [93.67389739906561]
  We focus on open foundation models, defined here as those with broadly available model weights. We identify five distinctive properties of open foundation models that lead to both their benefits and risks.
  arXiv  Detail & Related papers  (2024-02-27T16:49:53Z)
• Typology of Risks of Generative Text-to-Image Models [1.933681537640272]
  This paper investigates the direct risks and harms associated with modern text-to-image generative models, such as DALL-E and Midjourney. Our review reveals significant knowledge gaps concerning the understanding and treatment of these risks despite some already being addressed. We identify 22 distinct risk types, spanning issues from data bias to malicious use.
  arXiv  Detail & Related papers  (2023-07-08T20:33:30Z)
• Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning [20.449497882324785]
  We leverage distributional reinforcement learning to introduce two new notions of model equivalence. We demonstrate how our framework can be used to augment any model-free risk-sensitive algorithm.
  arXiv  Detail & Related papers  (2023-07-04T13:23:21Z)
• Measuring Re-identification Risk [72.6715574626418]
  We present a new theoretical framework to measure re-identification risk in compact user representations. Our framework formally bounds the probability that an attacker may be able to obtain the identity of a user from their representation. We show how our framework is general enough to model important real-world applications such as the Chrome's Topics API for interest-based advertising.
  arXiv  Detail & Related papers  (2023-04-12T16:27:36Z)
• On the Opportunities and Risks of Foundation Models [256.61956234436553]
  We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration.
  arXiv  Detail & Related papers  (2021-08-16T17:50:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.