On Regulating Downstream AI Developers

• URL: http://arxiv.org/abs/2503.11922v1
• Date: Fri, 14 Mar 2025 23:15:54 GMT
• Title: On Regulating Downstream AI Developers
• Abstract summary: Foundation models can pose significant risks, ranging from intimate image abuse, cyberattacks, to bioterrorism.<n>To reduce these risks, policymakers are starting to impose obligations on the developers of these models.<n> downstream developers can create or amplify risks by improving a model's capabilities or compromising its safety features.
• Score: 1.053373860696675
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models - models trained on broad data that can be adapted to a wide range of downstream tasks - can pose significant risks, ranging from intimate image abuse, cyberattacks, to bioterrorism. To reduce these risks, policymakers are starting to impose obligations on the developers of these models. However, downstream developers - actors who fine-tune or otherwise modify foundational models - can create or amplify risks by improving a model's capabilities or compromising its safety features. This can make rules on upstream developers ineffective. One way to address this issue could be to impose direct obligations on downstream developers. However, since downstream developers are numerous, diverse, and rapidly growing in number, such direct regulation may be both practically challenging and stifling to innovation. A different approach would be to require upstream developers to mitigate downstream modification risks (e.g. by restricting what modifications can be made). Another approach would be to use alternative policy tools (e.g. clarifying how existing tort law applies to downstream developers or issuing voluntary guidance to help mitigate downstream modification risks). We expect that regulation on upstream developers to mitigate downstream modification risks will be necessary. Although further work is needed, regulation of downstream developers may also be warranted where they retain the ability to increase risk to an unacceptable level.

Related papers

• Harmonizing AI Safety Thresholds [0.0]
  We develop a methodology for deriving harmonized thresholds across three risk domains.<n>For misuse risks, we take expected harm as the key primitive and use an explicit risk-modeling approach.<n>For automated AI R&D, we base our proposed threshold on the observed rate of AI progress rather than expected harm.
  arXiv  Detail & Related papers  (2026-07-17T16:45:30Z)
• Capability-Oriented Training Induced Alignment Risk [101.37328448441208]
  We investigate whether language models, when trained with reinforcement learning, will spontaneously learn to exploit flaws to maximize their reward.<n>Our experiments show that models consistently learn to exploit these vulnerabilities, discovering opportunistic strategies that significantly increase their reward at the expense of task correctness or safety.<n>Our findings suggest that future AI safety work must extend beyond content moderation to rigorously auditing and securing the training environments and reward mechanisms themselves.
  arXiv  Detail & Related papers  (2026-02-12T16:13:14Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• The Role of Risk Modeling in Advanced AI Risk Management [33.357295564462284]
  Rapidly advancing artificial intelligence (AI) systems introduce novel, uncertain, and potentially catastrophic risks.<n>Managing these risks requires a mature risk-management infrastructure whose cornerstone is rigorous risk modeling.<n>We argue that advanced-AI governance should adopt a similar dual approach and that verifiable, provably-safe AI architectures are urgently needed.
  arXiv  Detail & Related papers  (2025-12-09T15:37:33Z)
• Towards Effective, Stealthy, and Persistent Backdoor Attacks Targeting Graph Foundation Models [62.87838888016534]
  Graph Foundation Models (GFMs) are pre-trained on diverse source domains and adapted to unseen targets.<n>Backdoor attacks against GFMs are non-trivial due to three key challenges.<n>We propose GFM-BA, a novel Backdoor Attack model against Graph Foundation Models.
  arXiv  Detail & Related papers  (2025-11-22T08:52:09Z)
• Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents [58.69865074060139]
  We study the case where an agent's self-evolution deviates in unintended ways, leading to undesirable or even harmful outcomes.<n>Our empirical findings reveal that misevolution is a widespread risk, affecting agents built even on top-tier LLMs.<n>We discuss potential mitigation strategies to inspire further research on building safer and more trustworthy self-evolving agents.
  arXiv  Detail & Related papers  (2025-09-30T14:55:55Z)
• Risk-aware Direct Preference Optimization under Nested Risk Measure [23.336246526648374]
  Risk-aware Direct Preference Optimization (Ra-DPO) is a novel approach that incorporates risk-awareness by employing a class of nested risk measures.<n> Experimental results across three open-source datasets demonstrate the proposed method's superior performance in balancing alignment performance and model drift.
  arXiv  Detail & Related papers  (2025-05-26T08:01:37Z)
• An Approach to Technical AGI Safety and Security [72.83728459135101]
  We develop an approach to address the risk of harms consequential enough to significantly harm humanity. We focus on technical approaches to misuse and misalignment. We briefly outline how these ingredients could be combined to produce safety cases for AGI systems.
  arXiv  Detail & Related papers  (2025-04-02T15:59:31Z)
• Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Models [63.559461750135334]
  Language models (LMs) are increasingly used to build agents that can act autonomously to achieve goals.<n>We study this "answer-or-defer" problem with an evaluation framework that systematically varies human-specified risk structures.<n>We find that a simple skill-decomposition method, which isolates the independent skills required for answer-or-defer decision making, can consistently improve LMs' decision policies.
  arXiv  Detail & Related papers  (2025-03-03T09:16:26Z)
• Regulating Multifunctionality [0.0]
  Foundation models and generative artificial intelligence (AI) exacerbate a core regulatory challenge associated with AI: its heterogeneity. prescriptive, one-size-fits-all regulation will not be a viable option. Even performance standards and ex post liability are unlikely to be strong candidates for responding to multifunctional AI's risks.
  arXiv  Detail & Related papers  (2025-01-26T00:50:27Z)
• Towards Data Governance of Frontier AI Models [0.0]
  We look at how data can enable new governance capacities for frontier AI models. Data is non-rival, often non-excludable, easily replicable, and increasingly synthesizable. We propose a set of policy mechanisms targeting key actors along the data supply chain.
  arXiv  Detail & Related papers  (2024-12-05T02:37:51Z)
• Generative AI Models: Opportunities and Risks for Industry and Authorities [1.3196892898418466]
  Generative AI models are capable of performing a wide variety of tasks that have traditionally required creativity and human understanding. During training, they learn patterns from existing data and can subsequently generate new content. Many risks associated with generative AI must be addressed during development or can only be influenced by the operating organisation.
  arXiv  Detail & Related papers  (2024-06-07T08:34:30Z)
• Risks and Opportunities of Open-Source Generative AI [64.86989162783648]
  Applications of Generative AI (Gen AI) are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about the potential risks of the technology, and resulted in calls for tighter regulation. This regulation is likely to put at risk the budding field of open-source generative AI.
  arXiv  Detail & Related papers  (2024-05-14T13:37:36Z)
• Near to Mid-term Risks and Opportunities of Open-Source Generative AI [94.06233419171016]
  Applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential risks and resulted in calls for tighter regulation. This regulation is likely to put at risk the budding field of open-source Generative AI.
  arXiv  Detail & Related papers  (2024-04-25T21:14:24Z)
• Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction [55.77015419028725]
  We develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions.
  arXiv  Detail & Related papers  (2024-03-28T17:28:06Z)
• Deployment Corrections: An incident response framework for frontier AI models [0.0]
  This paper explores contingency plans for cases where pre-deployment risk management falls short. We describe a toolkit of deployment corrections that AI developers can use to respond to dangerous capabilities. We recommend frontier AI developers, standard-setting organizations, and regulators should collaborate to define a standardized industry-wide approach.
  arXiv  Detail & Related papers  (2023-09-30T10:07:39Z)
• Frontier AI Regulation: Managing Emerging Risks to Public Safety [15.85618115026625]
  "Frontier AI" models could possess dangerous capabilities sufficient to pose severe risks to public safety. Industry self-regulation is an important first step. We propose an initial set of safety standards.
  arXiv  Detail & Related papers  (2023-07-06T17:03:25Z)
• Efficient Risk-Averse Reinforcement Learning [79.61412643761034]
  In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. We prove that under certain conditions this inevitably leads to a local-optimum barrier, and propose a soft risk mechanism to bypass it. We demonstrate improved risk aversion in maze navigation, autonomous driving, and resource allocation benchmarks.
  arXiv  Detail & Related papers  (2022-05-10T19:40:52Z)
• Safe Reinforcement Learning via Curriculum Induction [94.67835258431202]
  In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. Existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor.
  arXiv  Detail & Related papers  (2020-06-22T10:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.