The Perils of Agency: How Developers Perceive, Prioritize, and Address Risks in Agentic AI Products

• URL: http://arxiv.org/abs/2606.15485v1
• Date: Sat, 13 Jun 2026 21:49:32 GMT
• Title: The Perils of Agency: How Developers Perceive, Prioritize, and Address Risks in Agentic AI Products
• Abstract summary: Agentic AI systems act autonomously, use tools, adapt to context, and operate in complex real-world environments.<n>We studied how industry developers perceive, prioritize, and address the risks in their agentic AI products.
• Score: 19.630749334675894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic AI systems act autonomously, use tools, adapt to context, and operate in complex real-world environments. However, these same characteristics can create or exacerbate product risks. We studied how industry developers (n=35) perceive, prioritize, and address the risks in their agentic AI products. We found that developers' perceptions of risk were closely tied to the qualities that made the product agentic, such as autonomy, tool use, and usage in a real-world context. Developers prioritized product and business risks before considering downstream societal risks like job displacement and end-user privacy. This prioritization also impacted developers' ability and motivation to mitigate agentic risks. Finally, developers lacked mature controls for containing agentic risks, often relying on constraining the same characteristics that make agents useful: e.g., autonomy and goal complexity. These findings reveal a capability vs. risk control tension in agentic AI development: developers need to address risks that emerge from agentic capabilities, yet they currently have limited support for doing so without constraining agentic functionality.

Related papers

• On Safety Risks in Experience-Driven Self-Evolving Agents [78.82066774532676]
  We investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments.<n>Our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation.
  arXiv  Detail & Related papers  (2026-04-18T11:32:36Z)
• Self-Sovereign Agent [75.57858796824469]
  Self-sovereign AI systems can economically sustain and extend their own operation without human involvement.<n>Recent advances in large language models and agent frameworks have substantially expanded agents' practical capabilities.
  arXiv  Detail & Related papers  (2026-03-04T03:11:26Z)
• AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security [126.49733412191416]
  Current guardrail models lack agentic risk awareness and transparency in risk diagnosis.<n>We propose a unified three-dimensional taxonomy that categorizes agentic risks by their source (where), failure mode (how), and consequence (what)<n>We introduce a new fine-grained agentic safety benchmark (ATBench) and a Diagnostic Guardrail framework for agent safety and security (AgentDoG)
  arXiv  Detail & Related papers  (2026-01-26T13:45:41Z)
• Emergent Risk Awareness in Rational Agents under Resource Constraints [2.69407449467596]
  This work aims to increase understanding and interpretability of emergent behaviours of AI agents operating under survival pressure.<n>We provide theoretical and empirical results that quantify the impact of survival-driven preference shifts.<n>We propose mechanisms to mitigate the emergence of risk-seeking or risk-averse behaviours.
  arXiv  Detail & Related papers  (2025-05-29T13:31:12Z)
• Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents [0.0]
  This paper introduces a comprehensive threat model tailored specifically for GenAI agents.<n>Research work identifies 9 primary threats and organizes them across five key domains.
  arXiv  Detail & Related papers  (2025-04-28T16:29:24Z)
• Agentic Business Process Management: Practitioner Perspectives on Agent Governance in Business Processes [0.7270112855088837]
  With the rise of generative AI, industry interest in software agents is growing.<n>This paper investigates how organizations can effectively govern AI agents.<n>It outlines six key recommendations for the responsible adoption of AI agents.
  arXiv  Detail & Related papers  (2025-03-23T20:15:24Z)
• Fully Autonomous AI Agents Should Not be Developed [50.61667544399082]
  This paper argues that fully autonomous AI agents should not be developed.<n>In support of this position, we build from prior scientific literature and current product marketing to delineate different AI agent levels.<n>Our analysis reveals that risks to people increase with the autonomy of a system.
  arXiv  Detail & Related papers  (2025-02-04T19:00:06Z)
• Risk Alignment in Agentic AI Systems [0.0]
  Agentic AIs capable of undertaking complex actions with little supervision raise new questions about how to safely create and align such systems with users, developers, and society. Risk alignment will matter for user satisfaction and trust, but it will also have important ramifications for society more broadly. We present three papers that bear on key normative and technical aspects of these questions.
  arXiv  Detail & Related papers  (2024-10-02T18:21:08Z)
• Risks and Opportunities of Open-Source Generative AI [64.86989162783648]
  Applications of Generative AI (Gen AI) are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about the potential risks of the technology, and resulted in calls for tighter regulation. This regulation is likely to put at risk the budding field of open-source generative AI.
  arXiv  Detail & Related papers  (2024-05-14T13:37:36Z)
• Model evaluation for extreme risks [46.53170857607407]
  Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks.
  arXiv  Detail & Related papers  (2023-05-24T16:38:43Z)
• Safe Reinforcement Learning via Curriculum Induction [94.67835258431202]
  In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. Existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor.
  arXiv  Detail & Related papers  (2020-06-22T10:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.