AI Safety in the Eyes of the Downstream Developer: A First Look at Concerns, Practices, and Challenges

• URL: http://arxiv.org/abs/2503.19444v3
• Date: Thu, 17 Jul 2025 01:03:31 GMT
• Title: AI Safety in the Eyes of the Downstream Developer: A First Look at Concerns, Practices, and Challenges
• Authors: Haoyu Gao, Mansooreh Zahedi, Wenxin Jiang, Hong Yi Lin, James Davis, Christoph Treude,
• Abstract summary: Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with minimal training overhead.<n>This study investigates developers' concerns, practices and perceived challenges regarding AI safety issues during AI-based software development.
• Score: 10.342989453672123
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with minimal training overhead. However, their adoption also introduces unique safety challenges, such as data leakage and biased outputs, that demand rigorous handling by downstream developers. While previous research has proposed taxonomies of AI safety concerns and various mitigation strategies, how downstream developers address these issues remains unexplored. This study investigates downstream developers' concerns, practices and perceived challenges regarding AI safety issues during AI-based software development. To achieve this, we conducted a mixed-method study, including interviews with 18 participants, a survey of 86 practitioners, and an analysis of 874 AI incidents from the AI Incident Database. Our results reveal that while developers generally demonstrate strong awareness of AI safety concerns, their practices, especially during the preparation and PTM selection phases, are often inadequate. The lack of concrete guidelines and policies leads to significant variability in the comprehensiveness of their safety approaches throughout the development lifecycle, with additional challenges such as poor documentation and knowledge gaps, further impeding effective implementation. Based on our findings, we offer suggestions for PTM developers, AI-based software developers, researchers, and policy makers to enhance the integration of AI safety measures.

Related papers

• Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects? [0.5352699766206809]
  We conduct an empirical investigation of developer-reported issues and solutions based on discussions from Hugging Face and GitHub.<n>This pipeline produces a dataset of 312,868 security discussions.<n>We conduct a thematic analysis of 753 posts sampled from our dataset and uncover a fine-grained taxonomy of 32 security issues and 24 solutions.
  arXiv  Detail & Related papers  (2025-12-29T11:22:11Z)
• International AI Safety Report 2025: First Key Update: Capabilities and Risk Implications [118.49965571969089]
  This update examines how AI capabilities have improved since the first AI Safety Report.<n>It focuses on key risk areas where substantial new evidence warrants updated assessments.
  arXiv  Detail & Related papers  (2025-10-15T15:13:49Z)
• "We are not Future-ready": Understanding AI Privacy Risks and Existing Mitigation Strategies from the Perspective of AI Developers in Europe [56.1653658714305]
  We interviewed 25 AI developers based in Europe to understand which privacy threats they believe pose the greatest risk to users, developers, and businesses.<n>We find that there is little consensus among AI developers on the relative ranking of privacy risks.<n>While AI developers are aware of proposed mitigation strategies for addressing these risks, they reported minimal real-world adoption.
  arXiv  Detail & Related papers  (2025-10-01T13:51:33Z)
• Responsible AI Technical Report [2.706528024660027]
  KT developed a Responsible AI (RAI) assessment methodology and risk mitigation technologies to ensure the safety and reliability of AI services.<n>We present a reliable assessment methodology that verifies model safety and robustness based on KT's AI risk taxonomy tailored to the domestic environment.<n>We also provide practical tools for managing and mitigating identified AI risks.
  arXiv  Detail & Related papers  (2025-09-24T12:26:33Z)
• Report on NSF Workshop on Science of Safe AI [75.96202715567088]
  New advances in machine learning are leading to new opportunities to develop technology-based solutions to societal problems.<n>To fulfill the promise of AI, we must address how to develop AI-based systems that are accurate and performant but also safe and trustworthy.<n>This report is the result of the discussions in the working groups that addressed different aspects of safety at the workshop.
  arXiv  Detail & Related papers  (2025-06-24T18:55:29Z)
• Comparative Analysis of AI-Driven Security Approaches in DevSecOps: Challenges, Solutions, and Future Directions [0.0]
  This study conducts a systematic literature review to analyze and compare AI-driven security solutions in DevSecOps. The findings reveal gaps in empirical validation, scalability, and integration of AI in security automation. The study proposes future directions for optimizing AI-based security frameworks in DevSecOps.
  arXiv  Detail & Related papers  (2025-04-27T08:18:11Z)
• AISafetyLab: A Comprehensive Framework for AI Safety Evaluation and Improvement [73.0700818105842]
  We introduce AISafetyLab, a unified framework and toolkit that integrates representative attack, defense, and evaluation methodologies for AI safety.<n> AISafetyLab features an intuitive interface that enables developers to seamlessly apply various techniques.<n>We conduct empirical studies on Vicuna, analyzing different attack and defense strategies to provide valuable insights into their comparative effectiveness.
  arXiv  Detail & Related papers  (2025-02-24T02:11:52Z)
• Computational Safety for Generative AI: A Signal Processing Perspective [65.268245109828]
  computational safety is a mathematical framework that enables the quantitative assessment, formulation, and study of safety challenges in GenAI. We show how sensitivity analysis and loss landscape analysis can be used to detect malicious prompts with jailbreak attempts. We discuss key open research challenges, opportunities, and the essential role of signal processing in computational AI safety.
  arXiv  Detail & Related papers  (2025-02-18T02:26:50Z)
• Advancing Trustworthy AI for Sustainable Development: Recommendations for Standardising AI Incident Reporting [2.209921757303168]
  The increasing use of AI technologies has led to increasing AI incidents, posing risks and causing harm to individuals, organizations, and society. This study recognizes and addresses the lack of standardized protocols for reliably and comprehensively gathering such incident data. It proposes nine actionable recommendations to enhance standardization efforts to address these gaps.
  arXiv  Detail & Related papers  (2025-01-01T17:34:57Z)
• Position: Mind the Gap-the Growing Disconnect Between Established Vulnerability Disclosure and AI Security [56.219994752894294]
  We argue that adapting existing processes for AI security reporting is doomed to fail due to fundamental shortcomings for the distinctive characteristics of AI systems.<n>Based on our proposal to address these shortcomings, we discuss an approach to AI security reporting and how the new AI paradigm, AI agents, will further reinforce the need for specialized AI security incident reporting advancements.
  arXiv  Detail & Related papers  (2024-12-19T13:50:26Z)
• "I Don't Use AI for Everything": Exploring Utility, Attitude, and Responsibility of AI-empowered Tools in Software Development [19.851794567529286]
  This study investigates the adoption, impact, and security considerations of AI-empowered tools in the software development process. Our findings reveal widespread adoption of AI tools across various stages of software development.
  arXiv  Detail & Related papers  (2024-09-20T09:17:10Z)
• Safety challenges of AI in medicine in the era of large language models [23.817939398729955]
  Large language models (LLMs) offer new opportunities for medical practitioners, patients, and researchers.<n>As AI and LLMs become more powerful and especially achieve superhuman performance in some medical tasks, public concerns over their safety have intensified.<n>This review examines emerging risks in AI utilization during the LLM era.
  arXiv  Detail & Related papers  (2024-09-11T13:47:47Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? [59.96471873997733]
  We propose an empirical foundation for developing more meaningful safety metrics and define AI safety in a machine learning research context. We aim to provide a more rigorous framework for AI safety research, advancing the science of safety evaluations and clarifying the path towards measurable progress.
  arXiv  Detail & Related papers  (2024-07-31T17:59:24Z)
• AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways [10.16690494897609]
  An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents.
  arXiv  Detail & Related papers  (2024-06-04T01:22:31Z)
• Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits [54.648819983899614]
  General purpose AI seems to have lowered the barriers for the public to use AI and harness its power. We introduce PARTICIP-AI, a framework for laypeople to speculate and assess AI use cases and their impacts.
  arXiv  Detail & Related papers  (2024-03-21T19:12:37Z)
• Testing autonomous vehicles and AI: perspectives and challenges from cybersecurity, transparency, robustness and fairness [53.91018508439669]
  The study explores the complexities of integrating Artificial Intelligence into Autonomous Vehicles (AVs) It examines the challenges introduced by AI components and the impact on testing procedures. The paper identifies significant challenges and suggests future directions for research and development of AI in AV technology.
  arXiv  Detail & Related papers  (2024-02-21T08:29:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.