Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development

• URL: http://arxiv.org/abs/2508.10108v1
• Date: Wed, 13 Aug 2025 18:04:01 GMT
• Title: Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development
• Abstract summary: The Amazon Nova AI Challenge is a global competition among 10 university teams to drive advances in secure AI.<n>This paper outlines the advancements made by university teams and the Amazon Nova AI Challenge team in addressing the safety challenges of AI for software development.
• Score: 17.799708565824428
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: AI systems for software development are rapidly gaining prominence, yet significant challenges remain in ensuring their safety. To address this, Amazon launched the Trusted AI track of the Amazon Nova AI Challenge, a global competition among 10 university teams to drive advances in secure AI. In the challenge, five teams focus on developing automated red teaming bots, while the other five create safe AI assistants. This challenge provides teams with a unique platform to evaluate automated red-teaming and safety alignment methods through head-to-head adversarial tournaments where red teams have multi-turn conversations with the competing AI coding assistants to test their safety alignment. Along with this, the challenge provides teams with a feed of high quality annotated data to fuel iterative improvement. Throughout the challenge, teams developed state-of-the-art techniques, introducing novel approaches in reasoning-based safety alignment, robust model guardrails, multi-turn jail-breaking, and efficient probing of large language models (LLMs). To support these efforts, the Amazon Nova AI Challenge team made substantial scientific and engineering investments, including building a custom baseline coding specialist model for the challenge from scratch, developing a tournament orchestration service, and creating an evaluation harness. This paper outlines the advancements made by university teams and the Amazon Nova AI Challenge team in addressing the safety challenges of AI for software development, highlighting this collaborative effort to raise the bar for AI safety.

Related papers

• Understanding Human-AI Collaboration in Cybersecurity Competitions [12.034897605949858]
  We study how participants' perception, trust, and expectations shift before versus after hands-on AI use.<n>We find that, as the competition progresses, teams increasingly delegate larger subtasks to the AI.<n>Remarkably, autonomous agents that self-direct their prompting and tool use bypass this bottleneck and outperform most human teams.
  arXiv  Detail & Related papers  (2026-02-24T01:15:18Z)
• Advancing AI Challenges for the United States Department of the Air Force [91.02589169578908]
  The DAF-MIT AI Accelerator is a collaboration between the United States Department of the Air Force (DAF) and the Massachusetts Institute of Technology (MIT)<n>This article supplements our previous publication, which introduced AI Accelerator challenges.<n>We provide an update on how ongoing and new challenges have successfully contributed to AI research and applications of AI technologies.
  arXiv  Detail & Related papers  (2025-10-31T21:34:57Z)
• International AI Safety Report 2025: First Key Update: Capabilities and Risk Implications [118.49965571969089]
  This update examines how AI capabilities have improved since the first AI Safety Report.<n>It focuses on key risk areas where substantial new evidence warrants updated assessments.
  arXiv  Detail & Related papers  (2025-10-15T15:13:49Z)
• The Singapore Consensus on Global AI Safety Research Priorities [128.58674892183657]
  "2025 Singapore Conference on AI (SCAI): International Scientific Exchange on AI Safety" aimed to support research in this space.<n>Report builds on the International AI Safety Report chaired by Yoshua Bengio and backed by 33 governments.<n>Report organises AI safety research domains into three types: challenges with creating trustworthy AI systems (Development), challenges with evaluating their risks (Assessment) and challenges with monitoring and intervening after deployment (Control)
  arXiv  Detail & Related papers  (2025-06-25T17:59:50Z)
• Lessons From Red Teaming 100 Generative AI Products [1.5285633805077958]
  In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems.<n>We offer practical recommendations aimed at aligning red teaming efforts with real world risks.
  arXiv  Detail & Related papers  (2025-01-13T11:36:33Z)
• Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI [52.138044013005]
  generative AI, particularly large language models (LLMs), become increasingly integrated into production applications. New attack surfaces and vulnerabilities emerge and put a focus on adversarial threats in natural language and multi-modal systems. Red-teaming has gained importance in proactively identifying weaknesses in these systems, while blue-teaming works to protect against such adversarial attacks. This work aims to bridge the gap between academic insights and practical security measures for the protection of generative AI systems.
  arXiv  Detail & Related papers  (2024-09-23T10:18:10Z)
• A Red Teaming Framework for Securing AI in Maritime Autonomous Systems [0.0]
  We propose one of the first red team frameworks for evaluating the AI security of maritime autonomous systems. This framework is a multi-part checklist, which can be tailored to different systems and requirements. We demonstrate this framework to be highly effective for a red team to use to uncover numerous vulnerabilities within a real-world maritime autonomous systems AI.
  arXiv  Detail & Related papers  (2023-12-08T14:59:07Z)
• DanZero+: Dominating the GuanDan Game through Reinforcement Learning [95.90682269990705]
  We develop an AI program for an exceptionally complex and popular card game called GuanDan. We first put forward an AI program named DanZero for this game. In order to further enhance the AI's capabilities, we apply policy-based reinforcement learning algorithm to GuanDan.
  arXiv  Detail & Related papers  (2023-12-05T08:07:32Z)
• The Promise and Peril of Artificial Intelligence -- Violet Teaming Offers a Balanced Path Forward [56.16884466478886]
  This paper reviews emerging issues with opaque and uncontrollable AI systems. It proposes an integrative framework called violet teaming to develop reliable and responsible AI. It emerged from AI safety research to manage risks proactively by design.
  arXiv  Detail & Related papers  (2023-08-28T02:10:38Z)
• Positive AI: Key Challenges in Designing Artificial Intelligence for Wellbeing [0.5461938536945723]
  Many people are increasingly worried about AI's impact on their lives. To ensure AI progresses beneficially, some researchers have proposed "wellbeing" as a key objective to govern AI. This article addresses key challenges in designing AI for wellbeing.
  arXiv  Detail & Related papers  (2023-04-12T12:43:00Z)
• Artificial Intelligence Security Competition (AISC) [52.20676747225118]
  The Artificial Intelligence Security Competition (AISC) was organized by the Zhongguancun Laboratory, China Industrial Control Systems Cyber Emergency Response Team, Institute for Artificial Intelligence, Tsinghua University, and RealAI. The competition consists of three tracks, including Deepfake Security Competition, Autonomous Driving Security Competition, and Face Recognition Security Competition. This report will introduce the competition rules of these three tracks and the solutions of top-ranking teams in each track.
  arXiv  Detail & Related papers  (2022-12-07T02:45:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.