Expanding External Access To Frontier AI Models For Dangerous Capability Evaluations

• URL: http://arxiv.org/abs/2601.11916v1
• Date: Sat, 17 Jan 2026 05:33:04 GMT
• Title: Expanding External Access To Frontier AI Models For Dangerous Capability Evaluations
• Abstract summary: The EU General-Purpose AI Code of Practice calls for "appropriate access", but does not specify what this means in practice.<n>There is no common framework for describing different types and levels of evaluator access.<n>We propose a taxonomy of access methods for dangerous capability evaluations.
• Score: 41.72971769197687
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier AI companies increasingly rely on external evaluations to assess risks from dangerous capabilities before deployment. However, external evaluators often receive limited model access, limited information, and little time, which can reduce evaluation rigour and confidence. The EU General-Purpose AI Code of Practice calls for "appropriate access", but does not specify what this means in practice. Furthermore, there is no common framework for describing different types and levels of evaluator access. To address this gap, we propose a taxonomy of access methods for dangerous capability evaluations. We disentangle three aspects of access: model access, model information, and evaluation timeframe. For each aspect, we review benefits and risks, including how expanding access can reduce false negatives and improve stakeholder trust, but can also increase security and capacity challenges. We argue that these limitations can likely be mitigated through technical means and safeguards used in other industries. Based on the taxonomy, we propose three descriptive access levels: AL1 (black-box model access and minimal information), AL2 (grey-box model access and substantial information), and AL3 (white-box model access and comprehensive information), to support clearer communication between evaluators, frontier AI companies, and policymakers. We believe these levels correspond to the different standards for appropriate access defined in the EU Code of Practice, though these standards may change over time.

Related papers

• Harmonizing AI Safety Thresholds [0.0]
  We develop a methodology for deriving harmonized thresholds across three risk domains.<n>For misuse risks, we take expected harm as the key primitive and use an explicit risk-modeling approach.<n>For automated AI R&D, we base our proposed threshold on the observed rate of AI progress rather than expected harm.
  arXiv  Detail & Related papers  (2026-07-17T16:45:30Z)
• Beyond the Binary: A nuanced path for open-weight advanced AI [0.0]
  Open-weight advanced AI models are systems whose parameters are freely available for download and adaptation.<n>This report moves beyond the binary framing of open'' versus closed'' AI.<n>We find significant disparities in safety practices across developers and jurisdictions.<n>We propose a tiered, safety-anchored approach to model release, where openness is determined by rigorous risk assessment and demonstrated safety.
  arXiv  Detail & Related papers  (2026-02-23T10:26:23Z)
• Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies [57.521647436515785]
  We define frontier AI auditing as rigorous third-party verification of frontier AI developers' safety and security claims.<n>We introduce AI Assurance Levels (AAL-1 to AAL-4), ranging from time-bounded system audits to continuous, deception-resilient verification.
  arXiv  Detail & Related papers  (2026-01-16T18:44:09Z)
• A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives [65.3369988566853]
  Recent studies have demonstrated that adversaries can replicate a target model's functionality.<n>Model Extraction Attacks pose threats to intellectual property, privacy, and system security.<n>We propose a novel taxonomy that classifies MEAs according to attack mechanisms, defense approaches, and computing environments.
  arXiv  Detail & Related papers  (2025-08-20T19:49:59Z)
• AI Companies Should Report Pre- and Post-Mitigation Safety Evaluations [5.984437476321095]
  frontier AI companies should report both pre- and post-mitigation safety evaluations.<n> evaluating models at both stages provides policymakers with essential evidence to regulate deployment, access, and safety standards.
  arXiv  Detail & Related papers  (2025-03-17T17:56:43Z)
• Securing External Deeper-than-black-box GPAI Evaluations [49.1574468325115]
  This paper examines the critical challenges and potential solutions for conducting secure and effective external evaluations of general-purpose AI (GPAI) models.<n>With the exponential growth in size, capability, reach and accompanying risk, ensuring accountability, safety, and public trust requires frameworks that go beyond traditional black-box methods.
  arXiv  Detail & Related papers  (2025-03-10T16:13:45Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• DocMIA: Document-Level Membership Inference Attacks against DocVQA Models [52.13818827581981]
  We introduce two novel membership inference attacks tailored specifically to DocVQA models.<n>Our methods outperform existing state-of-the-art membership inference attacks across a variety of DocVQA models and datasets.
  arXiv  Detail & Related papers  (2025-02-06T00:58:21Z)
• AI Safety Frameworks Should Include Procedures for Model Access Decisions [0.0]
  The downstream use cases, benefits, and risks of AI models depend significantly on what sort of access is provided to the model.<n>Existing safety frameworks and AI developer usage policies recognise that the risk posed by a given model depends on the level of access provided to a given audience.<n>This paper proposes that frontier AI companies build on existing safety frameworks by outlining transparent procedures for making decisions about model access.
  arXiv  Detail & Related papers  (2024-11-15T19:44:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.