Anthropic

1.3.2.1 Methodology precisely defined (70%) 25%

Details of the main components of the threat model (actors, pathways, use of MITRE ATT&CK framework) are given. However, important details are lacking, such as how “the compelling case that we have mapped out the most likely and consequential threat model” will be made in practice, how the bottlenecks mentioned will be identified, and so on.

Quotes:

“Threat model mapping: For each capability threshold, make a compelling case that we have mapped out the most likely and consequential threat models: combinations of actors (if relevant), attack pathways, model capability bottlenecks, and types of harms.” (p. 6)

“Follow risk governance best practices, such as use of the MITRE ATT&CK Framework to establish the relationship between the identified threats, sensitive assets, attack vectors and, in doing so, sufficiently capture the resulting risks that must be addressed…” (p. 9)

1.3.2.2 Mechanism to incorporate red teaming findings (15%) 0%

No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.

Quotes:

No relevant quotes found.

1.3.2.3 Prioritization of severe and probable risks (15%) 25%

Explicit mention that the company will prioritize monitoring capabilities with “the most likely and consequential threat models”: “For each capability threshold, make a compelling case we have mapped out the most likely and consequential threat models”. This implies that, among the full space of risk models, they then decide where to focus based on what risk models score highest on probability x severity.

However, importantly, they don’t provide an explanation into how likelihood and severity of risk models are determined, nor are these scores published.

They do indicate that external input helps prioritize these capabilities (“commissioned research reports, discussions with domain experts, input from expert forecasters, public research, conversations with other industry actors through the Frontier Model Forum, and internal discussions”). More detail on how this input influenced prioritization, as well as severity and probability scoring, would be an improvement.

Quotes:

“Threat model mapping: For each capability threshold, make a compelling case that we have mapped out the most likely and consequential threat models: combinations of actors (if relevant), attack pathways, model capability bottlenecks, and types of harms. We also make a compelling case that there does not exist a threat model that we are not evaluating that represents a substantial amount of risk” (p. 6)

“Overall, our decision to prioritize the capabilities in the two tables above is based on commissioned research reports, discussions with domain experts, input from expert forecasters, public research, conversations with other industry actors through the Frontier Model Forum, and internal discussions” (p. 5)

2.1.1.1 Risk tolerance is at least qualitatively defined for all risks (33%) 25%

They mention that the framework aims to “keep risks below acceptable levels”, but no qualitative definition is given of these acceptable levels.

Implicitly, the capability thresholds define a proto-risk tolerance. For instance, “CBRN-3: The ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons.” To improve, they must set out the maximum amount of risk the company is willing to accept, for each risk domain (though they need not differ between risk domains), ideally expressed in terms of probabilities and severity (economic damages, physical lives, etc), and separate from KRIs.

Quotes:

“In September 2023, we released our Responsible Scaling Policy (RSP), a public commitment not to train or deploy models capable of causing catastrophic harm unless we have implemented safety and security measures that will keep risks below acceptable levels.” (Executive Summary)

“The Required Safeguards for each Capability Threshold are intended to mitigate risk to acceptable levels.” (Executive Summary)

“CBRN-3: The ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons.” (p. 6)

2.1.1.2 Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks (33%) 0%

The risk tolerance, implicit or otherwise, is not expressed fully or partly quantitatively. To improve, the risk tolerance should be expressed fully quantitatively or as a combination of scenarios with probabilities.

Quotes:

No relevant quotes found.

2.1.1.3 Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks (33%) 0%

No mention of quantitative risk tolerance.

Quotes:

No relevant quotes found.

2.1.2.1 AI developers engage in public consultations or seek guidance from regulators where available (50%) 0%

Whilst they mention external input in the framework overall, it is important for the risk tolerance to specifically be developed with input from the public or regulators.

Quotes:

“We extend our sincere gratitude to the many external groups that provided invaluable guidance on the development and refinement of our Responsible Scaling Policy.”
“This policy is designed in the spirit of the Responsible Scaling Policy (RSP) framework introduced by the non-profit AI safety organization METR, as well as emerging government policy proposals in the UK, EU, and US.”

2.1.2.2 Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses) (50%) 0%

No justification process: No evidence of considering whether their approach aligns with or deviates from established norms.

Quotes:

No relevant quotes found.

2.2.1.1 KRI thresholds are at least qualitatively defined for all risks (45%) 50%

For each risk domain, two qualitative KRIs are defined. They could be grounded in risk modelling, but this is hard to tell given risk models are not clear. To improve, Anthropic could more precisely define these KRIs to reduce further the ambiguity when deciding upon specific thresholds on evaluation results. To improve, KRIs should map directly to evaluation tests performed. Further, KRI thresholds should be more granular for different attack pathways – for instance, “ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons” is quite broad, as CBRN covers four different weapon types.

It would significantly improve their KRI thresholds if they provided reasoning via risk modelling. For instance, they note that “We will consider it sufficient to rule out the possibility that a model has surpassed the two Autonomous AI R&D Capability Thresholds by considering an earlier (i.e., less capable) checkpoint: the ability to autonomously perform a wide range of 2-8 hour software engineering tasks.” However, justification could be given for why 2-8 hour software engineering tasks is an appropriate checkpoint.

Quotes:

“A Capability Threshold is a prespecified level of AI capability that, if reached, signals (1) a meaningful increase in the level of risk if the model remains under the existing set of safeguards (2) a corresponding need to upgrade the safeguards to a higher ASL Standard.”
“Chemical, Biological, Radiological, and Nuclear (CBRN) weapons. The ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons.”
“Autonomous AI Research and Development (AI R&D): The ability to fully automate the work of an entry-level, remote-only Researcher at Anthropic.”
“Cyber Operations: The ability to significantly enhance or automate sophisticated destructive cyber attacks.”

2.2.1.2 KRI thresholds are quantitatively defined for all risks (45%) 25%

AI R&D-5 is quite precisely defined in quantitative terms of what constitutes a “dramatic acceleration in the rate of effective scaling” but they do not indicate how they will measure this. KRIs should map to the actual evaluation tests performed. Further, the other capability thresholds do not have quantitative thresholds.

Quotes:

“AI R&D-4: The ability to fully automate the work of an entry-level, remote-only Researcher at
Anthropic.

AI R&D-5: The ability to cause dramatic acceleration in the rate of effective scaling. Specifically, this would be the case if we observed or projected an increase in the effective training compute of the world’s most capable model that, over the course of a year, was equivalent to two years of the average rate of progress during the period of early 2018 to early 2024. We roughly estimate that the 2018-2024 average scaleup was around 35x per year, so this would imply an actual or projected one-year scaleup of 35^2 = ~1000x.” (p. 16)

“CBRN-3: The ability to significantly assist individuals or groups with basic STEM backgrounds in obtaining, producing, or deploying CBRN weapons. We assess this by comparing what potential attackers could achieve with full model access versus 2023-level online resources, assuming they have funding and up to one year of time to invest, but no initial specialized expertise.

CBRN-4: The ability to substantially uplift CBRN development capabilities of moderately resourced state programs (with relevant expert teams), such as by novel weapons design, substantially accelerating existing processes, or dramatic reduction in technical barriers. We currently define this as uplifting a team of people with skills equivalent to entry-level PhD biologists to be able to approximate the capabilities of a world-class, state-backed bioweapons team.” (p. 16)

2.2.1.3 KRIs also identify and monitor changes in the level of risk in the external environment (10%) 0%

The KRIs only mention model capabilities. They do mention vulnerability reporting channels, which could feasibly lead to KRIs which satisfy this criterion.

Quotes:

ASL-2 Deployment Standard: “Vulnerability reporting channels: Clearly indicated paths within the product for users to report harmful or dangerous model outputs, as well as a bug bounty for universal jailbreaks.” (p. 15)

2.2.2.1 Containment KCIs (35%) 30%

2.2.2.2 Deployment KCIs (35%) 30%

2.2.2.3 For advanced KRIs, assurance process KCIs are defined (30%) 10%

The framework says that they expect that an affirmative case will be required for higher security levels to show they have “mitigated these [misalignment] risks to acceptable levels”. However, they do not indicate what “acceptable levels” constitutes, which is necessary for satisfying this criterion.

Quotes:

For models crossing AI R&D-4: “In addition, we will develop an affirmative case that (1) identifies the most immediate and relevant risks from models pursuing misaligned goals and (2) explains how we have mitigated these risks to acceptable levels. The affirmative case will describe, as relevant, evidence on model capabilities; evidence on AI alignment; mitigations (such as monitoring and other safeguards); and our overall reasoning.” (p. 4)

For AI R&D-5: “We also expect an affirmative case will be required.”

3.1.1.1 Containment measures are precisely defined for all KCI thresholds (60%) 50%

The RSP provides detailed containment measures for ASL-2 (in Appendix B) with specific requirements across six categories. However, even though the ASL-3 containment measures are the ones applied to latest Anthropic models, they are described more as high-level outcomes and examples rather than precise definitions.

Quotes:

“ASL-2 Security Standard: A security system that can likely thwart most opportunistic attackers. 1. Supply chain… 2. Offices… 3. Workforce… 4. Compartmentalization… 5. Infrastructure… 6. Operations…” (Appendix B, Page 15)
“Containment measures are largely information security measures that allow controlling access to the model for various stakeholders. For the potential loss of control risks, containment also includes containing an agentic AI model. Examples include extreme isolation of weight storage, strict application allow-listing, and advanced insider threat programs” (p. 8)

“Perimeters and access controls: Building strong perimeters and access controls around sensitive assets to ensure AI models and critical systems are protected from unauthorized access. We expect this will include a combination of physical security, encryption, cloud security, infrastructure policy, access management, and weight access minimization and monitoring” (p. 8)

“Existing guidance: Aligning where appropriate with existing guidance on securing model weights, including Securing AI Model Weights, Preventing Theft and Misuse of Frontier Models (2024); security recommendations like Deploying AI Systems Securely (CISA/NSA/FBI/ASD/CCCS/GCSB /GCHQ), ISO 42001, CSA’s AI Safety Initiative, and CoSAI; and standards frameworks like SSDF, SOC 2, NIST 800-53.” (p. 10)

3.1.1.2 Proof that containment measures are sufficient to meet the thresholds (40%) 25%

The RSP provides a high-level description of a process for evaluating whether containment measures meet requirements, but does not detail the proof or evidence for why they believe their measures will likely be sufficient. Instead, this evidence need only be collated when ASL-3 requirements need to be implemented. To improve, they should prove ex ante that the requirements are sufficient, to leave as little discretion as possible and ensure risk levels remain below the risk tolerance at all times.

Quotes:

“Audits: Develop plans to (1) audit and assess the design and implementation of the security program and (2) share these findings (and updates on any remediation efforts) with management on an appropriate cadence.” (p. 10)
“If, after the evaluations above, we determine that we have met the ASL-3 Required Safeguards, then we may proceed with deploying and training models above the Capability Threshold” (p. 10)

3.1.1.3 Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2]) 25%

In the containment measures section of their framework (i.e. the description of ASL-3), Anthropic describes comprehensive third-party assessment of their containment measures, but does not explicitly commit to it: “we expect this to include independent validation of threat modeling and risk assessment results…”

To improve, the framework should detail the actual intended process for verifying that the containment measures meet the containment KCI threshold, and ideally in advance as possible (e.g. prior to development) to ensure the KRI threshold is not crossed before the KCI measures are decided.

In a separate section dedicated to transparency and external input, they state that they will solicit input from external experts, but it’s unclear whether this applies specifically to containment measures.

Quotes:

“Audits: Develop plans to (1) audit and assess the design and implementation of the security program and (2) share these findings (and updates on any remediation efforts) with management on an appropriate cadence. We expect this to include independent validation of threat modeling and risk assessment results; a sampling-based audit of the operating effectiveness of the defined controls; periodic, broadly scoped, and independent testing with expert red-teamers who are industry-renowned and have been recognized in competitive challenges.” (p. 10)

“Expert input: We will solicit input from external experts in relevant domains in the process of developing and conducting capability and safeguards assessments.” (p. 13)

3.1.2.1 Deployment measures are precisely defined for all KCI thresholds (60%) 50%

Similar to containment measures, the RSP provides detailed deployment measures for ASL-2 but only high-level criteria for ASL-3. For instance, ASL-3 measures must satisfy certain evaluation criteria and principles (like “defense in depth”), but precisely defined deployment measures for all KCI thresholds are not given. Instead, they focus more on outcomes. To improve, the measures they will implement for ASL-3 Deployment Standard should be detailed – this is especially necessary given they currently have models which are deployed under ASL-3.

Quotes:

“ASL-2 Deployment Standard: 1. Acceptable use policies and model cards… 2. Harmlessness training and automated detection… 3. Fine-tuning protections… 4. Vulnerability reporting channels…” (Appendix B, p. 15)

“When a model must meet the ASL-3 Deployment Standard, we will evaluate whether the measures we have implemented make us robust to persistent attempts to misuse the capability in question […] To make the required showing, we will need to satisfy the following criteria: 1. Threat modeling… 2. Defense in depth… 3. Red-teaming… 4. Rapid remediation… 5. Monitoring… 6. Trusted users… 7. Third-party environments…” (p. 8)

“Defense in depth: Use a “defense in depth” approach by building a series of defensive layers, each designed to catch misuse attempts that might pass through previous barriers. As an example, this might entail achieving a high overall recall rate using harm refusal techniques. This is an area of active research, and new technologies may be added when ready” (p. 8)

3.1.2.2 Proof that deployment measures are sufficient to meet the thresholds (40%) 25%

The RSP provides a high-level description of a red teaming process for evaluating whether deployment measures meet requirements. However, it doesn’t provide actual proof or evidence that the deployment measures are sufficient ex ante. Instead, it relies on Antrhopic’s judgment at the time when ASL-3 deployment standards need to be implemented, making the decision vulnerable to discretion.

Quotes:

“Red-teaming: Conduct red-teaming that demonstrates that threat actors with realistic access levels and resources are highly unlikely to be able to consistently elicit information from any generally accessible systems that greatly increases their ability to cause catastrophic harm relative to other available tools.” (p. 8)

3.1.2.3 Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2]) 10%

In a general section of their framework dedicated to transparency and external input, Anthropic state that they will solicit input from external experts, but it’s unclear whether this applies specifically to deployment measures.

Quotes:

“We will solicit input from external experts in relevant domains in the process of developing and conducting capability and safeguards assessments.” (p. 13)

3.1.3.1 Credible plans towards the development of assurance properties (40%) 25%

Anthropic acknowledges that assurance processes don’t yet exist and commits to developing them for advanced AI R&D capabilities. They promise to create an “affirmative case” that identifies alignment risks and explains their mitigations, and say they’ll “continue to research potential risks and next-generation mitigation techniques.”
However, these are vague, high-level commitments without concrete details. Despite noting that current mitigations are insufficient for future capabilities, they provide no roadmap for how they’ll actually develop the necessary assurance processes.

Quotes:

“AI R&D-4: The ability to fully automate the work of an entry-level, remote-only Researcher at Anthropic. The ASL-3 Security Standard is required. In addition, we will develop an affirmative case that (1) identifies the most immediate and relevant risks from models pursuing misaligned goals and (2) explains how we have mitigated these risks to acceptable levels.” (p. 4)

“Since the frontier of AI is rapidly evolving, we cannot anticipate what safety and security measures will be appropriate for models far beyond the current frontier. We will thus regularly measure the capability of our models and adjust our safeguards accordingly. Further, we will continue to research potential risks and next-generation mitigation techniques.” (p. 1)

3.1.3.2 Evidence that the assurance properties are enough to achieve their corresponding KCI thresholds (40%) 10%

Anthropic acknowledges that assurance processes don’t yet exist and commits to developing them for advanced AI R&D capabilities. They promise to create an “affirmative case” that identifies alignment risks and explains their mitigations, and say they’ll “continue to research potential risks and next-generation mitigation techniques.”
However, these are vague, high-level commitments without concrete details. Despite noting that current mitigations are insufficient for future capabilities, they provide no roadmap for how they’ll actually develop the necessary assurance processes.

Quotes:

“AI R&D-4: The ability to fully automate the work of an entry-level, remote-only Researcher at Anthropic. The ASL-3 Security Standard is required. In addition, we will develop an affirmative case that (1) identifies the most immediate and relevant risks from models pursuing misaligned goals and (2) explains how we have mitigated these risks to acceptable levels.” (p. 4)

“Since the frontier of AI is rapidly evolving, we cannot anticipate what safety and security measures will be appropriate for models far beyond the current frontier. We will thus regularly measure the capability of our models and adjust our safeguards accordingly. Further, we will continue to research potential risks and next-generation mitigation techniques.” (p. 1)

3.1.3.3 The underlying assumptions that are essential for their effective implementation and success are clearly outlined (20%) 10%

The framework mentions that the “affirmative case” for assurance processes’ efficacy will include “overall reasoning,” which would presumably encompass underlying assumptions. However, no concrete implementation of this is provided, nor examples of assumptions made. To improve, the framework should detail the key technical assumptions necessary for the assurance processes to meet the KCI threshold, and evidence for why these assumptions are justified.

Quotes:

“In addition, we will develop an affirmative case that (1) identifies the most immediate and relevant risks from models pursuing misaligned goals and (2) explains how we have mitigated these risks to acceptable levels. The affirmative case will describe, as relevant, evidence on model capabilities; evidence on AI alignment; mitigations (such as monitoring and other safeguards); and our overall reasoning.” (p. 4)

3.2.1.1 Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors (30%) 75%

The framework acknowledges the need to match realistic attacker capabilities and lists some elicitation methods used (scaffolding, fine tuning, expert prompting). However, it doesn’t provide quantitative specifics, such as how much compute is used for fine tuning. More detail could be added on which elicitation methods they anticipate would be used by different threat actors, under realistic settings, to justify their elicitation method.

Quotes:

“Elicitation: Demonstrate that, when given enough resources to extrapolate to realistic attackers, researchers cannot elicit sufficiently useful results from the model on the relevant tasks. We should assume that jailbreaks and model weight theft are possibilities, and therefore perform testing on models without safety mechanisms (such as harmlessness training) that could obscure these capabilities.” (p. 6)

“We will also consider the possible performance increase from using resources that a realistic attacker would have access to, such as scaffolding, finetuning, and expert prompting. At minimum, we will perform basic finetuning for instruction following, tool use, minimizing refusal rates.” (p. 6)

“By ‘widely accessible,’ we mean techniques that are available to a moderately resourced group (i.e., do not involve setting up large amounts of custom infrastructure or using confidential information).” (Footnote 6, p. 6)

3.2.1.2 Evaluation frequency (25%) 100%

The framework clearly specifies evaluation frequency in terms of effective computing power (4x increase triggers comprehensive assessment). This is a quantitative threshold that directly addresses the criterion with appropriate detail. The framework clearly specifies a 6-month fixed time interval for accounting for post-training enhancements.

Quotes:

“The term ‘notably more capable’ is operationalized as at least one of the following: 1. The model is notably more performant on automated tests in risk-relevant domains (defined as 4x or more in Effective Compute).” (pp. 5-6)

“Adjusted evaluation cadence: We adjusted the comprehensive assessment cadence to 4x Effective Compute or six months of accumulated post-training enhancements (this was previously three months).” (p. 17)

“Six months’ worth of finetuning and other capability elicitation methods have accumulated. This is measured in calendar time, since we do not yet have a metric to estimate the impact of these improvements more precisely.” (p. 6)

3.2.1.3 Description of how post-training enhancements are factored into capability assessments (15%) 50%

The policy acknowledges the importance of accounting for “widely accessible” post-training enhancements in capability assessments. They ensure this buffer for post-training enhancements takes account of both the possibility of enhancing the capabilities of Anthropic’s deployments, as well as the capabilities of Anthropic’s models if they are stolen: “We include headroom to account for the possibility that the model is either modified via one of our own finetuning products or stolen in the months following testing, and used to create a model that has reached a Capability Threshold.” These both show nuance.

However, whilst they focus on “widely accessible post-training enhancements”, to improve they should qualify why they focus only on these enhancements.

It is commendable that they note that “exploring ways to integrate [post-training enhancements] into an overall metric is an ongoing area of research”, though an improvement would be to commit to contribute to this research.

Further, more detail could be added on how they account(ed) for how post-training enhancements’ risk profiles change with different model structures – namely, post-training enhancements are much more scalable with reasoning models, as inference compute can often be scaled to improve capabilities.

Quotes:

“For models requiring comprehensive testing, we will assess whether the model is unlikely to reach any relevant Capability Thresholds absent surprising advances in widely accessible post-training enhancements” (p. 6) and “By “widely accessible,” we mean techniques that are available to a moderately resourced group (i.e., do not involve setting up large amounts of custom infrastructure or using confidential information). We include headroom to account for the possibility that the model is either modified via one of our own finetuning products or stolen in the months following testing, and used to create a model that has reached a Capability Threshold. That said, estimating these future effects is very difficult given the state of research today” (Footnote 6, p. 6)

A model is notably capable if: “Six months’ worth of finetuning and other capability elicitation methods have accumulated. This is measured in calendar time, since we do not yet have a metric to estimate the impact of these improvements more precisely” (p. 6) and “Exploring ways to integrate these types of improvements into an overall metric is an ongoing area of research.” (Footnote 5, p. 6)

3.2.1.4 Vetting of protocols by third parties (15%) 10%

The policy mentions soliciting external expert input in developing and conducting capability assessments, which partially addresses protocol vetting. However, this is general input rather than a strong commitment specific to the vetting of evaluation protocols. Further, details on the level of expertise required, and why their chosen experts satisfy this criteria, should be detailed.

Quotes:

“Expert input: We will solicit input from external experts in relevant domains in the process of developing and conducting capability and safeguards assessments.” (p. 13)

3.2.1.5 Replication of evaluations by third parties (15%) 50%

The framework mentions in multiple stages of the risk assessment process that they will share materials related to the evaluation and seek input from experts, but not that these expert will reproduce or audit the results directly. To improve, a process for having evaluations externally assessed or audited should be detailed.

Quotes:

“To advance the public dialogue on the regulation of frontier AI model risks and to enable examination of our actions, we will also publicly release key materials related to the evaluation and deployment of our models with sensitive information removed and solicit input from external experts in relevant domains.” (p. 13)

“We will solicit both internal and external expert feedback on the [Capability] report” (p. 7)

“Expert input: We will solicit input from external experts in relevant domains in the process of developing and conducting capability and safeguards assessments. We may also solicit external expert input prior to making final decisions on the capability and safeguards assessments.” (p. 13)

3.2.2.1 Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed (40%) 50%

The framework provides a high-level description of monitoring procedures for deployment measures, with examples such as “jailbreak bounties, doing historical analysis or background monitoring, and any necessary retention of logs for these activities.” To improve, they should define what monitoring “on a reasonable cadence” is defined as.

They also mention that they will develop plans to audit the implementation of containment measures, but there is no commitment to audit assurance processes. They describe redteaming of the model with deployment measures to “[demonstrate] that threat actors with realistic access levels and resources are highly unlikely to be able to consistently elicit information from any generally accessible systems that greatly increases their ability to cause catastrophic harm relative to other available tools”.

However, they note that “This criterion does not attempt to specify the exact red-teaming protocol (e.g., number of hours, level of access, or pass-fail criteria). Setting a principled pass-fail threshold will depend on other factors, such as the quality of our monitoring and ability to respond to jailbreaks rapidly.” An improvement would be to specify the protocol as much as possible, to ensure transparency, and to conduct this redteaming continuously or at regular cadence.

It is commendable that they note the importance of “prespecify[ing] empirical evidence that would show the system is oeprating within the accepted risk range” for monitoring.

However, to improve, the framework should describe systematic, ongoing monitoring to ensure mitigation effectiveness is tracked continuously such that the KCI threshold will still be met, when required.

Quotes:

“Monitoring: Prespecify empirical evidence that would show the system is operating within the accepted risk range and define a process for reviewing the system’s performance on a reasonable cadence. Process examples include monitoring responses to jailbreak bounties, doing historical analysis or background monitoring, and any necessary retention of logs for these activities.” (p. 8)

“Audits: Develop plans to (1) audit and assess the design and implementation of the security program and (2) share these findings (and updates on any remediation efforts) with management on an appropriate cadence” (p. 10)

3.2.2.2 Vetting of protocols by third parties (30%) 25%

The policy mentions soliciting external expert input in developing and conducting safeguard assessments, which partially addresses protocol vetting. However, this is general input rather than a strong commitment specific to the vetting of evaluation protocols.

Quotes:

“Expert input: We will solicit input from external experts in relevant domains in the process of developing and conducting capability and safeguards assessments. We may also solicit external expert input prior to making final decisions on the capability and safeguards assessments.” (p. 13)

“Audits: Develop plans to (1) audit and assess the design and implementation of the security program and (2) share these findings (and updates on any remediation efforts) with management on an appropriate cadence. We expect this to include independent validation of threat modeling and risk assessment results” (p. 10)

3.2.2.3 Replication of evaluations by third parties (30%) 50%

The framework mentions that they will share materials related to KCI evaluations (i.e. safeguard assessments) and seek input from experts, but not that experts will reproduce the results directly.

Quotes:

“To advance the public dialogue on the regulation of frontier AI model risks and to enable examination of our actions, we will also publicly release key materials related to the evaluation and deployment of our models with sensitive information removed and solicit input from external experts in relevant domains.” (p. 13)

“Expert input: We will solicit input from external experts in relevant domains in the process of developing and conducting capability and safeguards assessments. We may also solicit external expert input prior to making final decisions on the capability and safeguards assessments.” (p. 13)

“Audits: Develop plans to (1) audit and assess the design and implementation of the security program and (2) share these findings (and updates on any remediation efforts) with management on an appropriate cadence. We expect this to include independent validation of threat modeling and risk assessment results” (p. 10)

3.2.3.1 Sharing of evaluation results with relevant stakeholders as appropriate (85%) 90%

The policy demonstrates strong commitment to sharing evaluation results with multiple stakeholders: the public (summaries), government entities, internal staff, Board of Directors, and the Long-Term Benefit Trust. Multiple channels and levels of disclosure are specified. There is a commitment to notifying a relevant authority if “a model requires stronger protections than the ASL-2 Standard.” They commit to publicly releasing “key information related to the evaluation and deployment of our models” – to improve, they should commit to publishing all KRI and KCI assessments (with sensitive information redacted).

Quotes:

“Public disclosures: We will publicly release key information related to the evaluation and deployment of our models (not including sensitive details). These include summaries of related Capability and Safeguards reports when we deploy a model” (p. 13)

“U.S. Government notice: We will notify a relevant U.S. Government entity if a model requires stronger protections than the ASL-2 Standard.” (p. 13)

“We will share summaries of Capability Reports and Safeguards Reports with Anthropic’s regular-clearance staff, redacting any highly-sensitive information.” (p. 12)

“The CEO and RSO decide to proceed with deployment, they will share their decision–as well as the underlying Capability Report, internal feedback, and any external feedback–with the Board of Directors and the Long-Term Benefit Trust before moving forward.” (p. 7)

3.2.3.2 Commitment to non-interference with findings (15%) 0%

No commitment to permitting the reports, which detail the results of external evaluations (i.e. any KRI or KCI assessments conducted by third parties), to be written independently and without interference or suppression.

Quotes:

No relevant quotes found.

3.2.4.1 Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains (50%) 0%

Despite noting that “for each capability threshold, [we will] make a compelling case that we have mapped out the most likely and consequential threat models: combinations of actors (if relevant), attack pathways, model capability bottlenecks, and types of harms. We also make a compelling case that
there does not exist a threat model that we are not evaluating that represents a substantial amount of risk”, there does not appear to be a process for identifying novel risks post-deployment which could signal alternative threat models. Hence, their risk modelling appears to be mostly informed a priori, rather than from empirical data of the model in deployment. To improve, they could establish a process for actively searching for novel risks or changed risk profiles of models.

They do mention “periodic, broadly scoped, and independent testing with expert red-teamers” for auditing their security programs. However, this is not for the purpose of arising novel risk profiles, so credit is not given.

Quotes:

No relevant quotes found.

3.2.4.2 Mechanism to incorporate novel risks identified post-deployment (50%) 10%

There is an indication that if novel risk models could exist that they had not considered, an effort will be made to incorporate this into their risk assessment. One indication of this is the general intention to incorporate findings from evaluations, which may uncover new risk profiles of models: “Findings from partner organizations and external evaluations of our models (or similar models) should also be incorporated into the final assessment, when available.”

They also mention that “as our understanding evolves, we may identify additional [capability] thresholds.” However, this statement does not explicitly commit to incorporating novel risks into their risk identification and prioritization process. They note that they will maintain a “list of capabilities that we think require significant investigation” and “could pose serious risks, but the exact Capability Threshold and the Required Safeguards are not clear at present.” This somewhat gives an indication that additional risks may be incorporated into their risk assessment. To improve, they could commit to engaging in risk modelling to map out the potential harms from risk profiles that may have changed, in order to ensure they are keeping in touch with the evolving risk landscape.

Quotes:

“Findings from partner organizations and external evaluations of our models (or similar models) should also be incorporated into the final assessment, when available.” (p. 6)

“These Capability Thresholds represent our current understanding of the most pressing catastrophic risks. As our understanding evolves, we may identify additional thresholds. For each threshold, we will identify and describe the corresponding Required Safeguards as soon as feasible, and at minimum before training or deploying any model that reaches that threshold.”

“We will also maintain a list of capabilities that we think require significant investigation and may require stronger safeguards than ASL-2 provides. This group of capabilities could pose serious risks, but the exact Capability Threshold and the Required Safeguards are not clear at present. These capabilities may warrant a higher standard of safeguards, such as the ASL-3 Security or Deployment Standard. However, it is also possible that by the time these capabilities are reached, there will be evidence that such a standard is not necessary (for example, because of the potential use of similar capabilities for defensive purposes). Instead of prespecifying particular thresholds and safeguards today, we will conduct ongoing assessments of the risks with the goal of determining in a future iteration of this policy what the Capability Thresholds and Required Safeguards would be.” (p. 5)