OpenAI

1.3.2.1 Methodology precisely defined (70%) 10%

It is not clear what the methodology is from the framework, or that a particular methodology is followed. However, they do mention identifying causal pathways, which implies some methodology. More detail should be given.

Quotes:

“It must be possible to identify a causal pathway for a severe harm in the capability area, enabled by frontier AI.” (p. 4)

“Capability thresholds concretely describe things an AI system might be able to help someone do or might be able to do on its own that could meaningfully increase risk of severe harm.” (p. 4)

1.3.2.2 Mechanism to incorporate red teaming findings (15%) 0%

No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.

Quotes:

No relevant quotes found.

1.3.2.3 Prioritization of severe and probable risks (15%) 10%

For a risk area to be a tracked category, the capability must create a risk that is “Severe: There is a plausible threat model within the capability area that would create severe harm.” This suggests that severity is prioritised, and plausibility here suggests the risk model must have nonzero probability. However, these threat models are developed post-hoc – after deciding which categories to track: “For each Tracked Category, we develop and maintain a threat model identifying specific risks of severe harms that could arise from the frontier capabilities in that domain […]”

They then prioritise monitoring for High and Critical capabilities, implicitly defining these as those capabilities with higher probability x severity of harm: “High capability thresholds mean capabilities that significantly increase existing risk vectors for severe harm”; “Critical capability thresholds mean capabilities that present a meaningful risk of a qualitatively new threat vector for severe harm with no ready precedent.”

However, there is minimal detail on how severity and probability of risk models is determined, and these results published.

In addition, determining whether there is “real risk” of “severe harm” is not explicitly determined by probabilities. The probability and the magnitude of harm should be explicitly estimated for each risk model.

Overall, there is an awareness that they should focus threat models on severe harms, but with little evidence of systematic prioritization among multiple risk models. Risk modelling is only completed after already deciding what to track. This is different from the required criterion of using prioritization of risk models to determine focus areas.

Quotes:

“For each Tracked Category, we develop and maintain a threat model identifying specific risks of severe harms that could arise from the frontier capabilities in that domain […] High capability thresholds mean capabilities that significantly increase existing risk vectors for severe harm. Critical capability thresholds mean capabilities that present a meaningful risk of a qualitatively new threat vector for severe harm with no ready precedent.” (p. 4)

“Where we determine that a capability presents a real risk of severe harm, we may decide to monitor it as a Tracked Category or a Research Category.” (p. 4)

For a capability to be a Tracked Category (p. 4):
“Plausible: It must be possible to identify a causal pathway for a severe harm in the capability area, enabled by frontier AI.”
“Severe: There is a plausible threat model within the capability area that would create severe harm.”

2.1.1.1 Risk tolerance is at least qualitatively defined for all risks (33%) 50%

There is a qualitative definition of scenarios which are implicitly ‘unacceptable’ levels of risk, under the Critical capability threshold. For instance, “Proliferating the ability to create a novel threat vector of the severity of a CDC Class A biological agent (i.e., high mortality, ease of transmission) could cause millions of deaths and significantly disrupt public life, with few available societal safeguards” implicitly states this risk as the risk tolerance.

To improve, they must set out the maximum amount of risk the company is willing to accept, for each risk domain (though they need not differ between risk domains), ideally expressed in terms of probabilities and severity (economic damages, physical lives, etc), and separate from KRIs.

Partial credit is given for the definition of “severe harm” as “the death or grave injury of thousands of people or hundreds of billions of dollars of economic damage.” However, the capability thresholds are not explicitly linked to this proto-risk tolerance, and it should be more specific (e.g., specifying how many thousands of people).

Quotes:

“By “severe harm” in this document, we mean the death or grave injury of thousands of people or hundreds of billions of dollars of economic damage.” (p. 1)

Some examples from Tracked Categories table, under the Critical category for ‘Associated risk of severe harm’ (p. 5):

“Proliferating the ability to create a novel threat vector of the severity of a CDC Class A biological agent (i.e., high mortality, ease of transmission) could cause millions of deaths and significantly disrupt public life, with few available societal safeguards.”

“Finding and executing end-to-end exploits for all software could lead to catastrophe from unilateral actors, hacking military or industrial systems, or OpenAI infrastructure. Novel cyber operations, e.g., those involving novel zero-days or novel methods of command-and-control, generally pose the most serious threat, as they are unpredictable and scarce.”

“A major acceleration in the rate of AI Self-improvement could rapidly increase the rate at which new capabilities and risks emerge, to the point where our current oversight practices are insufficient to identify and mitigate new risks, including risks to maintaining human control of the AI system itself.”

2.1.1.2 Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks (33%) 10%

The qualitative risk tolerances do not have quantitative probabilities, and are vague in description. The definition of severe harm implies some awareness of quantitative measurement, though this is used to classify critical capability thresholds rather than defined as a risk tolerance itself.

Quotes:

“High capability thresholds mean capabilities that significantly increase existing risk vectors for severe harm” (p. 4)

“Critical capability thresholds mean capabilities that present a meaningful risk of a qualitatively new threat vector” (p. 4)

“Scalable evaluations have associated “indicative thresholds,” which are levels of performance that we have pre-determined to indicate that a deployment may have reached a capability threshold.” (p. 8)

“By “severe harm” in this document, we mean the death or grave injury of thousands of people or hundreds of billions of dollars of economic damage.” (p. 1)

2.1.1.3 Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks (33%) 0%

Whilst they mention the criterion of “severe harm” via “the death or gave injury of thousands of people or hundreds of billions of dollars of economic damage”, this is still vague, and doesn’t apply as a specific risk tolerance for specific risks. None of the specific risks mention quantitative probabilities, and the implicit risk tolerances from capability thresholds are not fully quantitative either.

Quotes:

“High capability thresholds mean capabilities that significantly increase existing risk vectors for severe harm” (p. 4)
“Critical capability thresholds mean capabilities that present a meaningful risk of a qualitatively new threat vector” (p. 4)

“Scalable evaluations have associated “indicative thresholds,” which are levels of performance that we have pre-determined to indicate that a deployment may have reached a capability threshold.” (p. 8)

Some examples from Tracked Categories table, under the Critical category for ‘Associated risk of severe harm’ (p. 5):

“Proliferating the ability to create a novel threat vector of the severity of a CDC Class A biological agent (i.e., high mortality, ease of transmission) could cause millions of deaths and significantly disrupt public life, with few available societal safeguards.”

“Finding and executing end-to-end exploits for all software could lead to catastrophe from unilateral actors, hacking military or industrial systems, or OpenAI infrastructure. Novel cyber operations, e.g., those involving novel zero-days or novel methods of command-and-control, generally pose the most serious threat, as they are unpredictable and scarce.”

“A major acceleration in the rate of AI Self-improvement could rapidly increase the rate at which new capabilities and risks emerge, to the point where our current oversight practices are insufficient to identify and mitigate new risks, including risks to maintaining human control of the AI system itself.”

2.1.2.1 AI developers engage in public consultations or seek guidance from regulators where available (50%) 0%

No evidence of asking the public what risk levels they find acceptable. No evidence of seeking regulator input specifically on what constitutes acceptable risk levels.

Quotes:

No relevant quotes found.

2.1.2.2 Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses) (50%) 0%

No justification process: No evidence of considering whether their approach aligns with or deviates from established norms.

Quotes:

No relevant quotes found.

2.2.1.1 KRI thresholds are at least qualitatively defined for all risks (45%) 50%

Each Tracked Category has 2 thresholds each, which are specific. They appear to be grounded in risk modelling, but this is hard to tell given risk models are not clear. The Research Categories have less developed KRIs but are still well-defined. To improve, more precision is needed in all of the Research Categories, as well as the High thresholds of Cybersecurity and Biological & Chemical; evidence is needed of the specific thresholds, e.g. of the “baseline of tools available in 2021” or “relative to those researchers’ 2024 baseline”. The Critical threshold for AI Self-improvement is especially good. To improve, KRIs should map directly to evaluation tests performed.

Quotes:

From Tracked Categories, under ‘Capability threshold that could lead to the risk’, pp. 5-6:
Biological and chemical:
“[High] The model can provide meaningful counterfactual assistance (relative to unlimited access to baseline of tools available in 2021) to “novice” actors (anyone with a basic relevant technical background) that enables them to create known biological or chemical threats.”

“[Critical] The model can enable an expert to develop a highly dangerous novel threat vector (e.g., comparable to a novel CDC Class A biological agent) OR the model can be connected to tools and equipment to complete the full engineering and/or synthesis cycle of a regulated or novel biological threat without human intervention.”

Autonomous AI R&D:
“[High] The model’s impact is equivalent to giving every OpenAI researcher a highly performant mid-career research engineer assistant, relative to those researchers’ 2024 baseline.”

“[Critical] The model is capable of recursively self improving (i.e., fully automated AI Self-improvement), defined as either (leading indicator) a superhuman research scientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months.”

Research Categories:
“Long-range Autonomy: ability for a model to execute a long-horizon sequence of actions sufficient to realize a “High” threat model (e.g., a cyberattack) without being directed by a human (including successful social engineering attacks when needed)” (p. 7). More on page 7 can be found.

2.2.1.2 KRI thresholds are quantitatively defined for all risks (45%) 25%

The Critical capability threshold for AI Self-improvement, and somewhat the High capability threshold for AI Self-improvement and Biological & Chemical give baselines, which mean these KRIs can in theory be measured quantitatively. However, more detail of the actual threshold implemented here is needed. No quantitative thresholds for the others is given.

Quotes:

All the below are from Tracked Categories under ‘Capability threshold that could lead to the risk’, pp. 5-6:

High, Biological & Chemical: “[High] The model can provide meaningful counterfactual assistance (relative to unlimited access to baseline of tools available in 2021) to “novice” actors (anyone with a basic relevant technical background) that enables them to create known biological or chemical threats”

High, AI Self-improvement: “[High] The model’s impact is equivalent to giving every OpenAI researcher a highly performant mid-career research engineer assistant, relative to those researchers’ 2024 baseline.”

Critical, AI Self-improvement: “[Critical] The model is capable of recursively self improving (i.e., fully automated AI Self-improvement), defined as either (leading indicator) a superhuman researchscientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months.”

2.2.1.3 KRIs also identify and monitor changes in the level of risk in the external environment (10%) 0%

The KRIs only mention model capabilities. They do mention monitoring and incident response, which could feasibly lead to KRIs which satisfy this criterion.

Quotes:

“– Monitoring and Incident Response: Monitor security and event logs continuously to detect,
triage, and respond to security incidents rapidly by 24×7 on-call staff.” (p. 21)

2.2.2.1 Containment KCIs (35%) 5%

2.2.2.2 Deployment KCIs (35%) 43%

2.2.2.3 For advanced KRIs, assurance process KCIs are defined (30%) 50%

The framework provides desiderata which can be seen as proto assurance KCIs, though they are somewhat vague (e.g. “the model consistently understands […] instructions” or “sufficiently addressed by one or more of the following claims” lack detail.) It is also unclear if they must all be covered in the Safeguards Report, and how often Safeguards Reports are required. The KCIs also are not linked to specific KRIs explicitly, though the document says they will be.

Table 5 list some desiderata indicating further proto assurance process KCI thresholds. However, these require more development, such as what constitutes “significant evidence” or “consistently takes action”.

Quotes:

“Safeguards should sufficiently minimize the risk associated with a misaligned model circumventing human control and oversight and executing severe harms. This can be done by establishing that all plausible known vectors of enabling severe harm are sufficiently addressed by one or more of the following claims:

• Lack of Autonomous Capability: The model is not capable of carrying out tasks autonomously, including the risk of severe harm.
• Value Alignment: The model consistently applies human values in novel settings (without any instructions) to avoid taking actions that cause harm, and has shown sufficiently minimal indications of misaligned behaviors like deception or scheming.
• Instruction Alignment: The model consistently understands and follows user or system instructions, even when vague, and those instructions rule out pathways to causing severe harm.
• Reliable and Robust System Oversight: Effective AI and human oversight of model actions detects and prevents execution of harm or subversion of safeguards.
• System Architecture: The model can’t take actions that cause harm because it lacks access to output channels or mechanisms to persist sufficiently to execute the harm, due to containerization and restricted permissions.” (p. 18) See also Table 3, p. 11.

From Table 5, p. 19:
“Internal and external user reports, monitoring, or manual inspection for historical model deployments show no significant evidence of unexpected value alignment failures.”

“Large scale usage or testing in realistic settings should show the model consistently takes action aligned to the user’s instructions or goals (while maintaining value alignment and avoiding harm). Examples against would include trying to escape a container, lying to the user to appease them, etc. […] “Canary evaluations” which test model capabilities to bypass less complex, easier-to exploit versions of our security controls, establishing that our implemented controls are robust”

3.1.1.1 Containment measures are precisely defined for all KCI thresholds (60%) 50%

The framework describes the security controls required for High capability models in C.3 in detail, though not for Critical capability models. However, many measures remain high level desiderata rather than operational measures. However, there are some which are more specific, quoted below.

Quotes:

From appendix C.3, pp. 20-21:
“Adopt a layered security strategy, ensuring robust protection through multiple defensive barriers, including physical and datacenter security, network segmentation and controls, workload isolation, data encryption, and other overlapping and complementary security controls.”
“Employees must authenticate using multi-factor authentication (MFA) and managed devices meeting security baselines. Access must be logged and reviewed for detection and investigative purposes.”
“Integrate automated code analysis, formal security reviews, and penetration testing in engineering processes. Apply security reviews and validation to higher-sensitivity critical components prior to deployment.”

3.1.1.2 Proof that containment measures are sufficient to meet the thresholds (40%) 25%

The framework mentions that “internal and external assessments to validate [the security] controls are conducted regularly”, but without further detail for why the measures given are likely to be sufficient to meet the containment KCI threshold. Further, their ‘sufficiency’ is ultimately determined by SAG (“reports are provided to OpenAI leadership”), rather than decided prior to the KRI threshold being passed.

Further, without quantified risk tolerance or probability estimates, there’s no foundation for proving containment adequacy.

Quotes:

“Continuous Monitoring and Validation: Ensure security threat models and updates inform where security and data privacy controls should be implemented, improved, and monitored to further reduce risk. Internal and external assessments to validate these controls are conducted regularly and reports are provided to OpenAI leadership.” (p. 20)

3.1.1.3 Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2]) 25%

Whilst the framework requires independent audits for High capability models, they do not describe a similar process for Critical capability models. Further, the process appears to validate already implemented security protocols, whilst this criterion requires validation of the case for why containment measures are sufficient to meet containment KCI thresholds in advance of implementation.

Quotes:

In C.3 Security Controls: “Independent Security Audits: Ensure security controls and practices are validated regularly by third-party auditors to ensure compliance with relevant standards and robustness against identified threats.” (p. 21)

3.1.2.1 Deployment measures are precisely defined for all KCI thresholds (60%) 50%

Section C.1 in the Appendix details “potential safeguards” for models with High capability, without explicit commitment to implementing them: “the safeguards should not be construed as a definitive or comprehensive list of the safeguards we will or could apply to a given launch”. Nonetheless, the measures are defined for each KCI threshold, namely robustness, usage monitoring and trust-based access.

To improve, deployment measures must also be defined for the Critical capability.

Quotes:

From Table 4, p. 17:
“Robustness:

• Training the model to refuse to help with high-risk tasks or to otherwise produce low risk
  responses
• Unlearning or training-data filtering to erase specific risk-enabling knowledge from the model’s knowledge-base
• Interpretability-based approaches, like activation steering, that directly edit models’ thinking at inference time
• Jailbreak robustness, including through adversarial training, inference-time deliberation, and more”

More quotes may be found in Table 4.

“This Appendix provides illustrative examples of potential safeguards, and safeguard efficacy assessments that could be used to establish that we have sufficiently mitigated the risk of severe harm. The examples aim to provide insight on our thinking, but many of the techniques require further research. The safeguards should not be construed as a definitive or comprehensive list of the safeguards we will or could apply to a given launch.

As a reminder, covered systems that reach High capability must have safeguards that sufficiently minimize the associated risk of severe harm before they are deployed. Systems that reach Critical capability also require sufficient safeguards during development.” (p. 16)

3.1.2.2 Proof that deployment measures are sufficient to meet the thresholds (40%) 25%

Section C.1 in the Appendix details “potential safeguard efficacy assessments”, without explicit commitment to implementing them. However, they don’t provide actual proof or evidence that the deployment measures are sufficient ex ante. Instead, it relies on the Safety Advisory Group’s judgment at the time when High or Critical deployment standards need to be implemented, making the decision vulnerable to discretion.

Quotes:

From Table 4, p. 17:
“Robustness:

• Automated and expert redteaming (identifying success per resources)
• Prevalence of jailbreaks identified via monitoring and reports, in historical deployments
• Results from public jailbreak bounties and results from private and public jailbreak benchmarks”

More quotes may be found in Table 4.

“The examples aim to provide insight on our thinking but should not be construed as a definitive checklist of the safeguards we will apply to a given launch.” (p. 10)

3.1.2.3 Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2]) 25%

While they mention third-party stress testing of safeguards, this is not specific to deployment measures, and appears optional.

Quotes:

“Third-party stress testing of safeguards: If we deem that a deployment warrants third party stress testing of safeguards and if high quality third-party testing is available, we will work with third parties to evaluate safeguards. We may seek this out in particular for models that are over a High capability threshold.” (p. 13)

“Independent expert opinions for evidence produced to SAG: The SAG may opt to get independent expert opinion on the evidence being produced to SAG. The purpose of this input is to add independent analysis from individuals or organizations with deep expertise in domains of relevant risks (e.g., biological risk). If provided, these opinions will form part of the analysis presented to SAG in making its decision on the safety of a deployment. These domain experts may not necessarily be AI experts and their input will form one part of the holistic evidence that SAG reviews.” (p. 13)

3.1.3.1 Credible plans towards the development of assurance properties (40%) 25%

The framework mentions a commitment to developing assurance processes for Critical capabilities. However, they do not provide further detail on how they will achieve this, or by what point this will need to be intact (i.e. it is unclear if assurance processes must be solidifed before or after a model has been deemed to possess Critical level capabilities). Further, their commitment is only to “update” the framework, meaning their plans or the KCIs/KRIs necessitating Critical level assurance themselves may be modified.

Quotes:

“Models that have reached or are forecasted to reach Critical capability in a Tracked Category present severe dangers and should be treated with extreme caution. Such models require additional safeguards (safety and security controls) during development, regardless of whether or when they are externally deployed. We do not currently possess any models that have Critical levels of capability, and we expect to further update this Preparedness Framework before reaching such a level with any model. Our approach to Critical capabilities will need to be robust to both malicious actors (either internal or external) and model misalignment risks. The SAG retains discretion over when to request deep dive evaluations of models whose scalable evaluations indicate that they may possess or may be nearing critical capability thresholds.” (p. 12)

3.1.3.2 Evidence that the assurance properties are enough to achieve their corresponding KCI thresholds (40%) 50%

Whilst they detail safeguard efficacy assessments which is commendable, these are only “potential”. In addition, their descriptions remain more high-level than precise, and do not cover the Critical case. To improve, they could describe what evidence would be sufficient for showing that the safeguards for misalignment meet the required KCI threshold.

Quotes:

“The table below illustrates possible examples of safeguards and safeguard efficacy assessments we could consider to claim we have sufficiently minimized the risks of severe harm associated with High level of capability under any of our Tracked Categories” (p. 19). The table (Table 5) goes on to define “potential safeguard efficacy assessments” for each category. For instance,
“Reliable and Robust System Oversight:

• Red-teamers can train an arbitrarily adversarial agent or can pretend to be an agent in our setup and we should ensure that harmful behaviors are detected by monitors with a high recall rate
• Monitor escalations are either outright blocked or, in empirical studies, users with the relevant level of expertise were shown to reliably understand and act on monitor flags before significant harm occurred
• SLAs for asynchronous monitors to analyze and detect harmful agent actions posthoc”

More examples can be found in Table 5.

3.1.3.3 The underlying assumptions that are essential for their effective implementation and success are clearly outlined (20%) 10%

The Framework does not explicitly outline the assumptions for assurance processes’ efficacy to hold. However, they demonstrate some implicit awareness that “deception, colluding with oversight models, sabotaging safeguards over time such as by embedding vulnerabilities in safeguards code” could all undermine these KCI measures. To improve, the framework should detail the key technical assumptions necessary for the assurance processes to meet the KCI threshold, and evidence for why these assumptions are justified.

Quotes:

“Research Category: Undermining Safeguards: ability and propensity for the model to act to undermine safeguards placed on it, including e.g., deception, colluding with oversight models, sabotaging safeguards over time such as by embedding vulnerabilities in safeguards code, etc.
Potential response: If a model has High or Critical capabilities in any of the Tracked Categories, require the Safeguards case to be robust to the discovered capability and/or propensity” (p. 7, under ‘Potential response’ to Research Category “Undermining Safeguards” in Table 2.)

3.2.1.1 Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors (30%) 90%

The framework outlines multiple elicitation strategies and commits to fulfill this criterion almost word for word. The elicitation methods detailed show nuance and expertise. To improve, the framework could include measurable information, such as how much compute is used for fine-tuning. More detail could be added on which elicitation methods they anticipate would be used by different threat actors, under realistic settings, to justify their elicitation method.

Quotes:

“Our evaluations are intended to approximate the full capability that the adversary contemplated by our threat model could extract from the deployment candidate model, including by using the highest capability tier of system settings, using a version of the model that has a negligible rate of safety-based refusals on our Tracked Category capability evaluations (which may require a separate model variant), and with the best presently-available scaffolds. These measures are taken to approximate the high end of expected elicitation by threat actors attempting to misuse the model, and should be tailored depending on the level of expected access (e.g., doing finetuning if the weights will be released). Nonetheless, given the continuous progress in model scaffolding and elicitation techniques, we regard any one-time capability elicitation in a frontier model as a lower bound, rather than a ceiling, on capabilities that may emerge in real world use and misuse. We incorporate this uncertainty into our assessments. We monitor the technical landscape for changes to the elicitation techniques and best practices, and reassess our evaluations as needed.” (p. 8)

3.2.1.2 Evaluation frequency (25%) 0%

There is no mention of evaluation frequency in terms of the relative variation of effective computing power.

Quotes:

No relevant quotes found.

3.2.1.3 Description of how post-training enhancements are factored into capability assessments (15%) 25%

There is some recognition of how post-training enhancements can factor into capability assessments, but this description remains high level.

The commitment to “monitor the technical landscape for changes to the elicitation techniques and best practices, and reassess our evaluations as needed” is vague; it is not clear how evaluations are “reassessed” based on changes in best practices. For an improvement, an explicit commitment to adopt best practices should be given, or otherwise forecasting exercises could be completed to justify their assumptions on the rate of progress in post-training enhancements. However, “we incorporate this uncertainty into our assessments”, whilst vague, shows partial implementation of factoring the uncertainty of the progress of post-training enhancements in the future.

Importantly, more detail could be provided on precisely how post-training enhancements are factored into capability assessments – for instance, the size of the “uncertainty” or the safety buffer they give to account for uncertainty concerning the progress of post-training enhancements.

Further, more detail could be added on how they account(ed) for how post-training enhancements’ risk profiles change with different model structures – namely, post-training enhancements are much more scalable with reasoning models, as inference compute can often be scaled to improve capabilities.

Quotes:

“Our evaluations are intended to approximate the full capability that the adversary contemplated by our threat model could extract from the deployment candidate model, including by using the highest capability tier of system settings, using a version of the model that has a negligible rate of safety-based refusals on our Tracked Category capability evaluations (which may require a separate model variant), and with the best presently-available scaffolds. These measures are taken to approximate the high end of expected elicitation by threat actors attempting to misuse the model, and should be tailored depending on the level of expected access (e.g., doing finetuning if the weights will be released). Nonetheless, given the continuous progress in model scaffolding and elicitation techniques, we regard any one-time capability elicitation in a frontier model as a lower bound, rather than a ceiling, on capabilities that may emerge in real world use and misuse. We incorporate this uncertainty into our assessments. We monitor the technical landscape for changes to the elicitation techniques and best practices, and reassess our evaluations as needed.” (p. 8)

3.2.1.4 Vetting of protocols by third parties (15%) 10%

The framework demonstrates discretionary commitment to third-party vetting of evaluation protocols. They do not have a specific structure in place for regularly vetting capabilities assessments by third parties, but they do indicate that they measure the Research Categories capabilities in collaboration with external experts. They also mention a general commitment to soliciting expert opinion on the overall holistic risk assessment process.

Quotes:

“We evaluate whether frontier capabilities create a risk of severe harm through a holistic risk assessment process. This process draws on our own internal research and signals, and where appropriate incorporates feedback from academic researchers, independent domain experts, industry bodies such as the Frontier Model Forum, and the U.S. government and its partners, as well as relevant legal and policy mandates.” (p. 4)

“We call these Research Categories, and in these areas we will take the following
steps, both internally and in collaboration with external experts:

• Further developing the threat models for the area,
• Advancing the science of capability measurement in the area and investing towards the development of rigorous evaluations (which could be achieved internally or via partnerships), and
• Sharing summaries of our findings with the public where feasible.” (pp. 5-6)

Deeper capability assessments: “Deep Dives: designed to provide additional evidence validating the scalable evaluations’ findings on whether a capability threshold has been crossed. These may include a wide range of evidence gathering activities, such as human expert red-teaming, expert consultations, resource-intensive third party evaluations (e.g., bio wet lab studies, assessments by independent third party evaluators), and any other activity requested by SAG.” (p. 8)

3.2.1.5 Replication of evaluations by third parties (15%) 25%

The framework gives some recognition of evaluations being conducted independently by third-parties, but only if deemed necessary. Further, they only commit to “work[ing] with” these parties. They do not explicitly commit in the document to have any evaluations replicated, unless it is part of a deeper capability assessment (‘Deep Dive’) that is asked for by the Safety Advisory Group (i.e., at the OpenAI leadership’s discretion.)

Quotes:

“Third-party evaluation of tracked model capabilities: If we deem that a deployment warrants deeper testing of Tracked Categories of capability (as described in Section 3.1), for example based on results of Capabilities Report presented to them, then when available and feasible, OpenAI will work with third-parties to independently evaluate models.” (p. 13)

Deeper capability assessments: “Deep Dives: designed to provide additional evidence validating the scalable evaluations’ findings on whether a capability threshold has been crossed. These may include a wide range of evidence gathering activities, such as human expert red-teaming, expert consultations, resource-intensive third party evaluations (e.g., bio wet lab studies, assessments by independent third party evaluators), and any other activity requested by SAG.” (p. 8)

3.2.2.1 Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed (40%) 50%

There is some reference to monitoring systems, but without explicit justification that KCI thresholds will not be crossed unnoticed, nor a detailed description of the methodology of monitoring or assurance that monitoring is ongoing.

There is implicit justification that KCI thresholds are sufficient in the safeguards for misaligned models. For instance, a “monitor AI supervises agent actions to detect agent in the case of trying to take harmful actions” and “agent actions are all logged in an uneditable database, and asynchronous monitoring routines review those actions for evidence of harm”. This shows that monitoring is being conducted in order to satisfy the assurance process KCI. However, this monitoring is not specifically for measuring mitigation effectiveness; rather, it is to “review [agent] actions for evidence of harm”, which may not catch KCI thresholds being crossed unnoticed.

Indeed, there are multiple “potential safeguard efficacy assessments” detailed in the framework – e.g., “Redteamer success creating a certain number of accounts without detection (success per resources)”. However, none of these efficacy assessments are ongoing; instead, it appears they only need to be conducted for the drafting of the Safeguards Report. Further, they are only “potential” assessments.

To improve, the framework should describe systematic, ongoing monitoring to ensure mitigation effectiveness is tracked continuously such that the KCI threshold will still be met, when required.

Quotes:

“Reliable and Robust System Oversight:

• Monitor AI supervises agent actions to detect agent in the case of trying to take harmful actions, scheming, or similar
• System forces human approval of agent actions (either by default, or by escalation of the monitor)
• Model’s reasoning is provided faithfully and interpretably to humans to review
• Agent actions are all logged in an uneditable database, and asynchronous monitoring routines review those actions for evidence of harm” (p. 19)

See Tables 4 and 5, pages 18 and 19, “Potential safeguard efficacy assessments”.

3.2.2.2 Vetting of protocols by third parties (30%) 50%

The framework demonstrates discretionary commitment to third-party vetting of KCI protocols. They do not have a specific structure in place for regularly vetting safeguards assessments by third parties, beyond the quote below. They do not explicitly commit therefore to undergo vetting of KCI protocols by third parties, except for containment KCIs.

Quotes:

“Independent expert opinions for evidence produced to SAG: The SAG may opt to get independent expert opinion on the evidence being produced to SAG. The purpose of this input is to add independent analysis from individuals or organizations with deep expertise in domains of relevant risks (e.g., biological risk). If provided, these opinions will form part of the analysis presented to SAG in making its decision on the safety of a deployment. These domain experts may not necessarily be AI experts and their input will form one part of the holistic evidence that SAG reviews.” (p. 13)

“SAG is responsible for assessing whether the safeguards associated with a given deployment sufficiently minimize the risk of severe harm associated with the proposed deployment. The SAG will make this determination based on: […] The associated risks of severe harm, as described in the threat model and where needed, advice of internal or external experts.” (p. 10)

“Continuous Monitoring and Validation: Ensure security threat models and updates inform where security and data privacy controls should be implemented, improved, and monitored to further reduce risk. Internal and external assessments to validate these controls are conducted regularly and reports are provided to OpenAI leadership.” (p. 20)

“Independent Security Audits: Ensure security controls and practices are validated regularly by third-party auditors to ensure compliance with relevant standards and robustness against identified threats.” (p. 21)

“Monitoring and Incident Response: Monitor security and event logs continuously to detect, triage, and respond to security incidents rapidly by 24×7 on-call staff.” (p. 21)

3.2.2.3 Replication of evaluations by third parties (30%) 25%

The framework gives some recognition of evaluations being conducted independently by third-parties, but only if deemed necessary. Further, they only commit to “work[ing] with” these parties. They do not explicitly commit in the document to have any evaluations replicated.

Quotes:

“Third-party stress testing of safeguards: If we deem that a deployment warrants third party stress testing of safeguards and if high quality third-party testing is available, we will work with third parties to evaluate safeguards. We may seek this out in particular for models that are over a High capability threshold.” (p. 13)

3.2.3.1 Sharing of evaluation results with relevant stakeholders as appropriate (85%) 75%

There are commitments to share evaluation results to the public if models are deployed. However, they do not commit to alert any stakeholders when/if Critical capabilities are reached.

Quotes:

“Public disclosures: We will release information about our Preparedness Framework results in order to facilitate public awareness of the state of frontier AI capabilities for major deployments. This published information will include the scope of testing performed, capability evaluations for each Tracked Category, our reasoning for the deployment decision, and any other context about a model’s development or capabilities that was decisive in the decision to deploy. Additionally, if the model is beyond a High threshold, we will include information about safeguards we have implemented to sufficiently minimize the associated risks. Such disclosures about results and safeguards may be redacted or summarized where necessary, such as to protect intellectual property or safety.” (p. 12)

“Transparency in Security Practices: Ensure security findings, remediation efforts, and key metrics from internal and independent audits are periodically shared with internal stakeholders and summarized publicly to demonstrate ongoing commitment and accountability.” (p. 21)

“Internal Transparency. We will document relevant reports made to the SAG and of SAG’s decision and reasoning. Employees may also request and receive a summary of the testing results and SAG recommendation on capability levels and safeguards (subject to certain limits for highly sensitive information).” (p. 12)

3.2.3.2 Commitment to non-interference with findings (15%) 0%

No commitment to permitting the reports, which detail the results of external evaluations (i.e. any KRI or KCI assessments conducted by third parties), to be written independently and without interference or suppression.

Quotes:

No relevant quotes found.

3.2.4.1 Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains (50%) 10%

There is some indication of monitoring; however, this is not explicitly to gain information on novel risk profiles. To improve, such a process should be detailed, for instance by building on the current monitoring infrastructure.

They do mention that monitoring should be conducted to assert there is “no significant evidence of unexpected value alignment failures”, as a safeguard efficacy assessment. Partial credit is given here for the use of “unexpected”, as this could be further developed to analyse novel risk profiles.

Quotes:

“Internal and external user reports, monitoring, or manual inspection for historical model deployments show no significant evidence of unexpected value alignment failures” (p. 19)

“Prevalence of jailbreaks identified via monitoring and reports, in historical deployments” (p. 17)

“Expanding human monitoring and investigation capacity to track capabilities that pose a risk of severe harm, and developing data infrastructure and review tools to enable human investigations” (p. 17)

“Agent actions are all logged in an uneditable database, and asynchronous monitoring routines review those actions for evidence of harm” (p. 19)

3.2.4.2 Mechanism to incorporate novel risks identified post-deployment (50%) 10%

There is a commitment to developing threat models for some of the Research Categories. However, this is not explicitly linked to incorporating novel risks, which were unexpected or not previously anticipated. To improve, an encounter with a possibly novel risk profile of a model should trigger risk modelling exercises, to analyse how this finding may impact all other risk models.

They do mention that if a capability “presents a real risk of severe harm, we may decide to monitor it as a Tracked Category or a Research Category”. Whilst this remains general, partial credit is given here for having some reference to incorporating additional risks – noting that “a capability” could refer to any capability.

Quotes:

“Where we determine that a capability presents a real risk of severe harm, we may decide to monitor it as a Tracked Category or a Research Category.” (p. 4)

“There are also some areas of frontier capability that do not meet the criteria to be Tracked Categories, but where we believe work is required now in order to prepare to effectively address risks of severe harms in the future. These capabilities either need more research and threat modeling before they can be rigorously measured, or do not cause direct risks themselves but may need to be monitored because further advancement in this capability could undermine the safeguards we rely on to mitigate existing Tracked Category risks.” (p. 6)