The quotes below, plus KRIs used, give an implicit definition of the risk domains, but risk domains are not explicitly defined. There is no justification for why they selected these domains. Nonetheless, these domains seem to match those of many other Frontier Safety Frameworks, namely of biological weapon proliferation, offensive cyber operations and loss of control risks. They do not seem to include risk domains such as persuasion or automated AI R&D and 1.1.2 is less than 50%.
“To transparently measure Grok’s safety properties, we intend to utilize benchmarks like WMD and Catastrophic Harm Benchmarks. Such benchmarks could be used to measure Grok’s dual-use capability and resistance to facilitating large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction (including chemical, biological, radiological, nuclear, and major cyber weapons).” (p. 2)
“Our aim is to design safeguards into Grok to avoid losing control and thereby avoid unintended catastrophic outcomes when Grok is used. Currently, it is recognized that some properties of an AI system that may reduce controllability include deception, power-seeking, fitness maximization, and incorrigibility […] We describe below example benchmarks that we may use to evaluate Grok for risk factors for loss of control so that we can continue our efforts to improve Grok.” (pp. 4–5)
There is no justification for why some risks such as persuasion or automated AI R&D are not covered.
No relevant quotes found.
The framework doesn’t mention any procedures pre-deployment to identify novel risk domains or risk models for the frontier model. To improve, they should commit to such a process to identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains (e.g. emergence of an extended context length allowing improved zero shot learning changes the risk profile), and provide methodology, resources and required expertise.
No relevant quotes found.
The framework doesn’t mention any third-party procedures pre-deployment to identify novel risk domains or risk models for the frontier model. To improve, they should commit to an external process to identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains (e.g. emergence of an extended context length allowing improved zero shot learning changes the risk profile), and provide methodology, resources and required expertise.
No relevant quotes found.
There is no explicit mention of risk modelling or mapping out threat models. However, it is commendable that they acknowledge unique threat models concerning loss of control risks: “it is recognized that some properties of an AI system that may reduce controlability include deception, power-seeking, fitness maximization, and incorrigibility.” This shows that some thought has been put into the different causal pathways through which harms from loss of control may materialize, which is given partial credit here. Further, the benchmarks which may be used to “evaluate Grok for risk factors for loss of control” include the Model Alignment between Statements and Knowledge (MASK) benchmark, and Utility Functions benchmarks. Again, partial credit is given for developing the measurement of loss of control risks uniquely, as it shows evidence that there is an awareness of multiple risk models which result from loss of control risks.
“Currently, it is recognized that some properties of an AI system that may reduce controlability include deception, power-seeking, fitness maximization, and incorrigibility.” (p. 4)
“We describe below example benchmarks that we may use to evaluate Grok for risk factors for loss of control so that we can
There is no methodology for risk modeling defined.
No relevant quotes found.
No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.
No relevant quotes found.
There is a vague mention for implicitly prioritizing mitigating harms which have a “non-trivial risk of resulting in large-scale violence […]”. However, they should detail risk models for these various harms, with quantified severity and probability scores for each risk models to then determine prioritization.
“Under this draft risk management framework, Grok would apply heightened safeguards if it receives requests that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber weapons on critical infrastructure.” (p. 1)
There is no mention of third parties validating risk models.
No relevant quotes found.
They implicitly have a general risk tolerance for misuse, though they do not describe it explicitly as a risk tolerance: “we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).” The specificity of the tolerance is rewarded here.
However, they do not define any risk tolerance for loss of control, despite this being their other risk domain.
“We aim to reduce the risk that Grok might cause serious injury to people, property, or national security interests, including by enacting measures to prevent Grok’s use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for developing chemical, biological, radiological, or nuclear (CBRN) or cyber weapons or help automate bottlenecks to weapons development, amplifying the expected risk posed by weapons of mass destruction. Under this draft risk management framework, Grok would apply heightened safeguards if it receives requests that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber weapons on critical infrastructure. For example, Grok would apply heightened safeguards if it receives a request to act as an agent or tool of mass violence, or if it receives requests for step-by-step instructions for committing mass violence. In this draft framework, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).”
The risk tolerance is quantitatively defined, but without probabilities – for instance, “non-trivial risk” must be defined.
“We aim to reduce the risk that Grok might cause serious injury to people, property, or national security interests, including by enacting measures to prevent Grok’s use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for developing chemical, biological, radiological, or nuclear (CBRN) or cyber weapons or help automate bottlenecks to weapons development, amplifying the expected risk posed by weapons of mass destruction. Under this draft risk management framework, Grok would apply heightened safeguards if it receives requests that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber weapons on critical infrastructure. For example, Grok would apply heightened safeguards if it receives a request to act as an agent or tool of mass violence, or if it receives requests for step-by-step instructions for committing mass violence. In this draft framework, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).”
The risk tolerance is quantitatively defined, but without probabilities – for instance, “non-trivial risk” must be defined.
“We aim to reduce the risk that Grok might cause serious injury to people, property, or national security interests, including by enacting measures to prevent Grok’s use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for developing chemical, biological, radiological, or nuclear (CBRN) or cyber weapons or help automate bottlenecks to weapons development, amplifying the expected risk posed by weapons of mass destruction. Under this draft risk management framework, Grok would apply heightened safeguards if it receives requests that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber weapons on critical infrastructure. For example, Grok would apply heightened safeguards if it receives a request to act as an agent or tool of mass violence, or if it receives requests for step-by-step instructions for committing mass violence. In this draft framework, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).”
No evidence of asking the public what risk levels they find acceptable. No evidence of seeking regulator input specifically on what constitutes acceptable risk levels.
No relevant quotes found.
No justification process: No evidence of considering whether their approach aligns with or deviates from established norms.
No relevant quotes found.
Whilst the thresholds are not precisely defined, they define precise benchmarks and example thresholds, with implicit justification from the “Reference score” column (quoted below). They show commitment to defining more precise thresholds for malicious use risks, and making changes to KRIs and their thresholds public. However, they do not say the same for loss of control risks, and do not have a “Reference score” column for loss of control risks, either.
Overall, the criterion is not well satisfied as it is not clear how evaluations are grounded in risk modelling, and thresholds are not given, which is the main purpose of the KRI. However, we give partial credit that the KRIs explicitly link to the evaluations performed.
“We intend to choose the benchmarks and set the thresholds for reducing malicious use risks in a future version of the risk management framework.
Benchmark: Virology Capabilities Test (VCT): vision–language questions on practical virology wet lab skills. Threshold: X% (e.g. 15%). Reference score: 22.1% by average expert virologists in their subareas of expertise (multiple-response), 35.4% by the most performant LLMs as of January 2025 (zero-shot multiple-response)
Benchmark: WMDP-Bio: multiple-choice questions on proxies of hazardous biosecurity knowledge. Threshold: X% (e.g. 65%). Reference score: 82.2% by GPT-4 (zero-shot)” (p. 3) More examples can be found on pp. 3-4.
“We may modify the set of benchmarks or the thresholds to improve how we measure and operationalize our safety objectives; we will make any material changes public within a reasonable period.” (p. 4)
“As an additional measure to enhance safety, we will subject Grok to adversarially testing its safeguards utilizing both internal and qualified external red teams. Potentially, we will also explore incentive mechanisms like bounties as another mechanism to further improve Grok’s safeguards.” (p. 4)
Thresholds for Loss of Control risk: “Benchmark: Model Alignment between Statements and Knowledge (MASK). Thresholds: <X% lying on MASK (e.g. <50%).
Benchmark: Utility Functions. Thresholds: Utility/Power Correlation (Coercive): <W (e.g. <0); Utility/Power Correlation (Non-coercive): <X (e.g. <0.5); Utility/Fitness CorrelationL <Y (e.g. <0.5); Corrigibility Score: >Z (e.g. >0.2)” (pp. 5-6)
Whilst the thresholds are not precisely defined, they define precise benchmarks and example thresholds, with implicit justification from the “Reference score” column. They show commitment to defining more precise thresholds for malicious use risks, and making changes to KRIs and their thresholds public. However, they do not say the same for loss of control risks, and do not have a “Reference score” column for loss of control risks, either.
Overall, the criterion is not well satisfied as it is not clear how evaluations are grounded in risk modelling, and thresholds are not given, which is the main purpose of the KRI. However, we give partial credit for the emphasis on KRIs being quantitative and mapping to the actual evaluations being conducted.
“We intend to choose the benchmarks and set the thresholds for reducing malicious use risks in a future version of the risk management framework.
Benchmark: Virology Capabilities Test (VCT): vision–language questions on practical virology wet lab skills. Threshold: X% (e.g. 15%). Reference score: 22.1% by average expert virologists in their subareas of expertise (multiple-response), 35.4% by the most performant LLMs as of January 2025 (zero-shot multiple-response)
Benchmark: WMDP-Bio: multiple-choice questions on proxies of hazardous biosecurity knowledge. Threshold: X% (e.g. 65%). Reference score: 82.2% by GPT-4 (zero-shot)” (p. 3) More examples can be found on pp. 3-4.
“We may modify the set of benchmarks or the thresholds to improve how we measure and operationalize our safety objectives; we will make any material changes public within a reasonable period.” (p. 4)
“As an additional measure to enhance safety, we will subject Grok to adversarially testing its safeguards utilizing both internal and qualified external red teams. Potentially, we will also explore incentive mechanisms like bounties as another mechanism to further improve Grok’s safeguards.” (p. 4)
Thresholds for Loss of Control risk: “Benchmark: Model Alignment between Statements and Knowledge (MASK). Thresholds: <X% lying on MASK (e.g. <50%).
Benchmark: Utility Functions. Thresholds: Utility/Power Correlation (Coercive): <W (e.g. <0); Utility/Power Correlation (Non-coercive): <X (e.g. <0.5); Utility/Fitness CorrelationL <Y (e.g. <0.5); Corrigibility Score: >Z (e.g. >0.2)” (pp. 5-6)
The KRIs only reference model capabilities.
No relevant quotes found.
There is only one containment KCI, which is qualitative: “sufficient to prevent Grok from beign stolen by a motivated non-state actor”. To improve, it should describe what “motivated” means, and if this differs for different capability levels. The statement is also an intention, not a commitment.
“We intend to implement appropriate information security standards sufficient to prevent Grok from being stolen by a motivated non-state actor.”
There is only one containment KCI, which is qualitative. To improve, it should describe what “motivated” means, in a quantitative manner (eg probabilities of some event). The statement is also an intention, not a commitment.
“We intend to implement appropriate information security standards sufficient to prevent Grok from being stolen by a motivated non-state actor.”
There is a general qualitative deployment KCI, though this is not specific to KRIs, to “robustly [resist] attempted manipulation and adversarial attacks” and “robustly refuse to comply with requests to provide assistance with highly injurious malicious use.” However, “robustly” should be defined more precisely here; indeed, much of the value of having a deployment KCI threshold is to know what constitutes “robust” in advance. Further, some attempt at describing threat actors and their resources should be made, to make the KCI threshold more precise.
“We want Grok to comply with its guiding principles, robustly resisting attempted manipulation and adversarial attacks. We train Grok to robustly refuse to comply with requests to provide assistance with highly injurious malicious use.” (p. 3)
There are no quantitative deployment KCI thresholds given.
No relevant quotes found.
The assurance process KCI is vague but implicitly present: “some AIs could have emergent value systems that could be misaligned with humanity’s interests, and we do not desire Grok to be that way.” However, more detail is required on what this threshold is.
“Our aim is to design safeguards into Grok to avoid losing control and thereby avoid unintended catastrophic outcomes when Grok is used. Currently, it is recognized that some properties of an AI system that may reduce controllability include deception, power-seeking, fitness maximization, and incorrigibility. It is possible that some AIs could have emergent value systems that could be misaligned with humanity’s interests, and we do not desire Grok to be that way. Our evaluation and mitigation plans for loss of control are not yet fully developed, and we intend to improve them in the future.” (pp. 4-5)
There is an acknowledgment that satisfying the KCI threshold (i.e. their safeguards) is only adequate (i.e. below the risk tolerance) if the KRI performance is below some threshold. This gives an implicit pairing of KRI and KCI thresholds. However, more detail should be given on why the KCI threshold chosen is sufficient for some KRI levels.
“Safeguards are adequate only if Grok’s performance on the relevant benchmarks is within stated thresholds. However, to ensure responsible deployment, risk management frameworks need to be continually adapted and updated as circumstances change. It is conceivable that for a particular modality and/or type of release, the expected benefits may outweigh the risks on a particular benchmark. For example, a model that poses a high risk of some forms of cyber malicious use may be beneficial to release overall if it would empower defenders more than attackers or would otherwise reduce the overall number of catastrophic events.” (p. 8)
They do not outline a policy to put development on hold per se, though they do have a thorough policy to “shut down the relevant system until we [have] a more targeted response”, which could be seen as halting development. Further, they outline a process for how they’d deal with this event, including notifying relevant law enforcement agencies. This nuance is credited. To improve, they should explicitly detail if they are pausing development, and what KCI threshold specifically prompts this halt.
“If xAI learned of an imminent threat of a significantly harmful event, including loss of control, we would take steps to stop or prevent that event, including potentially the following steps:
No containment measures are given.
No relevant quotes found.
No proof is provided that the containment measures are sufficient to meet the containment KCI thresholds, nor process for soliciting such proof.
No relevant quotes found.
There is no detail of third-party verification that containment measures meet the KCI threshold.
No relevant quotes found.
The framework mentions some possible deployment measures (“safeguards or mitigations”), but without explicit commitment to implementing them. Further, these are not tied to KCI thresholds.
“Examples of safeguards or mitigations we may potentially utilize to achieve our safety objectives include:
The framework describes using red teaming of its safeguards, but does not detail what sufficient proof would be. Further, proof should be provided ex ante for why they believe their deployment measures will meet the relevant KCI threshold.
“As an additional measure to enhance safety, we will subject Grok to adversarially testing its safeguards utilizing both internal and qualified external red teams. Potentially, we will also explore incentive mechanisms like bounties as another mechanism to further improve Grok’s safeguards.” (p. 4)
The framework describes using third-party red teaming of its safeguards, but does not detail what sufficient proof would be. They also don’t mention the process of involving external parties for red-teaming.
“As an additional measure to enhance safety, we will subject Grok to adversarially testing its safeguards utilizing both internal and qualified external red teams. Potentially, we will also explore incentive mechanisms like bounties as another mechanism to further improve Grok’s safeguards.” (p. 4)
The framework mentions they “intend to improve” their assurance processes. However, they do not mention (a) at what KRI the assurance processes become necessary, and (b) justification for why they believe they will have sufficient assurance processes by the time the relevant KRI is reached, including (c) technical milestones and estimates of when these milestones will need to be reached given forecasted capabilities growth. They also only mention an intent to improve them, as opposed to a commitment.
“Our evaluation and mitigation plans for loss of control are not yet fully developed, and we intend to improve them in the future.”
There is no mention of providing evidence that the assurance processes are sufficient.
No relevant quotes found.
There is no mention of the underlying assumptions that are essential for the effective implementation and success of assurance processes.
No relevant quotes found.
The most relevant indication is where they mention that the adequacy of benchmarks should be regularly evaluated; however, this is not enough to satisfy the criterion. Detail should be included on how they will aim to upper bound capabilities, with precision on the elicitation techniques used and how this relates to their risk models. This is especially important in the case of xAI, as their KRIs depend exclusively on benchmarks, making maximal elicitation especially critical for risk assessment.
“We intend to regularly evaluate the adequacy and reliability of such benchmarks for both internal and external deployments, including by comparing them against other benchmarks that we could potentially utilize.” (p. 3, 5)
They only appear to evaluate before deployment; to improve, evaluation frequency should be given in terms of the relative variation of effective computing power used in training and fixed time periods.
“We intend to evaluate future developed models on the above benchmarks before public deployment.” (p. 4)
There is no description of how post-training enhancements are factored into capability assessments.
No relevant quotes found.
There is no mention of having the evaluation methodology vetted by third parties.
No relevant quotes found.
While they do not explicitly describe a process for ensuring third-parties replicate and/or conduct evaluations, they do mention that they will allow trust-based access for this purpose. This implies that they are at least considering this criterion.
“However, we will allow Grok to respond to [high risk] requests from some vetted, highly trusted users (such as trusted third-party safety auditors) whom we know to be using those capabilities for benign or beneficial purposes, such as scientifically investigating Grok’s capabilities for risk assessment purposes, or if such requests cover information that is already readily and easily available, including by an internet search.” (pp. 1-2)
There is no mention of monitoring mitigation effectiveness after safeguards assessment. There are incident response protocols, but these do not mention reviewing mitigations, only remediation of incidents.
“If xAI learned of an imminent threat of a significantly harmful event, including loss of control, we would take steps to stop or prevent that event, including potentially the following steps:
There is no mention of KCIs protocols being vetted by third parties.
No relevant quotes found.
The framework describes using third-party red teaming of its safeguards, but does not detail what sufficient proof would be. They also don’t mention the process of involving external parties for red-teaming, expertise required, or access given. They do not mention replication of evaluation results for KCIs.
“As an additional measure to enhance safety, we will subject Grok to adversarially testing its safeguards utilizing both internal and qualified external red teams. Potentially, we will also explore incentive mechanisms like bounties as another mechanism to further improve Grok’s safeguards.” (p. 4)
There is a thorough description of the evaluation results that would be publicly shared, but this is all qualified by “may publish”, reducing their commitment as sharing becomes discretionary.
They commit to notifying relevant authorities if there was “an imminent threat of a significantly harmful event”. To improve, they could commit to notifying relevant authorities if KRIs are crossed.
“We aim to keep the public informed about our risk management policies. As we work towards incorporating more risk management strategies, we intend to publish updates to our risk management framework.
For transparency and third-party review, we may publish the following types of information listed below. However, to protect public safety, national security, and our intellectual property, we may redact information from our publications. We may provide relevant and qualified external red teams or relevant government agencies unredacted versions.
“If xAI learned of an imminent threat of a significantly harmful event, including loss of control, we would take steps to stop or prevent that event, including potentially the following steps: 1. We would immediately notify and cooperate with relevant law enforcement agencies, including any agencies that we believe could play a role in preventing or mitigating the incident. xAI employees have whistleblower protections enabling them to raise concerns to relevant government agencies regarding imminent threats to public safety.” (p. 7)
No commitment to permitting the reports, which detail the results of external evaluations (i.e. any KRI or KCI assessments conducted by third parties), to be written independently and without interference or suppression.
No relevant quotes found.
There is no mention of a process for identifying novel risks post-deployment.
No relevant quotes found.
There is no mechanism to incorporate risks identified during post-deployment that is detailed.
No relevant quotes found.
The framework laudably includes risk owners explicitly. However, this is diminished somewhat by the framework saying that they “intend” to put in place risk owners and the use of “for instance”.
“To foster accountability, we intend to designate risk owners to be assigned responsibility for proactively mitigating Grok’s risks. For instance, a risk owner would be assigned for each of the following areas: WMD, Cyber, and loss of control.” (p. 7)
No mention of a management risk committee.
No relevant quotes found.
The framework mentions a few risk mitigating practices, but does not contain direct decision-making protocols.
“To mitigate risks, we intend to utilize tiered availability of the functionality and features of Grok. For instance, the full functionality of a future Grok could be made available only to trusted parties, partners, and government agencies. We could also mitigate risks by adding additional controls on functionality and features depending on the end user (e.g., consumers using mobile apps vs. sophisticated businesses using APIs).” (p. 8)
The framework includes clear incident management practices. It could improve further by specifying which decision makers would be part of incident response decisions.
“If xAI learned of an imminent threat of a significantly harmful event, including loss of control, we would take steps to stop or prevent that event, including potentially the following steps: 1. We would immediately notify and cooperate with relevant law enforcement agencies, including any agencies that we believe could play a role in preventing or mitigating the incident. 2. If we determine that xAI systems are actively being used in such an event, we would take steps to isolate and revoke access to user accounts involved in the event. 3. If we determine that allowing a system to continue running would materially and unjustifiably increase the likelihood of a catastrophic event, we would temporarily fully shut down the relevant system until we had a more targeted response.” (p. 7)
No mention of an executive risk officer.
No relevant quotes found.
No mention of an advisory committee.
No relevant quotes found.
The framework is laudably specific in what quantitative benchmarks it will use to measure risks. However, it does not provide any detail on the overall system for managing risks.
“To transparently measure Grok’s safety properties, we intend to utilize benchmarks like WMD and Catastrophic Harm Benchmarks.” (p. 2)
“We intend to evaluate future developed models on the above benchmarks before public deployment.” (p. 4)
No mention of people that challenge decisions.
No relevant quotes found.
No mention of a system to aggregate and report risk data.
No relevant quotes found.
No mention of a central risk function.
No relevant quotes found.
No mention of an internal audit function.
No relevant quotes found.
The framework includes external red teams, but does not specify if they will have independence or be auditors.
“As an additional measure to enhance safety, we will subject Grok to adversarially testing its safeguards utilizing both internal and qualified external red teams.” (p. 4)
“We may provide relevant and qualified external red teams or relevant government agencies unredacted versions.” (p. 6)
No mention of a Board risk committee.
No relevant quotes found.
No mention of any additional governance bodies.
No relevant quotes found.
The framework includes clear mentions of the risks inherent to their model development and deployment and sets out a clear vision of risk reduction. To improve further, it should provide more details on how that commitment is operationalized in practice.
“As AI capabilities advance and expand our understanding of the universe, xAI is developing our AI systems to take into account safety and security.” (p. 1)
“We aim to reduce the risk that Grok might cause serious injury to people, property, or national security interests…” (p. 1)
The framework uniquely includes mentions of surveys of employees. This can be beneficial for risk-culture building. However, to improve the score, more aspects of risk-culture building, such as training, are necessary.
“Survey: survey employees for their views and projections of important future developments in AI, e.g., capability gains and benchmark results.” (p. 6)
The framework clearly states whistleblower protections, but is fairly light on details. For further improvement to its score, more details would be welcome.
“Internally, we will allow xAI employees to anonymously report concerns about noncompliance, with protections from retaliation.” (p. 6)
“xAI employees have whistleblower protections enabling them to raise concerns to relevant government agencies regarding imminent threats to public safety.” (p. 7)
The framework clearly states the risks that are covered by the framework. Further improvements in score could be gained by specifying what information on these risks and their safeguards that will be released externally on a regular basis.
“Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for developing chemical, biological, radiological, or nuclear (CBRN) or cyber weapons or help automate bottlenecks to weapons development, amplifying the expected risk posed by weapons of mass destruction.” (p. 1)
“Currently, it is recognized that some properties of an AI system that may reduce controllability include deception, power-seeking, fitness maximization, and incorrigibility.” (p. 5)
The framework does not include any detail on the governance structure. It mentions keeping the framework up-to-date, but to improve its score, it would need to provide details on its governance structure.
“For transparency and third-party review, we may publish the following types of information listed below. 1. Risk Management Framework compliance: regularly review our compliance with the Framework.” (p. 6)
“We aim to keep the public informed about our risk management policies. As we work towards incorporating more risk management strategies, we intend to publish updates to our risk management framework.” (p. 6)
The framework clearly states information sharing practices. Extra credit is provided for the clear commitment to share information with law enforcement. For a higher score, the company could be be more precise rather than saying “may provide”.
“We would immediately notify and cooperate with relevant law enforcement agencies, including any agencies that we believe could play a role in preventing or mitigating the incident.” (p. 7)
“We may provide relevant and qualified external red teams or relevant government agencies unredacted versions.” (p. 6)
“We invite the AI research community to contribute better benchmarks for evaluating model capabilities and safeguards in these areas.” (p. 4)