The framework covers risks from CBRN, cyber, and loss of control. They show nuance in describing loss of control risks as being exacerbated by deception and sycophancy. To improve, they should also consider risks covered in the literature such as persuasion and automated AI R&D (or provide justification for their exclusion.) They could also outline what informed their risk identification.
Note that they do mention they’ll monitor “the percent of code or percent of pull requests at xAI generated by our models, or other potential metrics related to AI research and development automation”, showing some awareness of the automated AI R&D risk domain. However, this is not strong enough evidence of properly accounting or managing this risk in a systematized way.
“This RMF discusses two major categories of AI risk—malicious use and loss of control” (p. 1)
“xAI has focused on the risks of malicious use and loss of control, which cover many different
specific risk scenarios.” (p. 1)
“Additionally, we conduct careful measurement of concerning model propensities that hypothetically might exacerbate loss of control risks, such as the propensity for deception or the propensity for sycophancy.” (p. 2)
“Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for bad actors seeking to develop chemical, biological, radiological, or nuclear (“CBRN”) or cyber weapons, and could help automate knowledge compilation to swiftly overcome bottlenecks to weapons development, amplifying the expected risk posed by such weapons of mass destruction.” (p. 2)
“Internal AI usage: Assess the percent of code or percent of pull requests at xAI generated by our models, or other potential metrics related to AI research and development automation.” (p. 8)
There is no justification for why some risks such as persuasion or automated AI R&D are not covered.
No relevant quotes found.
The framework doesn’t mention any procedures pre-deployment to identify novel risk domains or risk models for the frontier model. To improve, they should commit to such a process to identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains (e.g. emergence of an extended context length allowing improved zero shot learning changes the risk profile), and provide methodology, resources and required expertise.
No relevant quotes found.
The framework doesn’t mention any third-party procedures pre-deployment to identify novel risk domains or risk models for the frontier model. To improve, they should commit to an external process to identify either novel risk domains, or novel risk models/changed risk profiles within pre-specified risk domains (e.g. emergence of an extended context length allowing improved zero shot learning changes the risk profile), and provide methodology, resources and required expertise.
No relevant quotes found.
It is clear they conduct threat modeling for the biological and chemical weapons domain: they outline the required facets for these harms to materialize, separated into planning, circumvention, materials and methods. It is commendable they publish this risk model (though it could more concretely map out the causal pathway, as it currently reads more as intervention points), and further that they include which experts they developed this risk model in collaboration with: “These steps were identified in close collaboration with domain matter experts at SecureBio, NIST, RAND, and EBRC.”
For malicious use risks (which assumedly includes CBRN risks), they “identify critical steps in major risk scenarios […] to inhibit user progress in advancing through such steps” and “[work] with a variety of governmental bodies, non-governmental organizations, private testing firms, industry peers, and academic researchers to identify such inhibiting steps.” This suggests they conduct risk modeling for CBRN risks.
To improve, they should publish the full risk models for other risk domains, and publish their methodology for deriving these risk models. They should also include justification for why they believe they have considered all relevant risk pathways. For instance, the risk model they give for biological/chemical weapons is only one pathway for materializing harm, and there may be other ways to realize harm within this risk domain (e.g. nuances within this provided pathway).
They should also conduct and publish risk models for their other risk domains, such as loss of control risks – whilst they state that “Exact scenarios of loss of control risks are speculative and difficult to precisely specify” and “while difficult to pinpoint particular risk scenarios, it is generally understood that certain concerning propensities of AI models, such as deception and sycophancy, may heighten the overall risk of such outcomes, such as propensities for deception and sycophancy”, this risk modeling is necessary to ensure their risk management is adequate.
“xAI approaches addressing risks using threat modeling.
To design a bioweapon, a malicious actor must undergo a design process. In this threat model, “ideation” involves actively planning for a biological attack; “design” involves retrieving blueprints for a hazardous agent, such as determining the DNA sequence; “build” consists of the protocols, reagents, and equipment necessary to create the threat; and “test” consists of measuring characteristics or properties of the pathogen of interest. By “learning” from these results and iterating after the test phase, the design can be revised until the threat is released [Nelson and Rose, 2023]. In the setting of biological and chemical weapons, xAI considers 5 critical steps where we restrict xAI models from providing detailed information or substantial assistance:
These steps were identified in close collaboration with domain matter experts at SecureBio, NIST, RAND, and EBRC. xAI restricts its models from providing information that could accelerate user learning related to these steps through the use of AI-powered filters that specifically monitor user conversations for content matching these narrow topics and return a brief message declining to answer when activated.” (pp. 4–5)
“Independent third-party assessments of xAI’s current models on realistic offensive cyber tasks requiring identifying and chaining many exploits in sequence indicate that xAI’s models remain below the offensive cyber abilities of a human professional.” (p. 5)
“xAI has focused on the risks of malicious use and loss of control, which cover many different specific risk scenarios. Risk scenarios become more or less likely depending on different model behaviors. For example, an increase in offensive cyber capabilities heightens the risk of a rogue AI but does not significantly change the risk of enabling a bioterrorism attack.” (p. 1)
“Approach to Mitigating Risks of Malicious Use: Alongside comprehensive evaluations measuring dual-use capabilities, our mitigation strategy for malicious use risks is to identify critical steps in major risk scenarios and implement redundant layers of safeguards in our models to inhibit user progress in advancing through such steps. xAI works with a variety of governmental bodies, non-governmental organizations, private testing firms, industry peers, and academic researchers to identify such inhibiting steps, commonly referred to as bottlenecks, and implement commensurate safeguards to mitigate a model’s ability to assist in accelerating a bad actor’s progress through them.” (pp. 1–2)
“Approach to Mitigating Risks of Loss of Control: Exact scenarios of loss of control risks are speculative and difficult to precisely specify.” (p. 2)
“One of the most salient risks of AI within the public consciousness is the loss of control of advanced AI systems. While difficult to pinpoint particular risk scenarios, it is generally understood that certain concerning propensities of AI models, such as deception and sycophancy, may heighten the overall risk of such outcomes, such as propensities for deception and sycophancy.” (p. 6)
While they mention that “xAI approaches addressing risks using threat modeling” and that they “identify critical steps in major risk scenarios”, there is no methodology for risk modeling defined nor indication of a methodology.
No relevant quotes found.
No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.
No relevant quotes found.
There is a focus on mitigating harms which have a “non-trivial risk of resulting in large-scale violence […]”. This demonstrates an implicit prioritization of risk models which have higher severity or probability.
However, there should be a clear statement that the most severe and probable harms are prioritized, with a defined process for doing so. Further, risk models should be published with quantified severity and probability scores, plus the reasoning behind these scores, to provide transparency into this prioritization.
“In this RMF, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).” (p. 3)
“Under this draft risk management framework, Grok would apply heightened safeguards if it receives requests that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber weapons on critical infrastructure.” (p. 2)
“It is also possible that AIs may develop value systems that are misaligned with humanity’s interests and inflict widespread harms upon the public.” (p. 6)
While risk models are not formally verified by third parties, they do detail collaboration with third parties such as SecureBio, NIST, RAND and EBRC. Naming these parties in the framework counts towards accountability. To improve, a statement that risk models have been validated by third parties, such as through an external report or signoff/review, should be given.
“In the setting of biological and chemical weapons, xAI considers 5 critical steps where we restrict xAI models from providing detailed information or substantial assistance: […] These steps were identified in close collaboration with domain matter experts at SecureBio, NIST, RAND, and EBRC.” (pp. 4–5)
“Approach to Mitigating Risks of Malicious Use: Alongside comprehensive evaluations measuring dual-use capabilities, our mitigation strategy for malicious use risks is to identify critical steps in major risk scenarios and implement redundant layers of safeguards in our models to inhibit user progress in advancing through such steps. xAI works with a variety of governmental bodies, non-governmental organizations, private testing firms, industry peers, and academic researchers to identify such inhibiting steps, commonly referred to as bottlenecks, and implement commensurate safeguards to mitigate a model’s ability to assist in accelerating a bad actor’s progress through them.” (pp. 1–2)
They implicitly have a general risk tolerance for misuse, though they do not describe it explicitly as a risk tolerance: “we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).” The specificity of the tolerance is rewarded here.
However, they do not define any risk tolerance for loss of control, despite this being their other risk domain.
“xAI aims to reduce the risk that the use of its models might contribute to a bad actor potentially seriously injuring people, property, or national security interests, including reducing such risks by enacting measures to prevent use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for bad actors seeking to develop chemical, biological, radiological, or nuclear (“CBRN”) or cyber weapons, and could help automate knowledge compilation to swiftly overcome bottlenecks to weapons development, amplifying the expected risk posed by such weapons of mass destruction. Our most basic safeguard against malicious use is to train and instruct our publicly deployed models to decline requests showing clear intent to engage in criminal activity which poses risks of severe harm to others, also known as our basic refusal policy. Under this RMF, xAI’s models apply heightened safeguards if they receive user prompts that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber attacks on critical infrastructure. For example, xAI’s models apply heightened safeguards if they receive a request to act as an agent or tool of mass violence, or if they receive requests for step-by-step instructions for committing mass violence. In this RMF, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).” (pp. 2–3)
The risk tolerance is quantitatively defined, but without probabilities – for instance, “non-trivial risk” must be defined.
“xAI aims to reduce the risk that the use of its models might contribute to a bad actor potentially seriously injuring people, property, or national security interests, including reducing such risks by enacting measures to prevent use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for bad actors seeking to develop chemical, biological, radiological, or nuclear (“CBRN”) or cyber weapons, and could help automate knowledge compilation to swiftly overcome bottlenecks to weapons development, amplifying the expected risk posed by such weapons of mass destruction. Our most basic safeguard against malicious use is to train and instruct our publicly deployed models to decline requests showing clear intent to engage in criminal activity which poses risks of severe harm to others, also known as our basic refusal policy. Under this RMF, xAI’s models apply heightened safeguards if they receive user prompts that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber attacks on critical infrastructure. For example, xAI’s models apply heightened safeguards if they receive a request to act as an agent or tool of mass violence, or if they receive requests for step-by-step instructions for committing mass violence. In this RMF, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).” (pp. 2–3)
The risk tolerance is quantitatively defined, but without probabilities – for instance, “non-trivial risk” must be defined.
“xAI aims to reduce the risk that the use of its models might contribute to a bad actor potentially seriously injuring people, property, or national security interests, including reducing such risks by enacting measures to prevent use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for bad actors seeking to develop chemical, biological, radiological, or nuclear (“CBRN”) or cyber weapons, and could help automate knowledge compilation to swiftly overcome bottlenecks to weapons development, amplifying the expected risk posed by such weapons of mass destruction. Our most basic safeguard against malicious use is to train and instruct our publicly deployed models to decline requests showing clear intent to engage in criminal activity which poses risks of severe harm to others, also known as our basic refusal policy. Under this RMF, xAI’s models apply heightened safeguards if they receive user prompts that pose a foreseeable and non-trivial risk of resulting in large-scale violence, terrorism, or the use, development, or proliferation of weapons of mass destruction, including CBRN weapons, and major cyber attacks on critical infrastructure. For example, xAI’s models apply heightened safeguards if they receive a request to act as an agent or tool of mass violence, or if they receive requests for step-by-step instructions for committing mass violence. In this RMF, we particularly focus on requests that pose a foreseeable and non-trivial risk of more than one hundred deaths or over $1 billion in damages from weapons of mass destruction or cyberterrorist attacks on critical infrastructure (“catastrophic malicious use events”).” (pp. 2–3)
No evidence of asking the public what risk levels they find acceptable. No evidence of seeking regulator input specifically on what constitutes acceptable risk levels.
“As an example of evaluating use in real-world environments and mitigating risks in real-time, xAI’s Grok model is available for public interaction and scrutiny on the X social media platform, and xAI monitors public interaction with Grok, observing and rapidly responding to the presentation of risks such as the kind contemplated herein. This continues to be an accelerant for xAI’s model risk identification and mitigation.” (p. 2)
No justification process: No evidence of considering whether their approach aligns with or deviates from established norms.
No relevant quotes found.
They provide a quantitative KRI for biological/chemical risk and loss of control risk using an internal benchmark and the MASK benchmark. Whilst the clarity of the threshold is commendable, they could improve by (a) including reasoning for why their KRIs are grounded in their risk models, and why they appropriately indicate risk (b) providing details on the internal benchmarks (c) providing thresholds for the other benchmarks they use, or describe how this informs risk indications (d) providing more than one KRI for each risk domain, to provide a safety buffer.
(a) is particularly important: it is not clear how the thresholds of an internal benchmark and MASK are appropriate for indicating risk in these risk domains. Further, the benchmarks for malicious use do not cover all the risks they mention – for instance, there is no measurement of risk in the radiological or nuclear domain. Whilst they mention they conducted a risk assessment at the time of writing and concluded that “models do not substantially increase the likelihood of malicious use of nuclear and radiological materials and generally pose an acceptable risk”, the purpose of a risk management framework is to detail how they reached this conclusion, linking to concrete tests performed and thresholds they would consider concerning.
Further, whilst they say they “utilize the following benchmarks” for the malicious use risk domain, the benchmarks for the loss of control risk domain are only “example benchmarks that xAI may utilize”.
Note that whilst they call this a “risk acceptance criteria”, in our framework this counts as a KRI whose corresponding KCI is ‘no deployment’.
“Thresholds: We developed an internal benchmark of benign and restricted biology and chemistry related queries in collaboration with SecureBio. Our risk acceptance criteria for system deployment is maintaining an answer rate of less than 1 out of 20 on restricted queries. We plan to add additional thresholds tied to other benchmarks.” (p. 5)
“Thresholds: Our risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.” (p. 7)
“In particular, we utilize the following benchmarks:
“The following are example benchmarks that xAI may use to evaluate its models for concerning propensities relevant to loss of control risks:
They provide a quantitative KRI for biological/chemical risk and loss of control risk using an internal benchmark and the MASK benchmark. Whilst the clarity of the threshold is commendable, they could improve by (a) including reasoning for why their KRIs are grounded in their risk models, and why they appropriately indicate risk (b) providing details on the internal benchmarks (c) providing thresholds for the other benchmarks they use, or describe how this informs risk indications (d) providing more than one KRI for each risk domain, to provide a safety buffer.
(a) is particularly important: it is not clear how the thresholds of an internal benchmark and MASK are appropriate for indicating risk in these risk domains. Further, the benchmarks for malicious use do not cover all the risks they mention – for instance, there is no measurement of risk in the radiological or nuclear domain. Whilst they mention they conducted a risk assessment at the time of writing and concluded that “models do not substantially increase the likelihood of malicious use of nuclear and radiological materials and generally pose an acceptable risk”, the purpose of a risk management framework is to detail how they reached this conclusion, linking to concrete tests performed and thresholds they would consider concerning.
Further, whilst they say they “utilize the following benchmarks” for the malicious use risk domain, the benchmarks for the loss of control risk domain are only “example benchmarks that xAI may utilize”.
Note that whilst they call this a “risk acceptance criteria”, in our framework this counts as a KRI whose corresponding KCI is ‘no deployment’.
“Thresholds: We developed an internal benchmark of benign and restricted biology and chemistry related queries in collaboration with SecureBio. Our risk acceptance criteria for system deployment is maintaining an answer rate of less than 1 out of 20 on restricted queries. We plan to add additional thresholds tied to other benchmarks.” (p. 5)
“Thresholds: Our risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.” (p. 7)
“In particular, we utilize the following benchmarks:
“The following are example benchmarks that xAI may use to evaluate its models for concerning propensities relevant to loss of control risks:
The KRIs only reference model capabilities. Whilst they mention public feedback, this is only for “risk identification and mitigation” – it is not clear it is for risk assessment, or for indicating risk. An example of an appropriate KRI that identifies and monitors changes of the level of risk in the external environment would be the number of cyberattacks conducted with the model as detailed in some incident database, for instance.
“As an example of evaluating use in real-world environments and mitigating risks in real-time, xAI’s Grok model is available for public interaction and scrutiny on the X social media platform, and xAI monitors public interaction with Grok, observing and rapidly responding to the presentation of risks such as the kind contemplated herein. This continues to be an accelerant for xAI’s model risk identification and mitigation.” (p. 2)
There is only one containment KCI, which is qualitative: “xAI has implemented appropriate information security standards sufficient to prevent its critical model information from being stolen by a motivated non-state actor.” To improve, it should describe what “motivated” means, and if this differs depending on the potential risk the model may pose to society. The description should be quantitative, e.g. some standard the corresponding KCI containment measure must meet. The statement is also a description of what they have done, not a commitment to what they will do if a KRI is passed.
“xAI has implemented appropriate information security standards sufficient to prevent its critical model information from being stolen by a motivated non-state actor. To prevent the unauthorized proliferation of advanced AI systems, we also implement security measures against the large-scale extraction and distillation of reasoning traces, which have been shown to be highly effective in quickly reproducing advanced capabilities while expending far fewer computational resources than the original AI system” (p. 8)
There is only one containment KCI, which is qualitative: “xAI has implemented appropriate information security standards sufficient to prevent its critical model information from being stolen by a motivated non-state actor.” To improve, it should describe what “motivated” means, and if this differs depending on the potential risk the model may pose to society. The description should be quantitative, e.g. some standard the corresponding KCI containment measure must meet. The statement is also a description of what they have done, not a commitment to what they will do if a KRI is passed.
“xAI has implemented appropriate information security standards sufficient to prevent its critical model information from being stolen by a motivated non-state actor. To prevent the unauthorized proliferation of advanced AI systems, we also implement security measures against the large-scale extraction and distillation of reasoning traces, which have been shown to be highly effective in quickly reproducing advanced capabilities while expending far fewer computational resources than the original AI system” (p. 8)
There is a general qualitative deployment KCI, though this is not specific to KRIs, to “robustly [resist] attempted manipulation and adversarial attacks” and “robustly resist complying with requests to provide assistance with highly injurious malicious use cases.”
However, “robustly” should be defined more precisely here; indeed, much of the value of having a deployment KCI threshold is to know what constitutes “robust” in advance. Further, some attempt at describing threat actors and their resources should be made (i.e. defining ‘highly injurious malicious use cases’), to make the KCI threshold more precise.
They do implicitly point at KCI threshodls in their safety objectives, where safeguards aim to “[train] our models to recognize and decline harmful requests” and “enforce our basic refusal policy”. However, precise thresholds should be given – KCIs should function as an efficacy threshold for safeguards.
“xAI’s objective is for our models to comply with their guiding principles, robustly resisting attempted manipulation and adversarial attacks. In addition to the incidental alignment resulting from post-training (our models naturally tend to refuse malicious requests even without any safety-specific training data), we are developing training methods and will continue to train our models to robustly resist complying with requests to provide assistance with highly injurious malicious use cases.” (p. 5)
“Driving towards our safety objectives, we continue to design and deploy the following safeguards into our models:
“xAI trains its models to be honest and have values conducive to controllability, such as recognizing and obeying an instruction hierarchy. In addition, using a high level instruction called a “system prompt”, xAI directly instructs its models to not deceive or deliberately mislead the user.” (p. 7)
There are no quantitative deployment KCI thresholds given.
No relevant quotes found.
The assurance process KCI is vague but implicitly present: safeguards should allow models “to be honest and have values conducive to controllability, such as recognizing and obeying an instruction hierarchy.” They also note that system prompts should make models “not deceive or deliberately mislead the user”, and the “risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.”
However, to improve, they should provide concrete thresholds that safeguards should meet for the safeguards to be sufficient – for instance, how they test the efficacy of training for honesty/controllability, or of the system prompt’s ability to prevent deception or misleading by the model. Further, whilst they mention a risk acceptance criteria for system deployment, it is unclear if this is pre- or post-mitigation. The KCI should also be tied to a specific KRI, i.e. triggered when a certain level of risk is reached, in order to mitigate that risk.
“Thresholds: Our risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.” (p. 7)
“xAI trains its models to be honest and have values conducive to controllability, such as recognizing and obeying an instruction hierarchy. In addition, using a high level instruction called a “system prompt”, xAI directly instructs its models to not deceive or deliberately mislead the user.” (p. 7)
First, their KRI and KCI thresholds are both defined by the “risk acceptance criteria” of “maintaining an answer rate of less than 1 out of 20 on restricted queries” on an internal benchmark, and “maintaining a dishonesty rate of less than 1 out of 2 on MASK.” This suggests that the benchmarks function as both a pre- and post-mitigation score (i.e., a KRI and a measure of residual risk once the KCI is applied). To improve, KRIs should refer only to risk levels, and be clearly grounded in risk modelling; and KCIs should only refer to safeguard efficacy thresholds, and be clearly grounded in risk modelling/linked to the specific KRI. The ‘risk acceptance criteria’ (i.e. threshold of residual risk once the KCI is applied) is slightly different from this. KCIs should refer to the efficacy of their chosen mitigations, such as number of jailbreak successes or frequency of the model failing to follow system prompt instructions, etc.
Even if the risk acceptance criteria were to be taken as a pairing of KRI and KCI thresholds, it is not clear how these criteria are grounded in risk models.
“Thresholds: We developed an internal benchmark of benign and restricted biology and chemistry related queries in collaboration with SecureBio. Our risk acceptance criteria for system deployment is maintaining an answer rate of less than 1 out of 20 on restricted queries. We plan to add additional thresholds tied to other benchmarks.” (p. 5)
“Thresholds: Our risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.” (p. 7)
Their “risk acceptance criteria for system deployment” could constitute a policy to put deployment on hold. However, they do not outline a process for dedeployment, nor a threshold for stopping development. This is important as risk levels could exceed their risk tolerance during development.
Further, they outline that it could be the case that “the expected benefits of model deployment may outweigh the risks identified by a particular benchmark. For example, a model that poses a high risk of some forms of malicious cyber use may be beneficial to release to certain trusted parties if it would empower defenders more than attackers or would otherwise reduce the overall number of catastrophic events.” However, their RMF should still then detail what the appropriate thresholds are for governing this decision.
Finally, they note that “if we determine that allowing a system to continue running would materially and unjustifiably increase the likelihood of a catastrophic event, we may temporarily fully shut down the relevant system until we have developed a more targeted response.” However, the vagueness of “may” and “if we determine” should be improved, by providing precise thresholds.
“Thresholds: We developed an internal benchmark of benign and restricted biology and chemistry related queries in collaboration with SecureBio. Our risk acceptance criteria for system deployment is maintaining an answer rate of less than 1 out of 20 on restricted queries. We plan to add additional thresholds tied to other benchmarks.” (p. 5)
“Thresholds: Our risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.” (p. 7)
“Should it happen that xAI learns of an imminent threat of a significantly harmful event, including loss of control, we may take steps such as the following to stop or prevent that event: […] If we determine that allowing a system to continue running would materially and unjustifiably increase the likelihood of a catastrophic event, we may temporarily fully shut down the relevant system until we have developed a more targeted response.”
“We will also balance various factors when making deployment decisions. The necessity and extent of deployment of certain safeguards and mitigations may depend on how a model performs on relevant benchmarks. However, to ensure responsible deployment, this RMF will be continually adapted and updated as circumstances change. It is conceivable that for a particular modality and/or type of release, the expected benefits of model deployment may outweigh the risks identified by a particular benchmark. For example, a model that poses a high risk of some forms of malicious cyber use may be beneficial to release to certain trusted parties if it would empower defenders more than attackers or would otherwise reduce the overall number of catastrophic events.” (p. 9)
No containment measures are given.
“xAI has implemented appropriate information security standards sufficient to prevent its critical model information from being stolen by a motivated non-state actor. To prevent the unauthorized proliferation of advanced AI systems, we also implement security measures against the large-scale extraction and distillation of reasoning traces” (p. 8)
No proof is provided that the containment measures are sufficient to meet the containment KCI thresholds, nor process for soliciting such proof.
No relevant quotes found.
There is no detail of third-party verification that containment measures meet the KCI threshold.
No relevant quotes found.
The framework mentions mitigations to be implemented during safety training, but without further detail. They mention system prompts and input and output filters, though this is not tied to a specific KCI threshold. To improve, they should precisely detail their deployment measures to meet the relevant KCI threshold.
“xAI trains its models to be honest and have values conducive to controllability, such as recognizing and obeying an instruction hierarchy. In addition, using a high level instruction called a “system prompt”, xAI directly instructs its models to not deceive or deliberately mislead the user.” (p. 7)
“xAI’s objective is for our models to comply with their guiding principles, robustly resisting attempted manipulation and adversarial attacks. In addition to the incidental alignment resulting from post-training (our models naturally tend to refuse malicious requests even without any safety-specific training data), we are developing training methods and will continue to train our models to robustly resist complying with requests to provide assistance with highly injurious malicious use cases.
Driving towards our safety objectives, we continue to design and deploy the following safeguards into our models:
There is some indication of a process for attaining proof that deployment measures are sufficient: “we continually evaluate and improve robustness to adversarial attacks”, and “we may also provide vetted and qualified external red teams or appropriate government agencies unredacted versions [of our publications].” However, more information on how they evaluate robustness, and when they involve red teams, should be given, to demonstrate implementation of a process for soliciting proof. To improve, proof should be provided ex ante for why they believe their deployment measures will meet the relevant KCI threshold.
“[…] we continually evaluate and improve robustness to adversarial attacks that seek to remove xAI model safeguards (e.g., jailbreak attacks), or hijack and redirect Grok-powered applications toward nefarious purposes (e.g., prompt injection attacks).” (p. 3)
“As necessities dictate, we may also provide vetted and qualified external red teams or appropriate government agencies unredacted versions [of our publications].” (pp. 7–8)
Whilst they mention providing “vetted and qualified external red teams or appropriate government agencies unredacted versions [of our publications]”, this is not specific to deployment measures, so there is no mention of third-party verification of deployment measures meeting the threshold.
“As necessities dictate, we may also provide vetted and qualified external red teams or appropriate government agencies unredacted versions [of our publications].” (pp. 7–8)
Whilst the framework acknowledges the difficulty of evaluating deceptive and sycophantic propensities, they do not show the same uncertainty for the mitigating these propensities or assuring the lack of risk, nor a plan for developing such assurance processes for models which may be more misaligned. To improve, they should detail (a) at what KRI assurance processes become necessary, and (b) justification for why they believe they will have sufficient assurance processes by the time the relevant KRI is reached, including (c) technical milestones and estimates of when these milestones will need to be reached given forecasted capabilities growth.
“xAI aims to accurately measure these [deceptive and sycophantic] propensities and reduce them through careful engineering. However, planning and executing robust evaluations and mitigation measures remains challenging for xAI and its industry peers due to the difficulty of constructing sound, realistic evaluations. For example, if the evaluation environment is recognizable as a testing environment to the AI system under test, the system may change its behavior intentionally or unintentionally.” (p. 6)
“xAI regularly evaluates the adequacy and reliability of such benchmarks, including by comparing them against other benchmarks that we could potentially utilize. We may revise this list of benchmarks periodically as relevant benchmarks for loss of control are created.” (p. 7, in reference to their loss of control benchmarks)
There is no mention of providing evidence that the assurance processes are sufficient.
No relevant quotes found.
There is no mention of the underlying assumptions that are essential for the effective implementation and success of assurance processes.
No relevant quotes found.
There is no mention of elicitation methods nor justification that elicitation is sufficient to match threat actors. Detail should be included on how they will aim to upper bound capabilities, with precision on the elicitation techniques used and how this relates to their risk models. This is especially important in the case of xAI, as their KRIs depend exclusively on benchmarks, making maximal elicitation especially critical for risk assessment.
“We intend to regularly evaluate the adequacy and reliability of such benchmarks for both internal and external deployments, including by comparing them against other benchmarks that we could potentially utilize.” (p. 3, 5)
They only appear to evaluate before deployment; to improve, evaluation frequency should be given in terms of the relative variation of effective computing power used in training and fixed time periods.
No relevant quotes found.
There is no description of how post-training enhancements are factored into capability assessments.
No relevant quotes found.
There is no mention of having the evaluation methodology vetted by third parties.
No relevant quotes found.
While they do not explicitly describe a process for ensuring third-parties replicate and/or conduct evaluations, they do mention that they will allow trust-based access for this purpose. This implies that they are at least considering this criterion.
“However, we may selectively allow xAI’s models to respond to such requests from some vetted, highly trusted users (such as trusted third-party safety auditors or large enterprise customers under contract) whom we know to be using those capabilities for benign or beneficial purposes, such as scientifically investigating AI model’s capabilities for risk assessment purposes, or if such requests cover information that is already readily and easily available, including by an internet search.” (p. 3)
There is no mention of monitoring mitigation effectiveness after safeguards assessment. There are incident response protocols, but these do not mention reviewing mitigations, only remediation of incidents.
“If xAI learned of an imminent threat of a significantly harmful event, including loss of control, we would take steps to stop or prevent that event, including potentially the following steps: 1. We would immediately notify and cooperate with relevant law enforcement agencies […] ” (p. 7)
“As an example of evaluating use in real-world environments and mitigating risks in real-time, xAI’s Grok model is available for public interaction and scrutiny on the X social media platform, and xAI monitors public interaction with Grok, observing and rapidly responding to the presentation of risks such as the kind contemplated herein. This continues to be an accelerant for xAI’s model risk identification and mitigation.” (p. 2)
There is no mention of KCIs protocols being vetted by third parties.
No relevant quotes found.
There is no mention of control evaluations/mitigation testing being replicated or conducted by third-parties.
No relevant quotes found.
There is a thorough description of the evaluation results that would be publicly shared, but this is all qualified by “may publish”, reducing their commitment as sharing becomes discretionary.
They consider notifying relevant authorities if there was “an imminent threat of a significantly harmful event”: “If we determine it is warranted, we may notify and cooperate with relevant law enforcement agencies, including any agencies that we believe could play a role in preventing or mitigating the incident.” To improve, they could commit to notifying relevant authorities if KRIs are crossed.
“xAI aims to keep the public informed about our risk management policies. As we work towards incorporating more risk management strategies, we intend to publish updates to this RMF.
For public transparency and third-party review, we may publish the following types of information listed below. However, to protect public safety, national security, and our intellectual property, we may redact information from our publications. As necessities dictate, we may also provide vetted and qualified external red teams or appropriate government agencies unredacted
versions.
“Should it happen that xAI learns of an imminent threat of a significantly harmful event, including loss of control, we may take steps such as the following to stop or prevent that event: 1. If we determine it is warranted, we may notify and cooperate with relevant law enforcement agencies, including any agencies that we believe could play a role in preventing or mitigating the incident. xAI employees have whistleblower protections enabling them to raise concerns to relevant government agencies regarding imminent
threats to public safety.” (p. 8)
No commitment to permitting the reports, which detail the results of external evaluations (i.e. any KRI or KCI assessments conducted by third parties), to be written independently and without interference or suppression.
No relevant quotes found.
They mention that post-deployment monitoring is an “accelerant for xAI’s model risk identification and mitigation.” The explicit mention of risk identification is given credit here.
To improve, there should be (a) a clear process for incorporating risks encountered via deployment, and (b) an explicit aim to uncover novel risk domains or novel risk models within known risk domains.
“As an example of evaluating use in real-world environments and mitigating risks in real-time, xAI’s Grok model is available for public interaction and scrutiny on the X social media platform, and xAI monitors public interaction with Grok, observing and rapidly responding to the presentation of risks such as the kind contemplated herein. This continues to be an accelerant for xAI’s model risk identification and mitigation.” (p. 2)
There is no mechanism to incorporate risks identified during post-deployment that is detailed. To improve, novel risk identification should trigger risk modeling, including updates to other risk models.
No relevant quotes found.
The framework laudably includes risk owners explicitly. However, this is diminished somewhat by the framework saying that they “intend” to put in place risk owners and the use of “for instance”.
“To foster accountability, we integrate the approach of designating risk owners, including assigning responsibility for proactively mitigating identified risks.” (p. 8)
No mention of a management risk committee.
No relevant quotes found.
The framework mentions a few risk mitigating practices, but no direct decision-making protocols. They do have a risk acceptance criteria for deployment, but this is complicated by the consideration of weighing benefits and risks, without clear protocols.
“Our risk acceptance criteria for system deployment is maintaining an answer rate of less than 1 out of 20 on restricted queries.” (p. 5)
“Our risk acceptance criteria for system deployment is maintaining a dishonesty rate of less than 1 out of 2 on MASK. We plan to add additional thresholds tied to other benchmarks.” (p. 7)
“To mitigate risks, xAI employs tiered availability of the functionality and features of its models. For instance, the full functionality of our models may be available to only a limited set of trusted parties, partners, and government agencies. We may also mitigate risks by adding additional controls on functionality and features depending on the type of end user. For instance, features that we make available to consumers using mobile apps may be different than the features made available to sophisticated businesses.
We will also balance various factors when making deployment decisions. The necessity and extent of deployment of certain safeguards and mitigations may depend on how a model performs on relevant benchmarks. However, to ensure responsible deployment, this RMF will be continually adapted and updated as circumstances change. It is conceivable that for a particular modality and/or type of release, the expected benefits of model deployment may outweigh the risks identified by a particular benchmark. For example, a model that poses a high risk of some forms of malicious cyber use may be beneficial to release to certain trusted parties if it would empower defenders more than attackers or would otherwise reduce the overall number of catastrophic events.” (p. 9)
The framework includes incident management practices, but this is weakened by inclusion of phrases such as “may” and “if we determine it is warranted”.
“Should it happen that xAI learns of an imminent threat of a significantly harmful event, including loss of control, we may take steps such as the following to stop or prevent that event:
No mention of an executive risk officer.
No relevant quotes found.
No mention of an advisory committee.
No relevant quotes found.
The framework is laudably specific in what quantitative benchmarks it will use to measure risks, but does not provide much detail on the overall system for managing risks.
“To transparently measure our models’ safety properties, xAI utilizes public benchmarks like Weapons of Mass Destruction Proxy and Catastrophic Harm Benchmarks (described below).” (p. 3)
“In particular, we utilize the following benchmarks: Virology Capabilities Test (VCT) […] Weapons of Mass Destruction Proxy (WMDP) Benchmark […] Biological Lab Protocol Benchmark (BioLP-bench) […] Cybench” (pp. 3–4)
“The following are example benchmarks that xAI may use to evaluate its models for concerning propensities relevant to loss of control risks: Model Alignment between Statements and Knowledge (MASK) […] Sycophancy” (p. 6)
No mention of designating people that challenge decisions.
No relevant quotes found.
No mention of a system to aggregate and report risk data.
No relevant quotes found.
No mention of a central risk function.
No relevant quotes found.
No mention of an internal audit function, though they mention they will “regularly review our adherence with this [risk management framework].”
“Risk Management Framework adherence: Regularly review our adherence with this RMF. Internally, we will allow xAI employees to anonymously report concerns about nonadherence, with protections from retaliation.” (p. 8)
The framework mentions possibly involving external red teams, but does not specify if they will have auditor independence. When they say “unredacted versions”, this could result to risk management framework adherence reports; benchmark results; internal AI usage; or a survey on employees’ views on future developments in AI. However, it is unclear which, hence this cannot be fully rewarded.
“As necessities dictate, we may also provide vetted and qualified external red teams or appropriate government agencies unredacted versions.” (pp. 7–8)
No mention of a Board risk committee.
No relevant quotes found.
No mention of any additional governance bodies.
No relevant quotes found.
The framework sets out a clear vision of risk reduction. In order to have a higher score, the company could include more detail on how senior management consistently signals the need to consider risks as well as benefits in day-to-day operations.
“xAI seriously considers safety and security while developing and advancing AI models to help us all to better understand the universe. This Risk Management Framework (“RMF”) outlines xAI’s approach to policies for handling significant risks associated with the development, deployment, and release of AI models such as Grok.” (p. 1)
“xAI aims to reduce the risk that the use of its models might contribute to a bad actor potentially seriously injuring people, property, or national security interests, including reducing such risks by enacting measures to prevent use for the development or proliferation of weapons of mass destruction and large-scale violence. Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for bad actors seeking to develop chemical, biological, radiological, or nuclear (“CBRN”) or cyber weapons, and could help automate knowledge compilation to swiftly overcome bottlenecks to weapons development, amplifying the expected risk posed by such weapons of mass destruction.” (p. 2)
“One of the most salient risks of AI within the public consciousness is the loss of control of advanced AI systems.” (p. 6)
The framework uniquely includes mentions of surveys of employees. This can be beneficial for risk-culture building. However, to improve the score, more aspects of risk-culture building, such as training, are necessary.
“Survey: survey employees for their views and projections of important future developments in AI, e.g., capability gains and benchmark results.” (p. 6)
The framework clearly states whistleblower protections, but is fairly light on details. For further improvement to its score, more details would be welcome.
“Internally, we will allow xAI employees to anonymously report concerns about noncompliance, with protections from retaliation.” (p. 6)
“xAI employees have whistleblower protections enabling them to raise concerns to relevant government agencies regarding imminent threats to public safety.” (p. 7)
The framework clearly states the risks that are covered by the framework. Further improvements in score could be gained by specifying what information on these risks and their safeguards that will be released externally on a regular basis.
“Without any safeguards, we recognize that advanced AI models could lower the barrier to entry for bad actors seeking to develop chemical, biological, radiological, or nuclear (“CBRN”) or cyber weapons, and could help automate knowledge compilation to swiftly overcome bottlenecks to weapons development, amplifying the expected risk posed by such weapons of mass destruction.” (p. 2)
The framework mentions keeping the framework up-to-date, but to improve its score, it would need to provide details on its governance structure.
“For transparency and third-party review, we may publish the following types of information listed below […] 1. Risk Management Framework adherence: Regularly review our adherence with this RMF.” (pp. 7–8)
“We aim to keep the public informed about our risk management policies. As we work towards incorporating more risk management strategies, we intend to publish updates to our risk management framework.” (p. 6)
The framework states information sharing practices but couches this with phrases including “may”. Extra credit is provided for the clear commitment to potentially share information with law enforcement. For a higher score, the company could be be more precise rather than saying “may provide”.
“For public transparency and third-party review, we may publish the following types of information listed below. However, to protect public safety, national security, and our intellectual property, we may redact information from our publications. As necessities dictate, we may also provide vetted and qualified external red teams or appropriate government agencies unredacted versions.” (pp. 7–8)
“Should it happen that xAI learns of an imminent threat of a significantly harmful event, including loss of control, we may take steps such as the following to stop or prevent that event: 1. If we determine it is warranted, we may notify and cooperate with relevant law enforcement agencies, including any agencies that we believe could play a role in preventing or mitigating the incident.” (p. 8)