Cohere

1.3.2.1 Methodology precisely defined (70%) 0%

There is no methodology for risk modeling defined.

Quotes:

No relevant quotes found.

1.3.2.2 Mechanism to incorporate red teaming findings (15%) 0%

No mention of risks identified during open-ended red teaming or evaluations triggering further risk modeling.

Quotes:

No relevant quotes found.

1.3.2.3 Prioritization of severe and probable risks (15%) 75%

There is a clear assessment and subsequent prioritization of risk models representing the most severe and probable harms. This appears to be from the full space of risk models. However, more detail on the scores given for likelihood and severity of different risk models should be published.

Quotes:

“We identify risks by first assessing potential risks arising from our models’ capabilities and the systems in which they may be deployed. We then assess the likelihood and severity of potential harms that may arise in enterprise contexts from the identified risks.” (p. 5)

“We therefore focus our secure AI work on risks that have a high likelihood of occurring based on the types of tasks LLMs are highly performant in, as well as the limitations inherent in how these models function. This is what we refer to as “model capabilities.” (p. 5)

Use case, likelihood of harm in context, severity of harm in context. For instance, “Insecure Code. Code generation for enterprise developers managing a company’s proprietary data within on-premises servers. Medium to High possibility of a vulnerability being introduced into company code. Medium to High [severity of harm in context], depending on the nature
of the vulnerability introduced and the type of data handled by the company. Severe vulnerabilities can leave companies vulnerable to cyber attacks affecting individuals and society.”

2.1.1.1 Risk tolerance is at least qualitatively defined for all risks (33%) 10%

Their risk tolerance for when the residual risk is “acceptable” is if there are “no significant regressions [demonstrated in evaluations and tests] compared to our previously launched model versions.” Risk tolerances are also allowed to differ based on the customer: “analysis of whether a model is “acceptable” from a risk management perspective must be adapted to the customer context”.

However, this risk tolerance is still vague, and allows Cohere to have plenty of discretion. To improve, they should predefine a risk tolerance that applies to all models, expressed in terms of probability of some severity.

Quotes:

“We consider models safe and secure to launch when our evaluations and tests demonstrate no significant regressions compared to our previously launched model versions, so that performance and security is maintained or improved for every new significant model version. This is Cohere’s bright line for determining when a model is “acceptable” from a risk management perspective and ready to be launched.” (p. 16)

“In this way, the analysis of whether a model is “acceptable” from a risk management perspective must be adapted to the customer context, and must be able to adapt to new requirements or needs that emerge post-deployment. Assurance here means working with our customers to ensure that our models and systems conform to their risk management obligations and standards.” (p. 17)

2.1.1.2 Risk tolerance is expressed at least partly quantitatively as a combination of scenarios (qualitative) and probabilities (quantitative) for all risks (33%) 0%

The risk tolerance, implicit or otherwise, is not expressed fully or partly quantitatively. To improve, the risk tolerance should be expressed fully quantitatively or as a combination of scenarios with probabilities.

Quotes:

No relevant quotes found.

2.1.1.3 Risk tolerance is expressed fully quantitatively as a product of severity (quantitative) and probability (quantitative) for all risks (33%) 0%

The risk tolerance, implicit or otherwise, is not expressed fully or partly quantitatively. To improve, the risk tolerance should be expressed fully quantitatively or as a combination of scenarios with probabilities.

Quotes:

No relevant quotes found.

2.1.2.1 AI developers engage in public consultations or seek guidance from regulators where available (50%) 0%

No evidence of engaging in public consultations or seeking guidance from regulators for risk tolerance.

Quotes:

No relevant quotes found.

2.1.2.2 Any significant deviations from risk tolerance norms established in other industries is justified and documented (e.g., cost-benefit analyses) (50%) 0%

No justification process: No evidence of considering whether their approach aligns with or deviates from established norms.

Quotes:

No relevant quotes found.

2.2.1.1 KRI thresholds are at least qualitatively defined for all risks (45%) 10%

There are implicit KRI assessments which are conducted, but the KRI thresholds are not given. To improve, thresholds which would trigger mitigations should be developed. The KRIs should also be grounded in risk modeling.

Quotes:

Key risks:
“Data acquisition and preparation stage:

• Data poisoning
• Supply chain vulnerabilities
• Model theft
• Insecure plugin design
• Unrepresentative data
  distributions
• Imbalance of data with harmful patterns and attributes vs. positive patterns and attributes
• Historically outdated representations in data
• Inaccurate proxies when used to measure representativeness or imbalances” (p. 12)

“Training, evaluations and testing.

• Data poisoning
• Data leakage
• Model theft
• Adversarial attacks
• Evaluation criteria and data are not representative of a population
• Disparate performance in different cases results in disproportionate impact on certain populations
• Models and data are fit for an aggregated, dominant population but sub-optimal for sub-groups within the population” (p. 13)

“Deployment and maintenance.

• Prompt injection
• Insecure output handling
• Model denial of service
• Excessive agency
• Sensitive information
  disclosure
• Misuse
• Unexpected post-deployment usage patterns that were not accounted for and result in unmitigated risk” (p. 13)

“Improvement and further fine-tuning.

• Prompt injection
• Insecure input/output handling
• Model denial of service
• Excessive agency
• Sensitive information
  disclosure
• Adversarial attacks
• Evaluation criteria and data are not representative of a population
• Model design choices amplify performance disparity across different examples in the data” (p. 14)

“Multi-faceted evaluations, including standard benchmarks and proprietary evaluations based on identified possible harms and harm reduction objectives” (p. 13)

2.2.1.2 KRI thresholds are quantitatively defined for all risks (45%) 0%

There is no evidence of KRI thresholds being quantitatively defined.

Quotes:

No relevant quotes found.

2.2.1.3 KRIs also identify and monitor changes in the level of risk in the external environment (10%) 10%

“Unexpected post-deployment usage patterns that were not accounted for and result in unmitigated risk” are described as a key risk to track during the deployment and maintenance stage. However, a threshold which triggers mitigations should be defined.

Quotes:

Key Risks: “Unexpected post-deployment usage patterns that were not accounted for and result in unmitigated risk” (p. 13)

2.2.2.1 Containment KCIs (35%) 13%

2.2.2.2 Deployment KCIs (35%) 5%

2.2.2.3 For advanced KRIs, assurance process KCIs are defined (30%) 0%

There are no assurance processes KCIs defined. The framework does not provide recognition of there being KCIs outside of containment and deployment measures.

Quotes:

No relevant quotes found.

3.1.1.1 Containment measures are precisely defined for all KCI thresholds (60%) 25%

While containment measures are defined, most remain high-level (e.g., “secure, risk- based defaults and internal reviews”, or “Supply chain controls for any third parties (e.g., data vendors or third-party data annotation)”, or “Blocklists”) More detail on the measures actually implemented or planned to be implemented is needed to improve. They should also be linked to specific KCI (and thus KRI) thresholds.

Quotes:

“These controls include:

• Advanced perimeter security controls and real-time threat prevention and monitoring
  
• Secure, risk-based defaults and internal reviews
  
• Advanced endpoint detection and response across our cloud infrastructure and distributed devices
  
• Strict access controls, including multifactor authentication, role-based access control, and just-in-time access, across and within our environment to protect against insider and external threats (internal access to unreleased model weights is even more strenuously restricted)
  
• “Secure Product Lifecycle” controls, including security requirements gathering, security risk assessment, security architecture and product reviews, security threat modeling, security scanning, code reviews, penetration testing, and bug bounty programs” (p. 9)

Key Mitigations We Apply:
“Data acquisition and preparation.

• Detailed data lineage controls, including tracking the source, pre-processing steps, storage location, and access permissions
• Supply chain controls for any third parties (e.g., data vendors or third-party data annotation)
• Traditional just-in-time access controls, robust authentication, zero-trust rules, etc.
• Data pre-processing (including cleaning, analysis, selection, etc.)
• Re-sampling, re-weighting, and re-balancing datasets to reduce identified representation issues or
  imbalances” (p. 12)

“Training, evaluations and testing.

• Multi-disciplinary red teaming
• Independent third-party security testing, e.g., penetration testing
• Continuous monitoring to detect anomalies and security issues
• Multi-disciplinary red teaming
• Consultation of domain experts
• Multi-faceted evaluations, including standard benchmarks and proprietary evaluations based on identifi ed possible harms and harm reduction objectives
• User research of local language
  and cultural contexts” (p. 13)

“Deployment and maintenance.

• Blocklists, custom classifiers, and prompt injection guard filters, and human review to detect and intercept attempts to create unsafe outputs
• Specific mitigations applied based on deployment type, e.g., isolated customer environments with focus on remediating security vulnerabilities that coexist
  between traditional application security and AI security
• Security Information and Event Management (SIEM) system leveraging heuristics and advanced detection capabilities to identify potential threats
• “Air-gapped”” safeguards to prevent lateral movement and unintended network calls across environments and kernel-based LLMs to prevent the leaking of shared memories or buffers that could expose sensitive data
• Blocklists
• Safety classifiers and human review to detect and intercept attempts to create unsafe outputs
• Human-interpretable explanation of outputs
• User research and customer feedback analysis” (p. 13)

“Improvement and further fine-tuning.

• Responsible Disclosure Policy to incent third-party security vulnerability discovery
• Specific mitigations applied based on deployment type, e.g., isolated customer environments with focus on remediating security vulnerabilities that coexist between traditional application security and AI security
• Continuous evaluation and user research
• Programs to incentivize research, including research grants and participation in external independent research efforts.
• Multi-disciplinary red teaming” (p. 14)

3.1.1.2 Proof that containment measures are sufficient to meet the thresholds (40%) 10%

Whilst there is a process for determining weaknesses in containment measures with internal API testing, it is not clear that this is prior to their implementation, and this does not cover other aspects of containment, such as securing model weights. Further, to improve, they should detail proof for why they believe the containment measures proposed will be sufficient to meet the KCI threshold, in advance.

Quotes:

“Where applicable, we also consider risks within the context of customer deployments. For example, because many of our users start building applications through our application programming interfaces (APIs) before moving to more advanced deployments, we extensively test and secure our APIs. Our API V2 underwent a heavy security design review before we made it available.” (p. 10)

3.1.1.3 Strong third party verification process to verify that the containment measures meet the threshold (100% if 3.1.1.3 > [60% x 3.1.1.1 + 40% x 3.1.1.2]) 10%

Whilst there is a process for determining weaknesses in containment measures, it is not clear that this is prior to their implementation. To improve, they should detail a process for third-parties to verify the case for why they believe the containment measures proposed will be sufficient to meet the KCI threshold, in advance.

Quotes:

“Prior to deployment, significant model releases undergo an independent third-party penetration test to validate the security of containers and models.” (p. 10)

“Independent third-party security
testing, e.g., penetration testing” (p. 13)

3.1.2.1 Deployment measures are precisely defined for all KCI thresholds (60%) 25%

While deployment measures are defined, most if not all remain high-level (e.g., “human-interpretable explanation of outputs”, or “multi-disciplinary red teaming”.) To improve, more detail on the measures actually implemented or planned to be implemented should be given. Further, the measures should be tied to specific KCI thresholds.

Quotes:

Key Mitigations We Apply:
“Data acquisition and preparation.

• Detailed data lineage controls, including tracking the source, pre-processing steps, storage location, and access permissions
• Supply chain controls for any third parties (e.g., data vendors or third-party data annotation)
• Traditional just-in-time access controls, robust authentication, zero-trust rules, etc.
• Data pre-processing (including cleaning, analysis, selection, etc.)
• Re-sampling, re-weighting, and re-balancing datasets to reduce identified representation issues or
  imbalances” (p. 12)

“Training, evaluations and testing.

• Multi-disciplinary red teaming
• Independent third-party security testing, e.g., penetration testing
• Continuous monitoring to detect anomalies and security issues
• Multi-disciplinary red teaming
• Consultation of domain experts
• Multi-faceted evaluations, including standard benchmarks and proprietary evaluations based on identifi ed possible harms and harm reduction objectives
• User research of local language
  and cultural contexts” (p. 13)

“Deployment and maintenance.

• Blocklists, custom classifiers, and prompt injection guard filters, and human review to detect and intercept attempts to create unsafe outputs
• Specific mitigations applied based on deployment type, e.g., isolated customer environments with focus on remediating security vulnerabilities that coexist
  between traditional application security and AI security
• Security Information and Event Management (SIEM) system leveraging heuristics and advanced detection capabilities to identify potential threats
• “Air-gapped”” safeguards to prevent lateral movement and unintended network calls across environments and kernel-based LLMs to prevent the leaking of shared memories or buffers that could expose sensitive data
• Blocklists
• Safety classifiers and human review to detect and intercept attempts to create unsafe outputs
• Human-interpretable explanation of outputs
• User research and customer feedback analysis” (p. 13)

“Improvement and further fine-tuning.

• Responsible Disclosure Policy to incent third-party security vulnerability discovery
• Specific mitigations applied based on deployment type, e.g., isolated customer environments with focus on remediating security vulnerabilities that coexist between traditional application security and AI security
• Continuous evaluation and user research
• Programs to incentivize research, including research grants and participation in external independent research efforts.
• Multi-disciplinary red teaming” (p. 14)

3.1.2.2 Proof that deployment measures are sufficient to meet the thresholds (40%) 0%

No proof is provided that the deployment measures are sufficient to meet the deployment KCI thresholds, nor is there a process to solicit such proof.

Quotes:

No relevant quotes found.

3.1.2.3 Strong third party verification process to verify that the deployment measures meet the threshold (100% if 3.1.2.3 > [60% x 3.1.2.1 + 40% x 3.1.2.2]) 0%

There is no mention of third-party verification of deployment measures meeting the threshold.

Quotes:

No relevant quotes found.

3.1.3.1 Credible plans towards the development of assurance properties (40%) 0%

There is an explicit aversiveness to preparing for assurance processes in advance: “Cohere’s approach to risk assurance, and to determining when models and systems are suffi ciently safe and secure to be made available to our customers, is focused on risks that are known, measurable, or observable today.” Further, they note that “more work is needed to develop methods for assessing these types of threats more reliably” – to improve, the framework could set out a commitment to contribute to this research effort.

Quotes:

“One approach to risk assurance in the AI industry is focused on risks described as catastrophic or severe, such as capabilities related to radiological and nuclear weapons, autonomy, and self-replication. In this context, thresholds relating to these potential catastrophic risks are developed, and the approach described in safety frameworks is designed to assess risks that are speculated to arise when models attain specific capabilities, such as the ability to perform autonomous research or facilitate biorisk. The models are then deemed to present “unacceptable” levels of risk when certain capability levels are attained. While it is important to consider long-term, potential future risks associated with LLMs and the systems in which they are deployed, studies regarding the likelihood of these capabilities arising and leading to real-world harm are limited in their methodological maturity and transparency, often lacking clear theoretical threat models or developed empirical methods due to their nascency. For example, existing research into how LLMs may increase biorisks fails to account for entire risk chains beyond access to information, and does not systematically compare LLMs to other information access tools, such as the internet. More work is needed to develop methods for assessing these types of threats more reliably.” (pp. 14-15)

“Cohere’s approach to risk assurance, and to determining when models and systems are sufficiently safe and secure to be made available to our customers, is focused on risks that are known, measurable, or observable today” (p. 15)

3.1.3.2 Evidence that the assurance properties are enough to achieve their corresponding KCI thresholds (40%) 0%

There is no mention of providing evidence that the assurance processes are sufficient.

Quotes:

No relevant quotes found.

3.1.3.3 The underlying assumptions that are essential for their effective implementation and success are clearly outlined (20%) 0%

There is no mention of the underlying assumptions that are essential for the effective implementation and success of assurance processes.

Quotes:

No relevant quotes found.

3.2.1.1 Justification that elicitation methods used during the evaluations are comprehensive enough to match the elicitation efforts of potential threat actors (30%) 0%

There is no mention of elicitation methods being comprehensive enough to match elicitation efforts of potential threat actors. Elicitation techniques, such as fine-tuning or scaffolding, are not mentioned.

Quotes:

No relevant quotes found.

3.2.1.2 Evaluation frequency (25%) 0%

Whilst the framework mentions conducting evaluations “throughout the model development cycle”, more detail is not given. The frequency does not appear to be tied to the variation of effective computing power during training, or fixed time periods.

Quotes:

“As described above, Cohere conducts evaluations throughout the model development cycle, using both internal and external evaluation benchmarks.” (p. 16)

3.2.1.3 Description of how post-training enhancements are factored into capability assessments (15%) 0%

There is no description of how post-training enhancements are factored into capability assessments.

Quotes:

No relevant quotes found.

3.2.1.4 Vetting of protocols by third parties (15%) 0%

There is no mention of having the evaluation methodology vetted by third parties.

Quotes:

No relevant quotes found.

3.2.1.5 Replication of evaluations by third parties (15%) 0%

There is no mention of having the evaluation methodology vetted by third parties.

Quotes:

No relevant quotes found.

3.2.2.1 Detailed description of evaluation methodology and justification that KCI thresholds will not be crossed unnoticed (40%) 25%

There is a description of “continuous monitoring of our security controls using automated and manual techniques” and “various evaluations to ensure that models actually adhere to these guardrails.” However, more detail is needed on the exact methodology of this monitoring to ensure that the KCI threshold will not be crossed unnoticed. Monitoring should also explicitly be linked to the monitoring of KCI measures. To improve, they could build on their existing monitoring infrastructure which monitors for “malicious attempts to prompt our models for harmful outputs” to link directly to KRIs and KCIs that they’d like to monitor.

Quotes:

“We are also progressing work to further study models when in use and assess the real-world effectiveness of mitigations, while upholding stringent levels of privacy and confidentiality and benefiting from external expertise where appropriate.” (p. 8)

“Where applicable, we also consider risks within the context of customer deployments. For example, because many of our users start building applications through our application programming interfaces (APIs) before moving to more advanced deployments, we extensively test and secure our APIs. Our API V2 underwent a heavy security design review before we made it available.” (p. 10)

“Moreover, we identify risks across our broader technology stack and environment by performing continuous monitoring of our security controls using automated and manual techniques. Models are developed and deployed in broader computational environments, and effectively managing AI risks requires us to identify, assess, and mitigate information security threats or vulnerabilities that may arise in these environments.” (p. 6)

“Beyond simply offering these features, Cohere conducts various evaluations to ensure that models actually adhere to these guardrails.” (p. 11)

“Continuous monitoring to detect anomalies and security issues” (p. 13)

“Responsible Disclosure Policy to incent third-party security vulnerability discovery” (p. 14)

“Where Cohere has direct visibility into the use of its models during deployment, we use that visibility to monitor for malicious attempts to prompt our models for harmful outputs, revoking access from accounts that abuse our systems. Cohere partners closely with customers who deploy Cohere’s AI solutions privately or on third-party managed platforms to ensure that they understand and recognize their responsibility for implementing appropriate monitoring controls during deployment.”

3.2.2.2 Vetting of protocols by third parties (30%) 0%

There is no mention of KCIs protocols being vetted by third parties.

Quotes:

No relevant quotes found.

3.2.2.3 Replication of evaluations by third parties (30%) 10%

There is an indication that third parties conduct red teaming of containment KCI measures to ensure they meet the containment KCI threshold, but detail on process, expertise required and methods are not given, and conducting independent testing is still discretionary. To improve, there should also be a process for replicating / having safeguard red teaming conducted by third parties for deployment KCI measures.

Quotes:

“Cohere conducts multidisciplinary red teaming during both the model development phase and post-launch. These red teaming exercises may include independent external parties, such as NIST and Humane Intelligence, and are conducted based on realistic use cases to attempt to break the model’s ability to fulfill alignment on risk mitigation goals in order to elicit information about areas of improvement.” (p. 16)

3.2.3.1 Sharing of evaluation results with relevant stakeholders as appropriate (85%) 50%

There is a commitment to make public documentation of evaluation results. However, there is no commitment to notify government agencies if risk thresholds are exceeded. Further, there is not a commitment to make KCI assessments public.

Quotes:

“Documentation is a key aspect of our accountability to our customers, partners, relevant government agencies, and the wider public. To promote transparency about our practices, we:

• Publish documentation regarding our models’ capabilities, evaluation results, confi gurable secure AI features, and model limitations for developers to safely and securely build AI systems using Cohere solutions. This includes model documentation, such as Cohere’s Usage Policy and Model Cards, and technical guides, such as Cohere’s LLM University. […]
• Offer insights into our data management, security measures, and compliance through our Trust Center.” (pp. 17-18)

3.2.3.2 Commitment to non-interference with findings (15%) 0%

No commitment to permitting the reports, which detail the results of external evaluations (i.e. any KRI or KCI assessments conducted by third parties), to be written independently and without interference or suppression.

Quotes:

No relevant quotes found.

3.2.4.1 Identifying novel risks post-deployment: engages in some process (post deployment) explicitly for identifying novel risk domains or novel risk models within known risk domains (50%) 50%

Their monitoring mostly focuses on security vulnerabilities; nonetheless, they mention a process for performing “continuous monitoring” explicitly to “identify risks”. Whilst they may not be novel risk domains, it does suggest a willingness to detect novel threat models, detected via observation in the deployment context.

Quotes:

“Moreover, we identify risks across our broader technology stack and environment by performing continuous monitoring of our security controls using automated and manual techniques. Models are developed and deployed in broader computational environments, and effectively managing AI risks requires us to identify, assess, and mitigate information security threats or vulnerabilities that may arise in these environments.” (p. 6)

“Cohere partners closely with customers who deploy Cohere’s AI solutions privately or on third-party managed platforms to ensure that they understand and recognize their responsibility for implementing appropriate monitoring controls during deployment.” (p. 12)

3.2.4.2 Mechanism to incorporate novel risks identified post-deployment (50%) 0%

Apart from incidence response, there is no mechanism to incorporate risks identified post-deployment detailed.

Quotes:

No relevant quotes found.