tl;dr
The OCP (Open Compute Project) S.A.F.E. (Security Appraisal Framework and Enablement ) framework was published in 2023 with the goal of providing a process for auditing firmware for vulnerabilities and assurance to end users of the security of code running on their devices. Unlike certifications, such as Common Criteria, FIPS, or PCI-DSS, which focus on meeting a list of explicit requirements, S.A.F.E. is strictly a framework by which firmware is reviewed. There is no concept of “passing” or “failing” a S.A.F.E. review, only an assurance that a review was performed with a common, standardized approach. Prior to this framework, there was no other standard that focused on firmware security, and with its widespread adoption, we have seen recurring questions come up related to the exact process for vendors to get evaluated against the framework. The goal of this blog post is to do exactly that, explain the process for a vendor from scoping the assessment to the publication of the first Short Form Report (SFR) and through follow-up incremental changes evaluations.
What’s the process?
Threat model
All OCP S.A.F.E. assessments start with a Threat Model (TM), that will inform the review, focusing on the areas detailed in the framework. If the Device Vendor (DV) already has a documented TM, the Security Review Provider (SRP) will assess it for completeness and alignment with the current architecture of the firmware and hardware. If there isn’t a formal TM yet, the SRP will create one.
The DV will provide the following, depending on the specifics of the product under review:
- technical documentation, including any non-customer facing documentation such as architecture documents, interface definitions, and detailed technical design documents, in addition to customer facing documents such as service manuals and installation/configuration guides,
- schematics, security-relevant component data sheets, and board layout information including component placement and reference designators,
- access to subject matter experts covering the overall system, applications, hardware/firmware, and related services.
For every follow-up OCP S.A.F.E. endorsement for new firmware version releases, a gap analysis should be performed, to ensure there are no new security implications.
Security review
After establishing the threat model, the SRP proceeds with the review phase. There are three scopes that can be followed, depending on the product’s TM. These scopes increase the complexity of attacks in the threat model and build one upon the other, so a review including all scopes would need to have the following sequence: Scope 1, Scope 2, Scope 3.
Scope 1
Scope 1 focuses on vulnerabilities that can be used to compromise a component.
Our practical experience
Firmware source code review during a Scope 1 assessment can include analysis of interfaces, communications, command handling, and other exposed attack surfaces to find common mistakes. This may also include examination of how cryptographic primitives are used and composed into a full cryptosystem, which often can hold obscure bugs.
Scope 2
Building on top of Scope 1, Scope 2 focuses on interaction between components.
Our practical experience
This review process looks at how various subsystems of a device work together. These may be logical divisions within a single processor, or multiple physical components communicating in an assembled device. Attention is given to identifying explicit, implicit, and assumed trust boundaries, looking for consistency in implementation across all the components. This is a more abstract examination of the device as a whole unit or set of subsystems. For example, in one device we discovered unclear trust boundaries between the main system and a newly-developed trusted execution environment, opening the way for one to exploit the other.
Scope 3
Scope 3 focuses on physical resilience protections and can be a mix of code and configuration review as well as hands-on testing.
Our practical experience
Hands-on testing may involve probing hardware interfaces, fault injection via electric, electromagnetic, optical, or other means, and analysis of side-channel data leakage. Strategies for testing physical devices are guided by details learned during Scope 1 & 2 phases, and will vary depending on the nature of the device and context in which it’s expected to be used. Recent work entailed fault injection into the main SoC of a device, with the intent to bypass the firmware validations of the secure boot process. While this type of work can take a great deal of effort, in a short time frame we were able to show that these attacks were plausible.
Reporting
At the end of the assessment, the SRP delivers a detailed report to the DV with the TM and findings, as well as remediation measures that could be taken to fix those issues.
Retest
DV will work on remediating issues and let the SRP know when a retest can be performed. The goal of the retest is to confirm whether the fixes adequately address the reported issues. When the retest has been completed, the SRP will provide the DV with an updated detailed report with the retest results added.
SFR publication
Upon the DV’s request, the SRP will generate the OCP S.A.F.E. Short Form Report (SFR). Note that the DV controls this process. No SFR or information from the review will be published without an explicit request from the DV.
To claim OCP S.A.F.E. endorsement for a product-firmware combination, this report must be published to the OCP S.A.F.E. GitHub repository. If there are still open issues in the SFR, it's recommended that the DV publish it to ensure that the DV is in control of messaging around any potential vulnerability disclosures. If all the issues have been fixed and the SFR has no issues to report, the SRP will publish the report.
Note: If the DV has not provided sufficient information for the SRP to perform a comprehensive assessment, and the SRP thinks that everything in scope for the OCP SAFE endorsement couldn’t be covered due to this, they can refuse to provide the SFR to the DV.
Follow-up tests
The process can be restarted for a new product-firmware combination release. The DV will provide the SRP details of the differences between the new release and the previously OCP S.A.F.E. endorsed release, including architecture, documentation and source code. The SRP will go through the process again, focusing on the added features, in order to endorse the new release. The process will ensure the added features don’t open any gaps in the TM and in the product-firmware release.
What Next?
We hope this has cleared up some of the more practical details surrounding reviews performed under the OCP S.A.F.E framework. When you’re ready to start the process, please reach out to us at ocpsafe@nccgroup.com. We look forward to partnering with you to validate and improve the security of your firmware products.