Welcome to the second post of the Private Key Management for Blockchain Operations series!
- [1/4]: Concepts, Types of Wallets and Signing Strategies
- [2/4]: Common Custody Solutions Architectures
- [3/4]: Private Key Storage and Signing Module
- [4/4]: Approvals and Policies
On the first post we talked about the key concepts around custody and blockchain operations. In this second one, we'll introduce some common architectures, and the most common components in the architecture of a custodial solution.
One of the primary aspects we focus on during architecture reviews and threat modeling is the presence of single points of compromise; a component or privileged role (such as the identity provider admin) whose compromise could lead to full platform control or total loss of client funds. To address this, many modern custody solutions adopt a Zero Trust model, in which every request is continuously authenticated, authorized, and validated. This approach helps ensure data integrity and minimizes the risk of systemic compromise.
Regardless of the architecture model chosen, it's important to pay special attention to how Zero Trust principles are applied and whether single points of failure or compromise are effectively eliminated.
Off-Chain Centralized Architecture
Let's take as an example this simplified diagram highlighting some of the most common and necessary components in an off-chain centralized architecture:
Signing Module
This is the element that has access to the private keys, meaning that it can use them for signing blockchain transactions. This does not necessarily mean the module has access to the plaintext private key, only that it can use it. How it accesses the private key depends entirely on the implementation and architecture. In some cases, private keys may be protected by hardware (e.g., in an HSM or KMS), and this module will have permission to use them. In others, this module may be implemented inside a secure enclave that decrypts the key securely within the enclave for use.
As the module with access to the keys, it is one of the most critical components of any architecture involving blockchain operations. Vulnerabilities in the design or implementation of this element may compromise the private keys.
This element, by design, is the main Single Point of Failure. As the most critical component in this kind of infrastructure, we will analyze it in detail in the next post: State of the Art of Private Key Security in Blockchain Ops - 3. Private Key Storage and Signing Module.
Approvals and Policies
This is the other critical piece that we consider necessary in any blockchain operations solution. There are two main elements:
Approval policies are critical in any blockchain operations solution. There are two main elements:
- Policies: Define which actions can be taken, who can take them, and the required number of approvals (threshold) depending on the action.
- Approval Processes: Ensure transactions are approved by the correct number of participants.
Approvals is another topic that deserves its own post, so we can explore it in detail. As so. We will dedicate another post just for this:
Databases or Other Storage Types
Most applications will include some sort of databases where they need to store different configurations. The main threat to think when assessing these elements in a custodial architecture is to, for each critical asset stored within the database, review if an unauthorized change (e.g. by a database administrator) may allow to compromise the solution.
For example, if whitelisted addresses are saved in a database, and a threat actor changes them to different addresses, would it compromise the integrity of the transactions made? In this case, whitelisted addresses should be signed by the approvers, but what if instead of whitelisted addresses, an address book, to name addresses, is used? Would that be enough to cheat approvers to make them think they are approving a transfer to a common address without realising they are approving a transfer to a different address?
For each of the configuration items or assets stored, the same kind of question needs to be answered. If an attacker is able to change this asset, can they compromise the solution?
Transaction Manager
Another piece of any blockchain-related architecture will be a transaction manager, or piece of code that will connect to a Blockchain node to both write information to the blockchain (submit transactions) and read information (state, new events, transactions)
Depending on the architecture, this may be divided in modules depending its purpose:
- Process transactions sent to the network
- Monitor if a transaction is confirmed
- Fallback mechanisms for transactions
- Cancelling (Dropping) Transactions
- Gas (fees) decisions
- Transaction re-submission with higher fees
- Events monitoring
- State checking
Blockchain Nodes
Some organizations decide to run their own nodes in their infrastructure, while others avoid that hassle by using third party services that offer node provider capabilities. The main risk to consider in this case is whether too much control is given to a specific party. If the node provider suffers a denial of service condition, the blockchain project would not be able to connect to the blockchain. Also, if the node provider is compromised it could give malicious data to hide or tamper the information seen by the project. This would be more risky if the provider for the nodes is the same one providing custodial services.
For these reasons it is usually recommended to have several providers, if possible running different node software, to be able to have high availability even if one provider fails, or if a specific node software bug causes a DoS.
Settlement (Custodials)
Used when custodying funds from different entities into the same wallet. A typical case of use is the one of many centralized exchanges, which basically have a book of how much of each crypto asset each user has. In this kind of setup, the critical vulnerability we see more often is race conditions that may allow a user to perform double transfers / double withdrawals to duplicate their crypto currencies, causing financial losses.
Message Queues
An element that is very common in modern architectures, specially in microservices architectures, even if not particular to solutions related to blockchain operations, is the message queues. These are often used to communicate all different modules within each other. The most important aspect to consider when using a message queue for your elements is to ensure proper authentication, and follow the principle of least privilege. Failures in the design of this element may convert it as well in a Single Point of Failure, meaning that if that queue is compromised, or any microservice with access to the queue is compromised, could allow to request arbitrary messages to be signed, potentially compromising the whole solution.
Off-Chain MPC Architecture
The diagram below depicts a basic example of an MPC solution, showing the core elements:
The main difference between this solution and the previous, centralized one, is that the keys are distributed between the approvers and potentially other systems. This architecture eliminates the main potential single point of failure by design (considering a correct MPC implementation), as signing a message requires N of M keys, which are distributed.
In the case of Fireblocks, one of the most known MPC providers, the different approvers have one key each on their mobile application, requiring N of M of them to cryptographically sign a transaction for it to be approved.
However, for many operations (hot wallets and operational wallets) it may be needed an automated way of signing a transaction without human intervention. In this cases, Fireblocks defined a piece called Co-Signer, simplified in the diagram above as the MPC Signing Module, which would be a similar concept to the signer module in the centralized solution, but in a different trust boundary, which can be controlled by a different party. These automatic signers can also be useful to enforce specific checks before signing a transaction, possibly stopping human errors that could happen during manual approval processes.
On-Chain Custody
The diagram below shows what we consider the basic components of a typical On-Chain multisig, in this case showing multi-sig based on smart contract wallets.
We could compare this with a simplified version of Safe{Wallet} (however, what we mention in just 2 components on the "Dispatcher Infrastructure" has way more functionality in the Safe Infrastructure, with more than 10 different micro-services)
The main piece of potentially centralized infrastructure needed is a web3 application that allows the interaction with the smart contracts.
The second piece, which may not be necessary in case of several transactions going on-chain, but that is completely needed for off-chain signing (e.g. EIP-712 messages, like in Safe{Wallet}), is the piece we called Approval Dispatcher.
As with any Web3 application, client-side vulnerabilities becomes critical. A cross-site scripting would allow to tamper the signatures sent to the wallets, or the information shown by the web application.
The approval dispatcher purpose is to collect the messages or transactions to be signed, and present them to the rest of the signers. A bad implementation of this piece may allow unauthorized parties to "propose" malicious transactions to be signed, increasing the risk of the signers being tricked.
In this diagram, we showcased the most common method used to sign transactions. A browser wallet extension (like Metamask or Rabby Wallet) connected to a Hardware wallet (like Trezor or Ledger). It is important to notice that the web browser wallet can also hold private keys, but there are several reasons of why using this is not secure:
- Private keys in a browser wallet can be exported if the passphrase is compromised
- If the device running the extension (generally, a laptop) is compromised, then all the information shown by the web browser and by the extension may be compromised and cannot be trusted. Instead, if additionally using a hardware wallet, the hardware wallet will show the real information that is being signed, even if all other devices and interfaces are compromised.
In this case, we are focusing on the off-chain side of the infrastructure. However, the MultiSig Smart Contract could implement functionalities such as policy checks, whitelisting of smart contracts and functions that can be interacted with, different roles, signatures from different roles, etc.
As we've seen, every architecture (centralized, MPC-based, or on-chain) introduces its own set of trade-offs, risks, and trust boundaries. Our next step will be to dive into the beating heart of all of them: how private keys are actually stored and used to sign transactions.
Stay tuned for the next post in the series: Private Key Storage and Signing Module.