By Philip Glazman, Software Engineer
River Financial specializes in Bitcoin brokerage and custody services. Core to our operations is our infrastructure enabling us to custody and transact Bitcoin securely and at scale on behalf of our clients.
When starting the company, there was no off-the-shelf Bitcoin software solution that was capable of powering the product we were building. We needed a Bitcoin wallet service with the following traits:
- Capable of watching and rescanning millions of wallets, each with many thousands of scripts
- Capable of interfacing with heterogeneous set of signers (single and multisig)
- Flexible enough to upgrade quickly as protocol improves (Schnorr, Taproot, etc.)
- Capable of interfacing cleanly with monitoring and security tooling
- Has a clean, reliable API that is flexible enough to serve internal software services that power a quickly changing Bitcoin financial services institution
- Can generate transactions for Lightning Network channel management
- Modular enough to physically isolate highly sensitive components (e.g. signers)
Historically, Bitcoin software built for exchanges and brokerages - which we refer to as enterprise Bitcoin software - has been proprietary and severely lags consumer Bitcoin software. This is evidenced by the multi-year delays between network protocol upgrades and support from even the largest exchanges. Many companies also have the disadvantage of needing to support arbitrary numbers of cryptocurrencies, preventing them from optimizing their infrastructure for Bitcoin.
At River we decided to reimagine what Bitcoin infrastructure could look like and built what we believe is the world's best enterprise Bitcoin software from the ground up. This post goes into the details of our Bitcoin infrastructure, the design decisions we made, and what we learned along the way.
Our Bitcoin architecture consists of three main components:
- Walletd: A service responsible for maintaining UTXO ownership data for an arbitrary number of wallets
- Signerd: A signer responsible for hot private key material
- Indexd: An index that maintains a view of the current state of the Bitcoin network
Written in the Go language, these components are services with singular responsibilities and ownership of local data, communicating via a RabbitMQ AMQP message broker. We also run Bitcoin Core nodes through which we interact with the public Bitcoin network. Indexd talks directly to a Bitcoin Core node (see diagram below).
Enterprise bitcoin infrastructure is primarily responsible for observing the state of the blockchain, updating internal systems based on changes to that state, and contributing valid transactions to the network. A transaction is an object that contains information on how bitcoin is being spent. When a client sends Bitcoin to their River deposit address, we will see a transaction on the network creating a new output owned by River wallets. When a client withdraws Bitcoin, Walletd attempts to create a valid transaction with an output that the withdrawal recipient will own. Ownership of an output is defined by the scriptPubKey, or locking script, that sets the rules for how the bitcoin can be spent. The various addresses one sees, such as P2PKH (beginning with 1) or P2WSH (beginning with bc1), are different ways to lock bitcoin.
Describing the standard transaction lifecycle can help illustrate how our Bitcoin infrastructure works as a whole. Typically, a Bitcoin transaction will first be detected by our system when it enters one of our nodes’ mempools, a local set of unconfirmed transactions. A transaction can stay in the mempool until it is either invalid, confirmed, or is evicted due to elapsed time or space constraints. Although some transactions do not enter the mempool before they are included in a block because they were sent to a miner directly, most transactions use the mempool to signal willingness for block inclusion.
Upon detection, Indexd parses the transaction and serializes relevant information for persistence. We persist all transactions that have occurred in the Bitcoin blockchain and network in a PostgreSQL database. This allows us to quickly query for transactions (via txid), scripts (scriptPubKey), and outpoints (txid:output). Using the message broker, our services use events to communicate new changes to the state of a transaction. Transaction detection in the mempool is an event that Indexd will subsequently publish to Walletd. The wallet parses the message and updates the state of UTXOs and scripts accordingly.
There are several events that the system is prepared to handle from the network including new transactions, blocks, rejected transactions, and block reorganizations.
A critical feature of this system is the persistence and ordered property of messages which flow through the AMQP message-broker. Given that we can always assume a new block will be parsed approximately every ten minutes, we do not need to optimize for scale or throughput. There is also an upper limit for compute resources for each block. Instead, we optimize for availability and reliability.
The ordering of state changes matters in the context of Bitcoin because a transaction can become detected in the mempool, confirmed in a block, orphaned, no longer valid, or evicted from the mempool. A guarantee of our message-broker is the ordering of messages. This is an especially useful property when block reorganizations happen.
When River Financial detects a transaction in our mempool, Indexd will parse the transaction and save the new unspent transaction outputs (UTXOs), scripts, and spent outpoints. It emits a message to Walletd with the new state of this particular transaction. Outlined below are some states that a transaction can experience within the course of its lifetime:
A transaction is accepted into the mempool. This transaction is waiting for block inclusion. A River client will see a pending transaction.
A transaction is confirmed in a block. Due to the risks of block reorganization, River does not consider the transaction to be in a final confirmed state until the confirmed block has a certain depth (e.g. 6 blocks).
A transaction can be rejected for a number of reasons. In this example, the transaction is evicted from the mempool. Though the transaction can appear in a block at a later time, we mark the transaction as rejected. A double-spend is another form of transaction rejection that can occur.
A transaction can be unconfirmed during a block reorganization. When this happens, the transaction is no longer in a block and the transaction lives in the mempool until a new valid block can be found. This particular state can be short-lived, or lead to transaction rejection should a double-spend occur.
Walletd is a service with many wallets that each have their own set of scripts and UTXOs. Each wallet is defined by a wallet descriptor that creates the scope and script semantics that the wallet is expected to follow. Walletd uses an interface to communicate with different signers and does not contain any key material. As a result, the service can only maintain watch-only wallets. Avoiding key material inside the Walletd process allows us to have a diverse set of signers isolated by hardware and software.
At the root of each wallet is the Output Script Descriptor language, authored by Pieter Wuille, which is also used in the Bitcoin Core wallet. The descriptor language helps us describe wallets in a maintainable and expressive way that is interoperable with other wallet software. Our Field Report with Bitcoin Optech goes into further detail on the rationale behind River’s choice to use descriptors. With a wallet descriptor, Walletd can generate the appropriate scripts and addresses.
Descriptors ensure interoperability and reduce the complexity for creating new wallets. Before this standard, proprietary methods were used to pass the information for what constitutes a wallet. With Walletd, there is a straightforward API method that can be reduced to providing a single descriptor.
Walletd is designed to scale to millions of wallets, as each wallet will have different use-cases. In our enterprise wallet, we maintain visibility of cold wallets and hot wallets. Our cold wallets are bound to multi-signature P2WSH script generation while our hot wallets use the highly available P2WPKH script. In addition to differing script semantics, each wallet has different policies with regards to spend eligibility and other properties. A hot wallet servicing withdrawals will have different policies than a wallet responsible for creating funding transactions on the Lightning Network.
Upon detection of a transaction, Walletd can associate one of the outputs’ scriptPubKeys with a script belonging to a wallet descriptor. A scriptPubKey is the locking script that contains the semantics on how to spend Bitcoin. When processing a transaction, Walletd loops through each output in a transaction and checks its set of scripts for any matches. On a match, the output is a deposit to one of the wallets in Walletd. The state machine notices the new UTXO associated with the script and emits balance update event messages to any Walletd consumers.
As soon as the UTXO satisfies our spending eligibility requirements, Walletd can create a transaction using the new outpoint. Throughout the transaction creation process, we use Partially Signed Bitcoin Transactions (PSBTs), a serialization standard for describing unsigned transactions proposed in BIP174. Adopting this standard reduces complexity and maintains interoperability with other wallets. Most importantly, it allows us to treat any signer as an interface. Each wallet has its own signer source that understands the PSBT format. A signer can be a software signer, a hardware device, or a set of both. Signerd is an example of a signer that has an API that can parse PSBTs, validate transactions, and sign transactions.
When Signerd successfully signs the transaction, Walletd can finalize the PSBT and securely broadcast the transaction to the Bitcoin p2p network.
The focus and flexibility of our architecture lets us build protocol-specific features to improve the experience of sending and receiving bitcoin. After broadcasting the transaction, a fee spike may occur causing our fee estimation to underdeliver the target for block inclusion. Walletd can leverage Child Pays For Parent (CPFP) to incentivize quicker block confirmation and speeding up delivery of funds for our client. For example, the next transaction from our system will also spend an output in the pending transaction in the mempool with a higher fee rate. This transaction package incentivizes block inclusion for miners. During periods with many withdrawals, we can also batch transactions to lower the overall cost of each withdrawal output. This involves grouping each withdrawal output into a single transaction. This set of tooling among other utilities like UTXO consolidation and coin splitting are requirements for an enterprise system.
It is imperative that an enterprise Bitcoin architecture maximizes security, continuity, and functionality. As a result, all of our design decisions for this architecture are optimized for these requirements.
Security is the foundational principle of our architecture. The most important function of the system is to protect private keys. Isolation of processes and services significantly reduces the attack surface. Our highly available signers responsible for withdrawals live outside of the Walletd process to ensure software isolation. Furthermore, our infrastructure is hosted on our own physical servers ensuring strong physical hardware isolation guarantees that cloud cannot provide.
Operational security is a consideration in the design of Walletd architecture. We use a requester-approver model for critical phases along the transaction creation lifecycle. PSBTs significantly reduces the complexity of Walletd communicating with a diverse set of hardware and software signers. The PSBT format provides a common interface that the wallet can parse and serialize, therefore the API between wallets and signers is common no matter the composition of the signer itself - whether human or computer.
Continuity is a critical function of our engineering. The privilege of custodying of clients’ Bitcoin means that our ownership of the Bitcoin should not be contingent on the employment of a particular set of people. The design of wallets, their scripts, and stakeholders are a part of the business continuity strategy. At River, Bitcoin is locked in different wallets that have different responsibilities. Walletd can import and create new wallets at any given time, therefore the particular script semantics of a wallet can be adjusted to accommodate new requirements.
In today’s architecture, there are primarily two wallets to be concerned about: a cold and hot wallet. The hot wallet is a highly available wallet that spends P2WPKH outputs to service withdrawals on the platform. Due to the trade-off of availability and security, the amount of Bitcoin in this wallet is very limited. On the contrary, the cold wallet is made significantly less available to improve security. Its function is to receive client deposits and spend P2WSH outputs. Scoping a wallet with a particular responsibility allows River to create new wallets in the future that have new script semantics and signer parties. Continuity of the Bitcoin custody is contingent on wallet schemes that are related to the total value of Bitcoin held or expected to be held in the scripts.
A system that is designed to be robust, yet flexible, is critical to an enterprise setting. Clean code that is manageable should have a long shelf life and be flexible for future product iterations. This was a reason for choosing Go as the primary language for the aforementioned services. Working in Bitcoin means that there will always be new changes and challenges to be prepared for. Given the staggering rate of change for the base protocol and the Lightning Network, building the Bitcoin tech stack in-house allows us to quickly accommodate future protocol upgrades and provide the best Bitcoin experience to clients.
The ability to have broad functionality is particularly useful when implementing the Lightning Network into our core products. The utilities and state machine in Walletd is part of the Lightning domain. Wallets maintained by Walletd can create channel funding transactions through PSBTs. As the requirements for Lightning grow, more of the Lightning state machine will be in the same domain as on-chain Bitcoin. More information about our Lightning Network integration will be part of another blog post.
The future aim of this architecture is to continue to service more Bitcoin products. This includes continuing to focus on new non-custodial product offerings like our Hardware Wallet Account, which allows clients to register their Hardware Wallet device and generate deposit addresses, track their transaction history, and use our performance tools. Building natively on Bitcoin allows us to have an attention to detail that leads to interesting and exciting new products. Furthermore, we are well prepared to deal with future developments on the Bitcoin base layer and on second layer solutions like the Lightning Network.