Scaling Lightning Network at River

The Lightning Network (LN) is exploding with growth with over 2,300 Bitcoin publicly in the system, and over 70,000 announced channels. There's an entirely new wave of bitcoiners participating, a government actively onboarding its country, and innovative companies built around this permissionless network. In this blog post, we cover some background on LN, our technical experiences and share our excitement for its future.

Why Lightning Network

The LN protocol is a scaling solution for Bitcoin using bidirectional payment channels and building upon primitives in the Bitcoin scripting language and peer-to-peer networking. Once a payment channel is created among two parties, it can remain open and process a nearly limitless amount of transactions. It is also trustless due to cryptographic signatures, bitcoin script validation, and enforcement of the blockchain. These channels are connected between online peers and form a network called the Lightning Network. The network is politically neutral, and payments are routed along payment channels trustlessly.

As on-chain Bitcoin transaction fees continue to rise, settlement using the blockchain will become prohibitively expensive for low-valued transactions. Alternatively, settlement on the LN will continue to be an affordable, attractive option for those that want a trustless exchange of value on the internet. River Financial acts as both an interface to the Lightning Network and a reliable router for payments. We have been operating in the network since October 2019 and are excited to continue sharing developments in the space with our clients.

To learn more about the fundamental concepts underpinning Lightning, please refer to our River Learn article on the subject.

River's Commitment to Lightning Integration

River Financial has operated nodes in the Lightning Network for more than two years, and we've learned several lessons along the way about user experience, engineering, and operational knowledge. The most immediate lesson we learned was the importance of education for our clients. The dichotomy between on-chain and Lightning Network Bitcoin is apparent, and there are countless efforts to help bridge the gap.

We started with the assumption that clients shouldn't care about the underlying payment rail used to fulfill their transfer. On our transfer page, a client can paste a BOLT11 invoice or an on-chain address, and we handle the parsing to guide them to the next step.

When a client submits a valid BOLT11 invoice, we probe the network to estimate an upper-bound fee limit suitable to fulfill the payment. A probe can fail for various reasons and provides a growing data set into the reliability issues related to our configuration with the network. Failures can occur when routes attempt to navigate poorly balanced channels or unreachable end-user destinations. We provide a confirmation page for the client to accept the payment transfer on a successful fee estimation request. The payment is then sent along a route in the Lightning Network onwards to its final destination.

We have identified several user groups that transfer on the LN in a variety of ways. These groups include consumers, traders, and hobbyists. Clients that fit into these groups have different requirements and expectations about channel liquidity. These requirements ultimately dictate our operating decisions with our nodes in the network.

Consumers are the retail users that use LN as a payments platform for instant settlement. They purchase Bitcoin on River and use our LN withdraw feature to send payments to merchants. Some of the outbound payments are sent towards familiar businesses, including Bitrefill and Fold. This behavior requires us to maintain outbound liquidity to merchants.

Traders are using River as an easy-to-use wallet for transferring Bitcoin between exchanges. They use River for access to USD liquidity. This behavior is unpredictable and requires balanced channels between some major exchanges, including Bitfinex and Okcoin.

Hobbyists are technically savvy power users who primarily use River as a wallet to conveniently interact with the LN. An emergent user story among this group is using River as a swap service. Users can transfer value both on-chain and LN, and therefore can swap their bitcoin on the Lightning Network for an on-chain UTXO without much hassle.

River's Technical Strategy for Lightning Infrastructure and Products

Integrating Lightning into River's tech stack presents unique challenges, considerations, and opportunities for the engineering team. However, we have found that operational considerations typically supersede engineering challenges.

Technical Overview

Our in-house wallet service manages the transfer of funds for both the chain and Lightning Network. Written in Go, the wallet can use LND as a library in addition to a daemon. Given the security requirements for digital asset custody, the wallet and LN daemon nodes run on on-prem hardware we control. We expose a single routing node to the public network, which governs funded channels and maintains relationships with peers. Meanwhile, several end nodes are connected to the routing node with unannounced channels. These end nodes manage transferring payments, while the routing node manages liquidity with the rest of the network.

Our integration to the network using LND has been smooth sailing. The project supports a reliable gRPC service and subservers organized by domain. The daemon exposes several bidirectional-streaming RPCs that allow the client granular control over some functionality. For example, we rely on the ChannelAcceptor RPC so that a client application with business logic can make decisions for accepting inbound channels. We leverage some of the unary RPC endpoints for various utilities, as well as funding lightning channel transactions with PSBT packets.

A Lightning Network node requires high uptime guarantees and a highly available signer to be available. In the context of custody, most LN nodes in production are hot wallets. More so, the state machine of payment channels produces toxic waste, which can lead to a loss of funds. If we lose data integrity in our disks, it's possible for a node to publish a previous commitment state for the channel. Publishing an earlier state of the current protocol regime would lead to a justice transaction that would forfeit our entire claim to any balance. However, there are improvements to eliminating this feature of LN (eltoo).

We maintain static channel backups and perform a backup on any changes to our channel set. For the Data Loss Prevention (DLP) protocol to work as intended, the set of channel peers is limited to ones that we view as reputable and have consistently high uptime.

Since our first efforts at integration, managing our Lightning Network infrastructure has been an operational challenge. We have spent time building internal tooling to scale several of these processes to deliver audits, improve liquidity management, and increase operational security. The observability requirements necessary to sustain a positive user experience have been a challenge since the beginning. A non-exhaustive list of items that require monitoring includes channel liquidity, peer reliability, and payment success rates. Managing outbound payments is straightforward since the feedback is immediate. However, it is difficult to monitor the success of inbound payments from noncustodial and custodial wallets. This is an area of work we consider important moving forward.

To ensure reliable payments, our routing node maintains well-capitalized channels with peers in the network. Although we peer with our clients directly, most channel relationships are with other routing nodes, merchants, and businesses. In a nascent ecosystem, the communication for forming and closing channels is generally done on messaging apps like Telegram or Signal and is still very adhoc. Communication regarding channels is likely to change as the ecosystem evolves and matures.

Security Design

Operating a secure LN integration requires a different approach than standard blockchain security. For operational security, the principle of least privilege dictates access to running production nodes. Access to the running container is locked down, and access is limited. The attack vector for a node is substantially higher than a secured cold wallet. Therefore there is a strict maximum of notional value that is permitted to be owned by a node. On a successful channel close, funds get swept to our secure cold wallet custody. Opening a channel from our side requires our segregated on-chain wallet to sign a PSBT packet. The actual revocation and commitment state machine operates within the node. There is, however, an ongoing effort to continue to own that domain outside of a node.

Our peers might have noticed that we maintain an allowlist to help mitigate random channel opens. We choose to limit our inbound peers to reduce any chance of an attack surface and keep our peer relationship limited to trusted parties.

With the exponentially increasing amount of Bitcoin being transacted on the network, we will continue to operate in a security-first environment.

Moving Forward

At River, our mission is to accelerate the adoption of Bitcoin. Lightning Network is part of this story because it enables cheaper and faster payments. Our goal is to increase the volume and liquidity of the Lightning Network, and our integration is a first step in that direction.