The Real Blockchain Scalability Challenge

Blockchain technology has been unable to serve certain markets due to challenges scaling the protocol to handle the required throughput. There are two key measurements that impact blockchain scalability: real time performance and replay performance.

The Steem blockchain is able to process about 50,000 vote operations per second in real time and in replay. Each vote operation takes about 20us to apply. The problem is that we cannot ever allow the Steem blockchain to approach that transaction volume or no one would be able catch up to real time once they fell behind.

Ideally the replay time would not grow by more than 30 minutes per year. After 10 years of at-scale operation the blockchain would require 5 hours of replay time to catch up. Assuming we target Reddit scale of 500 transactions per second (real time), then we need a blockchain that is able to apply 500 * 60 * 60 * 24 * 365 transactions in 30*60 seconds. In other words we need a blockchain that has real-time processing capacity of 9 million transactions per second.

Even though we can process 50,000 votes in theory, in practice our blockchain has only scaled to 1000 transactions per second in the wild. We need to increase our real-world performance by a factor of 10,000 to achieve our target of 500 transactions per second (real time) while maintaining manageable replay times of 30 minutes per year of operation.

We can relax our requirements a bit and allow for 1 hour per year and only target 250 transactions per second. Now we only need to improve performance by a factor of 2500x.

Why Replay Performance Matters

It is only by replaying the blockchain that anyone can validate the current state. Without the ability to replay the blockchain in a timely manner it becomes increasingly difficult to deploy new nodes without trusting the derived state of past evaluations. Any time the derived state is corrupted, our database schema changes, or new capacity needs to be brought on line, blockchain administrators need to rebuild the state from the blockchain. When this happens it could result in days or weeks of service downtime while they catch up.

Just ask anyone tasked with deploying a Bitcoin service and having to replay the bitcoin blockchain. It can take over 12 hours on good hardware and they are only processing 5 transactions per second real time or an average of 2 transactions per second over the last 8 years.

Steemit has a Solution

I have long said that single threaded performance is the bottleneck of blockchain scalability. This is based upon the premise that every transaction has an impact on the global state and could potentially impact the validity of subsequent transactions. I have even indicated that synchronization overhead would negate any benefits of attempting to parallelize block evaluation.

We are determined to not let anything get in the way of scaling Steem and Steemit to an unlimited degree. This week our team has come up with a design and roadmap for scaling Steem to the speeds required to handle as many users and transactions as your all can throw at us. We are in the process of documenting our design and producing a roadmap for its development and deployment.

I can honestly say I have never been more excited about the potential for decentralized blockchains scaling to handle operations the size of Reddit and Facebook.