Blockchain Update 2: HF20 Progress & Bandwidth Changes
In our last blockchain update, we announced that we were making HF20 a priority. Since then, we’ve worked hard to accelerate the release. In today’s post, we will explore an especially important part of HF20: improving the bandwidth formula of the Steem blockchain.
Reforming the Bandwidth Formula
As explained in our previous post:
as we continue to scale the blockchain to more and more users, the bandwidth formula that we use to allocate resource usage across all of those users becomes increasingly important.
Move to Resource Credit System
To address this issue, we have completely rewritten the bandwidth system and created a new, state-of-the-art resource allocation system built around a new unit called “Resource Credits” (RCs).
This resource allocation system is the first of its kind in the blockchain / cryptocurrency space and will be the most advanced “freemium” blockchain model in existence.
RC Design Goals
Three primary design goals guided our development of the new bandwidth system. We wanted to create a system that 1) more efficiently allocates blockchain resources; 2) more accurately measures the true cost of running the blockchain; and 3) enables Steem developers to create more predictable user experiences.
Intelligent Resource Allocation
The most important goal of the resource allocation system is intelligent allocation of the blockchain’s resources. The more efficiently resources are allocated, the more user activity can be sustained. This is a critical component of scaling the blockchain. It is also critical that the system disincentivize the over-consumption of resources by users as well as spam.
Accurate Transaction Cost
Every transaction on the Steem blockchain has an associated “cost” in terms of the strain it places on the blockchain’s resources. For example, storing more data in the blockchain can eventually lead to higher RAM requirements for steemd nodes. Increased resource requirements can lead to higher costs for node operators (including developers, witnesses, and exchanges).
This is why it was critical that the blockchain calculate a transaction’s cost more accurately, based on as many factors as possible, and be empowered to limit which transactions can be performed based on their cost to the network.
These factors could include things such as:
- Blockchain history size
- Reindex time
- State file size
- Memory usage
- Disk iops
- Network bandwidth
Predictable UX
Under the current bandwidth system for Steem, users are given a certain number of “bytes” to use based on the amount of Steem Power (SP) they have. The blockchain reduces or increases the number of bytes they are given based on the activity occurring on the blockchain.
While this solution got us to where we are today, it confuses end-users by providing them with a purely binary choice: either acquire more SP, or be unable to transact on Steem. End-users cannot, for example, be told how much bandwidth they will receive from their purchase of more SP. The amount of bandwidth that a certain amount of SP is worth also changes constantly, depending on the activity level of the network. In other words, as the system stands, it is all but impossible for Steem users to form an accurate mental model of bandwidth.
Enabling the formation of a mental model is critical to satisfying our third design requirement: more predictable user experiences on Steem. The predictable model that will result from this new system will enable simple and effective feedback from user interfaces (UIs) to inform users about their RC usage, their remaining RCs, the cost of new transactions (measured in RCs), and options for purchasing additional RCs by powering up SP if they would like additional bandwidth.
Dynamic Cost Calculation
An important design principle carried over from the previous bandwidth implementation was that of dynamic costs based on the blockchain’s activity, as this is a very efficient system. Transactions should “cost” more during times when the blockchain is busy and less during times when it is not. However, as we’ve discussed above, this approach comes with certain downsides in terms of users being able to form a predictable mental model of bandwidth use.
The solution we propose in our RC system is to have a one-to-one correlation between the number of RCs one can get with a certain amount of SP, but at the same time have the costs of various transactions (measured in RCs) change based on the blockchain’s current activity level. The costs are determined based on an internal, market-based system, which is explained below.
Gamification
An added benefit of the new system is that it will further gamify the Steem experience. When a user can know how many RCs they will have, how long it takes for those RCs to recharge, and how many RCs each transaction will cost, they will be able to make informed decisions about how to spend their RCs.
While a primary objective for Steem is to minimize the cognitive load on users, this system will create new opportunities for developers looking to create more engaging applications. Developers will be able to provide interfaces that encourage their users to think tactically about how best to use their limited resources, because more granular data about resource consumption by individual users will be readily available. In addition, those users who may be inclined to over-consume blockchain resources (i.e. spammers) will be discouraged from doing so, as they will be shown very clearly just how much of their limited resources their efforts are consuming.
Maintaining a Frictionless Experience
It is important to reiterate that the appropriate baseline user experience for Steem is: as little friction and cognitive load as possible. We refuse to offer solutions that interfere with this principle. However, another important aspect of any protocol is that it be maximally flexible. Users who simply want to read a few posts, leave a few comments, and send some Steem should be able to do so with ease. But it is also our responsibility to consider the people who want a more active and gamified user experience, as well as those seeking to waste network resources. In a sense, we want Steem to be a game that everyone can enjoy playing, regardless of the type of user experience they prefer or their motivation for playing.
What are RCs?
The most important thing to understand about RCs is that they will be non-transferable credits given to each account based on how much Steem Power it has, which get “spent” whenever a user transacts with the Steem blockchain. Of course, if RCs did not replenish over time, then eventually everyone would become unable to transact at some point. For that reason, RCs regenerate over time.
Fueling Transactions
Once the RC system goes into effect, all transactions on the Steem blockchain will be given a cost in terms of RCs. If an account doesn’t have sufficient credits, the transaction will not be allowed to occur. This is the same as the existing system, which does not allow transactions to occur if the account does not have sufficient bandwidth.
Resource Budget Pools
The new system differs from the current one in the creation of “resource budget pools” which will be established for each resource type (RAM usage, reindex time, etc.). Each pool could be viewed as the blockchain’s “stockpile” of the corresponding resource. RCs will be the “currency” that the blockchain uses to price the resources in those stockpiles, and the internal medium of exchange for said resources.
Again, this is all happening behind the scenes and will require no conscious engagement by the user. It is simply a technical solution to the problem of maximizing the efficient usage of the computational resources available to the network in a scalable manner. Markets, after all, are arguably humanity’s most practical method of distributing resources among populations beyond the bounds of a small tribe.
Computing Price
The price of each resource will be based on the current level of the stockpile. As the stockpile decreases, the price of that resource (in terms of RCs) increases. In other words, as the stockpile goes down, accounts will have to pay more RCs to use the remaining resources.
Cost in RCs won’t translate into anything like a price increase in terms of STEEM or USD, due to the fact that RCs are non-transferrable. The goal is only to leverage a market system to ration resources, not to create speculative opportunities or manufacture another token that can be used to purchase goods or services.
Computing Cost
For each transaction, the System will statelessly compute a value and exchange rate in RC for resources like CPU megacycles, state memory, and history size.
From the github PR:
If CPU cycles cost 5 RC / megacycle, state memory costs 8 RC / byte, and history size costs 4 RC / byte, a transaction which takes 2 megacycles, creates 50 bytes of state, and has a 150 byte transaction size will cost 25 + 508 + 150*4 = 1010 RC.
Technical Details
For those interested in the technical details behind the new RC Bandwidth system, more information can be found in our wiki article on RC-Bandwidth-Parameters.
Summary
These changes will lead to a far more pleasant, more transparent, and more predictable user experience across all Steem applications, and will more accurately allocate network resources by leveraging an efficient market mechanism.
Discussion Welcome
The RC System is only one aspect of HF20 that we have been hard at work on. Because it is a significant change and a dramatic improvement, we wanted to take this opportunity to thoroughly explore this issue with the community.
If you’d like to learn more about the RC System, or if you have any questions relating to it, please leave them in the comments section below.
Thank you for reading.
Steemit Blockchain Team
Overall this looks like a good approach but if you are doing this level of revamp I would take a closer look at how you handle state bytes. Transactions which reduce net state usage should have zero or negative cost for state (negative cost would mean crediting RCs back to the user), and transactions which use state for a longer period of time should cost more than those which use it for a short period of time. Permanent state should perhaps be a separate resource pool of its own and only grow at a very slow rate (making it relatively expensive to use)
State is not really something which can simply grow at a set-rate per block, unless the rate is extremely low (comparable to the rate of hardware technology improvements). Setting a per block rate this low without some allowance for reduction in state usage will probably constrain blockchain utility greatly, and setting the limit too high will open paths for cheap denial of service attacks.
State limit is more of a static parameter (as a function of hardware realities) should perhaps be set by witness vote or community consensus (hard fork). Approaching the state limit should increase the price of adding any state (not per block) and likewise the benefit of reducing state usage.
In any case, this proposed model is certainly an improvement over the status quo which attempts to price everything (including both temporary and permanent state) into bandwidth.
Will the change in rewards be retroactive?
If I just got 2 big votes immediately after posting comments, and HF20 will be enacted in less than 7 days, will it mean that the votes that I got will be returned to the reward pool?
What is the 15 minutes instead of 30 minutes and what is a reverse auction?
I don't know if it would be retroactive, but I expect so. Probably payouts that occur after the fork will use the new rules. Doing otherwise would complicate the system significantly for a one time event and is probably not a good idea. The fork will be scheduled well in advance so everyone should know what rules will apply to upcoming payouts.
The 15 minutes instead of 30 minutes as well as the term reverse auction refers to the time period that early votes are penalized. Currently it is 30 minutes (100% penalty on immediate votes, 50% penalty at 15 minutes, 0% penalty at 30 minutes) and after the fork it is to be changed to 15 minutes.
I suggest bundling witness vote decay or expiry in the next HF.
The competition for the top 20 ranks is almost completely stalled.
There's no competition. Decay had been discussed for HF16, but I believe it has security risks involved. Either way, it needs some solution, witnesses are doing nothing for Steem right now.
I would also recommend cutting the witness voting to 5 witnesses. There's no reason to vote for 30 witnesses; it'll force people to think and vote for the very best witnesses.
Cutting to 5-10 votes would also limit collusion/friend possibilities to lock up the top 20-30 positions.
I agree with you almost completely about the witnesses.
Will the change in rewards be retroactive?
If I just got 2 big votes immediately after posting comments, and HF20 will be enacted in less than 7 days, will it mean that the votes that I got will be returned to the reward pool?
What is the 15 minutes instead of 30 minutes and what is a reverse auction?
I'd like to see a similar concept implemented into reputation scores to actually make reputation an actual viable concept
I very strongly agree. IRL there is nothing more important than rep, and the gamability of rep on Steemit poorly reflects it's value to society. We see the mechanisms for gaming rep IRL as corruption, and the consequences of gamification of reputation can be existential, both to folks that suffer according it to those that gain it absent merit, and those that are found to abuse it.
Reputation is far more valuable to society than it's current implementation on Steemit recognizes, and potentiating a useful reputation mechanism will increase the utility of Steemit to society.
Agreed.
I second this
Throwing in my "me too" here. This is one of the most important things that can be done for Steem right now, in my opinion.
And yet there is still no formula to allow free account registrations without manual approval (SMS verification): The #1 barrier for new users wanting to join STEEM and DTube.
It's a necessary step for proper scaling on the blockchain. Thank you for the weekly update, they have been greatly appreciated.
http://a1leathershop.com/
This has indeed been something that often has been very confusing for new users. While the previous balancing worked (though it sometimes really did get hammered), the idea of a more tangible, easily understood system is a good step to empowering these users to make better choices — based on understanding their own abilities, instead of smashing a button and praying the numbers swing ever in their favour.
This is something I spend a lot of time explaining to users new and old, and to the skeptics out there. I appreciate the move towards something more predictable that will help people better actually understand the technology they enjoy. Cheers.
Good to know that RC (Resource Credit) and Bandwidth are related from Hard Fork 20.
Last week we started the roll out of steemhost.com. Steemhost shows the blog list of individuals (or groups) but redirects people to steemit or busy to view (and vote) the specific post.
In addition to being able to include the content of specific people on the site, it also adds the element of banner ads advertising. What is important about this is that the advertisements are purchased with steem which should cause upward pressure on its price.
Part of what will motivate steemians to administer their own pages and steem visitors to view those sites are that they are rewarded in steem power delegation.
To view an example of a page where two bloggers are on the same page is
https://steemhost.com/@steemitblog
I set both @steemitblog and @ned to act as contributors to this page.
If you are a visitor, you need to log in with steemconnect to be rewarded.
"Error when processing request!" awesome :D
Could you confirm which browser you are using? Were you attempting the login (it uses steemconnect to only identify who you were. Thanks for the error checking.
i'm using firefox with some security addons i didn't even try to do some thing just clicking the url you provided.
i do use add blockers and cookie blockers and the Noscript add on but still 99% of the website can handle that these days
I think I see what the problem is. When you are logged into the public pages they add some additional feature (allowing you to bookmark pages, add additional hosts etc). It also allows you to be rewarded on your click and views. Normally if you are not logged in, it passes all the rewards to the host of the page. While there is no problem with Chrome, the best that I can do is pass the message on to the programmer. We do have a couple of cookies ... mainly associated with remembering who you are. We have some tracking functions ... in order to record what who has visited which pages and when, to prevent people from tapping the F5 key.
There should be a banner ad just above the search button. Possibly your ad blocker doesn't like something if that section is missing.
We have identified that Microsoft Edge doesn't always cooperate but that is on the steemconnect pages ... so it is out of our hands. I have sent this on to tech support so in a day or so, I will post something when it is resolved.
the add blocker i use is : ublock origin ,
the tracker blocker is : Privacy badger and / or ghostery .
it's firefox 60.02 (64bits) that i used for the link
It might be fixed now. One programmer had applied ssl to the site while another hadn't updated webconfig. Firefox is a little more sensitive. Meanwhile Microsoft Edge is just annoying.
i never use edge , it's bad enough that i need windows to play my games .
It sounds to me that administrators and developers will be able to schedule when they undertake certain tasks. For example, if they send out a transaction to all their members, they can do it at a time when RCs are cheap to preserve the application's users to conduct other activitie son the blockchain.
Am I understanding correctly?
I think this is pretty much it, just how electricity is cheaper at some places during daytime because people are out for work and its more expensive at night because people have ACs and heaters on. There will be times of the day when it will be cheaper to send batches of transactions when the network is having less use... It will be tricky to figure out since Steem has people from all over the globe but its possible.
It seems to me that this will make for a more familiar experience. Currently, when it comes to times of high blockchain usage, the "You don't have enough bandwidth" is a very unfamiliar problem for most users.
In a lot of freemium apps these days, you can do a particular number of transactions each day for free but you have to buy the premium version if you want anything more than that.
I think that the act of familiarizing the process like this will make for a better user experience because they'll "get it" much more easily.
News and Updates like this should be pinned at the top for at least 24 hours.