Did We Learn Any Lessons From HF20?

---

Let’s not mince words or beat around the bush...

Hard Fork 20 has not been a “success.”

Yes, the hard fork was accepted and there seemed to be no trouble updating to version 0.20.2. However, immediately after updating, the actual effect on nearly all users of the Steem blockchain was disastrous. Twice in just over a week, the blockchain was essentially rendered useless for 12+ hours, the second time lasting about 24 hours.

Both instances could have been avoided. If not avoided entirely, effective downtime could have been mitigated or at least reasonably expected and dealt with accordingly.

Regarding the actual update to version 0.20.2, we were given an 11th-hour admission by the Steemit, Inc. communication team that they essentially had no idea what was going to happen once HF20 was live. It shouldn’t be surprising to know that the uncertainty likely stemmed from the fact that the negative RC balance problem which caused the user issues was known during the testing period.

Despite knowing about the negative RC balance problem during testing, the planned fork went ahead as scheduled.

Coupled with this irresponsibility was the fact that both @ned and the @steemitblog account publicly declared that the hard fork was a “success” – while nearly the entire user base was left unable to comment to the contrary and looked on in absolute amazement at the wholesale detachment from reality by Steemit, Inc.

This is not the first time that a hard fork has left users in a state of disbelief – or even the first time that a near-complete reset of blockchain functions/abilities/rewards has occurred. We witnessed a nearly identical result after HF18 went live, where post rewards were reset to zero...without warning or the least bit of preparation. There was a “Whoops! Sorry, everyone!” from the Steemit, Inc. team and the usual excuses that we have all unfortunately grown accustomed to accepting.

Without getting too far into all of the arguments again, I just want to say...

We deserve better.

“Why didn’t the witnesses prevent this???”

I’ve seen this asked everywhere the last few days. And I don’t really have a great answer to the question, but I do have some answers, so I’ll try my best to communicate what could have been done, what should have been done, and what actually happens.

Yes, witnesses are to blame, but they are not solely to blame and not all witnesses share the same burden of blame, if they share any at all. What I mean by this is – there is a pecking order for witness influence and how hard forks are proposed and accepted. Here’s my best take on how things really work, according to my own observations for over two years and based on conversations and feedback from other witnesses.

The top-20 witnesses – by design – clearly hold all of the cards when it comes to hard forks. Hard forks require acceptance from a super-majority of the top-20 witnesses in order for the proposed fork to be accepted as the new version of the Steem blockchain going forward. If the top-20 either rejects or accepts the fork, the rest of the witnesses must then operate their nodes accordingly on the old version (if rejected) or the new version (if accepted). If they are not operating on the proper version, then their witness node is disabled and they do not sign blocks.

So regardless of whether or not a witness is competent enough to review and test code, if they are not in the top-20, then their vote on approving or rejecting a hard fork essentially does not matter.

Because the top-20 witnesses are theoretically supposed to be the people/organizations within the Steem community with the highest level of support and confidence to do their job, and because of the responsibilities bestowed upon them to do their job and to do it properly, these witnesses are compensated much more than the rest of the witnesses in the community.

With that and with their ability to reject or accept hard forks as a collective group kept in mind, it is wholly incumbent upon these top-20 witnesses to seriously and thoroughly review and to robustly test any protocol changes proposed as a hard fork. After all, it is the sake of the network that depends on their actions and decisions during these processes – the very network that pays them and where they are presumably invested themselves.

From all accounts that I’ve seen regarding HF20, this review and testing by our top witnesses largely did not occur. Despite not testing the new code, all of them updated to version 0.20.2 by Tuesday’s fork date.

But it’s at this point that we need to discuss some extenuating circumstances.

As pointed out by myself and many others over at least the past year and a half, the proposed protocol changes in the recent hard forks (17-20) were mostly too large to adequately review and there was not a proper testing grounds for the new code. The former issue has been mentioned directly to Steemit, Inc. many times. They continually propose massive amounts of protocol changes in hard forks, and witnesses – even if they could review it because of their knowledge of C++ code – could not possibly go line-by-line in a reasonable amount of time, then also run formal tests afterwards to an adequate degree that would allow them to make an informed decision about whether or not the code is safe for acceptance.

Granted, with the rewards that the top-20 witnesses have received and do receive, they could hire some help to review and test. However, the better solution would be to simply reduce the number of simultaneous changes and the size of them as much as possible. If that were to happen, then there would be absolutely no excuse for not reviewing and testing any new code.

But on this front, the repeated requests from witnesses to reduce the size and scope of hard forks has fallen on Steemit, Inc.’s deaf ears.

So what can be done by witnesses when these types of hard forks are proposed?

That’s an easy answer: Reject them.

Reject the hard fork as proposed and ask for a “clean fork” that can be both easily reviewed and tested, and/or that represents an immediate need, and/or that involves one major protocol change at a time. Yes, there may be changes in a large hard fork proposal that will improve the blockchain and user experience, but we should not forgo our duties as witnesses simply because we’re afraid to say, “No.”

And this brings us to another problem for witnesses: Pressure to tow the line.

I’m not saying that this happens with all witnesses, but there is a culture of cheerleading and shunning that occurs in this community and has occurred since day one. It has permeated every aspect of the platform, including – and especially – among witnesses. If you are critical of Steemit, Inc. or popular blockchain protocols (regardless of whether or not you believe that either are actually beneficial to the blockchain and community), then you can quickly find yourself on the outside looking in.

But the worst part of this culture is that bad ideas, faulty code, and poor behavior is often blindly supported and accepted because criticism is not handled well by those being criticized and because those who criticize are targeted and shunned by many different people and groups within the community...including by Steemit, Inc. themselves. It doesn’t even matter if the criticism is articulated well, if it’s pointed/accurate, and if it’s being done in the best interest of the Steem blockchain and its users/investors.

Regardless of whether or not you like the people who are critical, it’s no reason to support and accept those bad ideas, faulty code, and poor behavior. Even if it’s not intentional, this support and acceptance still happens far too often.

For those who are critical and spend a lot of time articulating their positions, if you say the wrong thing to the wrong people or if they see that you’re not supportive of their plans, then you will not be supported, you will be denied access to discussion groups where critical information and updates are shared, and you are pretty much guaranteed to never see the top-20 as a witness. Again – looking out for the best interest of the blockchain and its users, instead of a corporate entity’s best interests, may leave you working your ass off for pennies.

The fear of losing a top witness position is certainly enough to keep people “in line” or the incentive to reach one of those spots is certainly enough to encourage witnesses to “get in” it. It is partly because of this reason that hard forks are almost never examined critically and especially publicly by witnesses. But it’s not the only reason.

This leads us to another serious problem with many of our witnesses, the ones at the very top notwithstanding: Many do not understand enough about the blockchain (and coding), economics (and behavior), and social media to make informed decisions.

Some are skilled with C++ coding (but very few). Some know a few things about economics. Some know a few things about social media. But how many know enough about all of these things in order to make rational, informed decisions regarding the dynamics of each proposed protocol change? It’s not even a question about code review and testing. It’s a question about whether or not the concepts and the rationale for any proposed changes actually makes sense within the context of the Steem blockchain and the general socio-economic vision for it.

I’m definitely not saying that everyone needs to be well-versed in socio-economics in order to be a witness. However, it would be nice to see an effort by witnesses to at least better understand basic concepts of economics and monetary policy in order to properly manage things like Steem Dollars, APR, and bias. Many of our witnesses never even bother setting these parameters...or know their purpose.

I’m also not saying that every witness ought to be a professional behavioral psychologist. But it would help if they better understood incentivized behavior...if they had a better understanding of things that shape behavior, like antecedents and consequences, so that when behavioral changes are discussed (as with our latest hard fork), it would allow for some careful consideration of the protocol changes in question.

The Steem protocols are part of a complex system that includes making a lot of assumptions about user behavior and incentives. If we have no idea about things like profit motives and interest rates, or what a time sink is, then it’s going to be exponentially more difficult to comprehend protocol changes and witness parameters where these concepts are already implied.

So, all of that being said...

In the end, we’re left with a far less-than-ideal culture that largely punishes critical voices while encouraging and rewarding yes-men, excessively compensates the unskilled, inexperienced, and/or unmotivated, and allows and often excuses the same repeated mistakes from the same people/entity.

It’s no surprise then that hard forks are proposed, accepted, and fail to deliver on promises...or become implementation nightmares. This happens for all of the aforementioned reasons – but it shouldn’t and doesn’t have to be that way.

I would encourage all users – witnesses, whales, and everyone else – to please give more consideration to your witness votes. Take a minute to look around, ask around, and contact individual witnesses directly to find out more about us and what we are currently doing on/with/for the Steem blockchain. If you’re not satisfied with the way things are going, then you are the ones with the ability to change direction.

(And that includes you, @freedom. You are the single largest voter of witnesses. I would hope that you take that vote seriously.)

This last hard fork should not have been accepted. It was simply too large, included too many moving parts, was clearly under-tested, and the dev team and testers had at least one major concern about it. Witnesses dropped the ball by accepting the fork and the Steemit, Inc. dev team was not open enough with witnesses and the community about what was likely to occur...again.

We are now on our fourth iteration of HF20 but it has only been officially in production for three days. And we can’t even be sure that another patch or two won’t be necessary soon.

This is unacceptable.

My promise to you as a Steem witness.

I recently mentioned some of this on a post from @remlaps, but I want to include it here, for the record.

If I were to find myself as a top-20 witness, I can promise you that I will do the following:

In addition to my usual availability and engagement with the community and my general reviews of hard fork proposals, I will...

1. Operate a public RPC node.
2. Have the code for all proposed protocol changes professionally audited.
3. Direct my team to participate in any official testnet for new code.

Regarding specific proposals for protocol changes, I will accept them based on whether or not...

1. There is a problem in need of a solution – OR – There is a protocol change that is persuasively an improvement upon existing code.
2. The rationale for the code is coherent with the existing set of protocols.
3. The proposal is presented in a size that is easily manageable/auditable.
4. The code has been professionally audited.
5. The code has been adequately/robustly tested.

If any proposals do not meet this criteria, then my default position will be to reject the proposal.

I would hope that other witnesses take a similar stance on protocol changes and actually stick to it. I understand the eagerness to try new things or to “move the blockchain forward,” but that does not necessarily mean that we need to constantly tinker with the economics of the platform. Stability can be just as attractive and productive than continual change. Forking the blockchain should not be so easy to accomplish. It should not be a given.

Share your concerns and give us feedback.

I know some of you out there will just complain that I’m being critical once again. I understand that.

I also don’t care.

We need more critical voices – more people willing to go against the status quo. It is the critical voice that makes you more adept at defending your ideas. It’s the people looking for and pointing out errors that pushes you to be more careful to not make them. You don’t harden your project by cheerleading, rubberstamping, and shutting out criticism. You don’t learn from people telling you that you’re never wrong.

So be critical. Find the things that you don’t like. Express your displeasure. Communicate your concerns to those people that you believe will actually listen and may be able to help.

But don’t just be critical. Highlight the things that you believe are good…and ask for more of them or propose ways to improve upon them.

As stakeholders, this blockchain is the responsibility of all of us. It’s time that we start acting like it instead of trying to prematurely kill it…either by fattening it up to morbid obesity through cake-eating and complacency, or by killing the proverbial golden egg-layer.

It's in our own hands.

---

Vote for

Block-change you can believe in!