Complexity

As someone who does software engineering as a profession, there are two things that make the engineering process "uncomfortable". They are uncertainty and complexity. Uncertainty will always be there and something that you just have to fight through. But complexity is different. Complexity is something that can be tamed, but it's not an easy beast to tame.

Why does complexity suck? It makes the overall system harder to reason about. It makes the overall system harder to learn. It increases the surface area where different potential errors can pop up and hide. The more complex a software system becomes, the more chaotic potential that system has. This concept is called software entropy. As an engineer, I do not find entropy to be desirable.

Nobody wants their product to exhibit behavior that is outside the scope of the expected behavior. Nobody wants to find bugs in their code. Because this introduces uncertainty, that other thing that I don't really care for. And with all cases of uncertainty, research and exploration are required to remove that uncertainty. And the process of finding bugs is tedious as hell.

With a simple system, bugs are pretty easy to find since there is only a limited number of possible states and transitions between those states. Given sufficient knowledge, one can trivially reason what is going wrong. With a complex system, you first must traverse the mechanisms and isolate the problem. Hopefully, the issue is a simple fix and not the symptom of a much larger problem.

But let's say that you uncover the dirty underbelly of the behemoth sitting there in the code. The simple patch is always tempting, but if we're being honest with ourselves, it often doesn't address the actual problem, but simply addresses the single erroneous case. As a college student, I did a lot of this in different projects I worked on. It worked because I could throw those projects away after the end of the semester. But most of the time you don't want to and can't throw away production code.

A lot of times, the naive approach is to simply introduce more technical debt to your software system. Technical debt is the concept of accepting the future cost of harder work for a simpler (and often dirtier) solution in the present. As more technical debt accumulates, the more interest (or rework) you have to pay for in the future.

Technical debt and complexity go hand and hand. Often complexity is a result of poor, sloppy, or inexperienced programming. You can either take the time to carefully design a system reducing the technical debt (and often the complexity) or you can jump into the problem and code and experiment your way into a solution.

Now I might have given the impression that technical debt is an inherently bad thing, but it's often a necessity reality. Given constraints like tight deadlines and huge requirements specifications, you might have to accept some technical debt in order to deliver the product to your customer. Also experimentation and writing bad code is often the best way to finding a better solution, so even if you do spend a lot of time planning, you might not know until later that your solution to that problem is less than ideal given the circumstances. Everyone would like to write bug-free code, but that's simply never going to realistically happen in a reasonable amount of time.

So why talk about complexity?

Given that I have recently restarted a project at work partially on the basis that some of the logic was too hard to reason about for some folks, I just want to talk about something to think about especially for those folks developing the future products of tomorrow. Because a code base can reach a certain level of technical debt where it makes sense to essentially declare the code bankrupt and restart the project fresh with the new perspective gained. A lot of prototyping works in this way.

But while declaring bankruptcy might make sense for some projects looking to rebuild something that is more adaptable to adding more features or it easier to test and verify correctness, it is certainly not appealing for other projects. Like blockchains.
'
Because one doesn't simply rebuild a blockchain. There's a lot at stake. While most software systems are cold-blooded machines, blockchains are a little more like something actually living. You have to be much more surgical. Because once the blockchain starts, the margin of error to fix and improve the blockchain is amplified because of certain properties that blockchains claim to satisfy about the data on the blockchain.

While organizations can simply transplant a database from one front-end to another and make changes to that database as they please in order to simplify design choices and improve the overall performance of the system, such behind-the-scenes adjustments don't work with blockchains. Transparency and censorship resistance get in the way.

This leads to a situation where blockchains can accept technical debt easily (through hardforks and updates) but struggle to find ways to repay that debt. While you could "restart" a blockchain, you are essentially starting from scratch, otherwise you would be violating some of the core properties of the blockchain that make blockchain worthwhile in the first place.

To summarize this point, blockchains are limited in the way they can be modified to fix issues because they are blockchains. If you could change them like databases, they wouldn't be blockchains, they would be databases masquerading as blockchains.

In my opinion, Ethereum is the king of blockchain technical debt mountain. I have almost zero faith in developers being able to transition from a Proof-of-Work system to a Proof-of-Stake system without taking massive amounts of complexity and technical debt. You would need a set of omniscient developers to do otherwise, and I simply don't believe in omniscient developers. And that complexity is going to make the system harder to maintain and harder to continue building upon.

Of course any conversation on technical debt that is occurring on the Steem blockchain, should probably talk about Steem, so we'll end with Steem.

I have massive amounts of skepticism when it comes to the upcoming hardfork for SMTs. I have seen complexity wreck havoc in prior hardforks. And that complexity multiplies with more tokens, more states, and a code base that is harder for developers to understand. When it comes to reasoning about adding further improvements to the system, there are so many different variables and mechanics to consider that it becomes very hard not only to model user behavior but to ensure that the system as a whole is working correctly as intended by its design.

Whenever I hear of new and interesting ideas that could be applied to the blockchain, I get the feeling that it might be easier to start over rather than continue to trudge through the complexity. Complexity is hard. And I feel that complexity bleeds over into the user domain and it scares potential users away as well as providing a learning curve for currently existing users.

But this issue isn't just a Steem issue, I think it applies to all blockchains. However, blockchains which have more rigid and strict governance models and fewer forks and upgrades tend to have less technical debt moving forward as there are less opportunities to generate technical debt. And so far, Steem has lent itself to being very progressive when it comes to making code changes and modifications. And I have to give a lot of credit to the developers, it's a very complex system that appears to work fairly decently, but I worry about Steem's potential agility and adaptability moving forward.

One could argue that Steem is a very successful prototype (or experiment) blockchain that could educate those creating a simpler and more elegant blockchain. One that doesn't suffer from having to adjust and adapt to all the prior mistakes and adjustments that have been made over the past three and half years. If things don't turn around (in terms of user growth and adoption), that may be the best path forward in terms of complexity and "debt-free" development.

When facing overwhelming complexity, one must recognize when and how an organization or group needs to pivot.

In my job, we decided to pivot towards a different architecture that was already proven rather than potentially accept a lot of technical debt building components that the team had little experience with.

In Steem, almost everyone agreed that something needed to change after the price fell from all-time highs and adoption began to decline. Changes were made and some complexity was added. More changes are in the pipeline and more complexity will be added. It will be interesting to see if these systems will either attract new talent to continue their development or inspire new talent to find a simpler, more elegant solution.