Performance and Scalability Updates
Over the past few months, the Steemit development team has been working hard on several major performance improvements to the condenser code (steemit.com) as well as the steemd blockchain API. These developments include improved speed, efficiency, and reliability of the blockchain as well as the Steemit website. This post is to update you all on our progress.
Jussi
Historically, all calls to steemit.com have retrieved data directly from the blockchain, which was not the most efficient way to process requests. Therefore, we developed a new backend service called jussi which creates a layer between the website and blockchain (as well as the other backend services we are developing), allowing queries to be cached and processed in a much more effective way. It also creates an additional layer of resilience against attacks.
These changes have been deployed to production for several days now, and we have seen a dramatic improvement in the efficiency of read requests from the Steemit website to the blockchain to retrieve data. Jussi is built to leverage additional opportunities for improving efficiency by increasing modularity through the new AppBase backend architecture.
AppBase
Originally, a single steemd blockchain instance could use multiple threads to process different tasks, but only a single thread would handle API requests. This created a bottleneck on the amount of queries a single steemd node could process at once, as they were processed one by one on the API thread.
AppBase is a new architecture that creates distinct “modules” which enable separate processor threads to process different API requests at the same time, which is significantly more efficient. Modularizing steemd enables it to take full advantage of the parallel nature of modern processors. Many additional improvements can be built on top of this, but the foundation for a modular blockchain has now been laid.
The development team has been working around the clock to get AppBase ready for production, and we plan to launch it very soon.
Peer-to-Peer (P2P) network changes
The P2P code is what keeps all of the steemd blockchain nodes in sync. Efficiently transmitting operations (such as posts, votes, etc.) to other nodes in the network allows transactions sent from websites like Steemit to be quickly processed and included in one of the blocks by the witnesses before the transaction expires.
We have completed multiple performance improvements to the peer-to-peer communication code, and we may make additional improvements to this system over the coming months.
Communication
Many of these improvements take significant time and resources to investigate, design, develop, test, and deploy. We understand that you all want to be kept up to date with the latest progress, but there are many competing factors when providing the community with frequent updates. Our top priority remains improving the platform as rapidly as possible, and we do not want to distract from that. We also want to avoid drawing unnecessary attention to anything that a potential attacker could exploit until we have fully evaluated the situation.
The Steemit development team takes any issue that impacts the usability of the website very seriously. We have multiple performance monitors in place, which are running on the website 24/7, and we are alerted instantly if the site ever goes down. We track many metrics that inform us of how long things are taking as users are interacting with the website. We know when there are issues. We also appreciate the many alerts that we receive from users when they notice something that could be wrong. We may not always communicate publicly about every challenge we face, but you can rest assured that we are actively working to resolve all issues as quickly as possible.
These improvements we are making have us very excited about the future of the platform because they will enable us to handle the exponential user growth that we are expecting. We appreciate your patience as we continue to scale and build a world-class website.
Steem on,
Team Steemit
Thank you for communicating.
I appreciate how you are working around the clock, but I also expect communication when something that was working stops working. For the past couple of weeks, for many including me, Steemit has become almost unusable. Before that, it was functional. Something changed, but I don't know what because there's no technical communication about the issues. If those issues are so sensitive that communicating about them would provide real attack vectors, that's security by obscurity which is concerning.
I'm glad you have metrics in place giving you visibility on the time it takes for transactions from the browser to hit your servers. Does that include logging of browser errors? Due to the complexities of JavaScript-based front ends, I imagine tracking the things which time out and never make it to your server is quite difficult. If you do have that visibility, but it hasn't been a priority to discuss until now, that's also quite frustrating. Communication takes time away from development, sure, but it's critical for building trust with this community. I believe it's really important, not something to get to two weeks later. Many share my position on the value of communication and why it should be prioritized higher than it has been in the past. IMO, this post has a theme of "We're working on it" but not much of a "We're sorry for letting you down." Maybe you don't feel an apology is needed, but I think it would have been helpful here. The frustration many of us have had trying to use the site is quite high and though you may know that intellectually, the data shows you can't fully know it experientially. Or if you can, I'm not sure how.
I ran some SQL queries on the steem database to see how often the Steemit, inc team has published comments and posts since the DDoS attack. They are not encouraging numbers. As a SaaS software developer myself, I really strongly believe in eating your own dog food. I hope the team prioritizes this to fully feel what users are feeling along with seeing the data and the metrics in reports. As I've experienced bugs using my own software, I've found it motivates me quickly to prioritize and fix them.
Illustration: Have you ever been on a wifi connection that kind of works but not really? That's really, really frustrating. It's hard to explain how frustrating to people unless they've experienced it. When I talk about the emotional frustration and feelings of the community, it's like that. I hope the steemit team decides to prioritize using the site more often to get that same experiential knowledge.
I imagine you're all under a lot of stress to get things done prior to Steemfest2. I hope you get plenty of rest and meet and exceed your goals. Many of us believe in you, while many others have lost faith and are vocal about their bitterness. My previous post on this topic wasn't meant to troll or be accusatory as I was trying to speak in terms of facts and bring helpful solutions. I hope this comment will be received well. I really do want you to succeed, and I want the community's experiences to be fully validated and appreciated.
Thank you. I look forward to meeting you all in Lisbon.
Again, thank you for communicating. I hope it continues as we see improvements in site performance.
Please remember Mr Luke Stokes it is important to have table manners even when eating your own dog food.
I hope you can see I'm putting in a real effort into communicating seriously and honestly about the frustration I see in this community directed at Steemit, inc. If I've been rude or offensive, I apologize as that's not my intention. If you think I have been, please address it directly because your response here doesn't make sense to me.
I was hoping for a more serious reply.
@ned, do you think eating your own dog food is important?
btw...Luke, I felt that your comment was done in completely good spirit, and was very on point. I really appreciate everything you had to say. Especially this.
There's absolutely no reason Steemit Inc can't afford to have someone on constant communication. Not doing so is suspiciously incompetent. I'm not trying to be mean, but come on...you have 40% of the STEEM, which is barely short of 100 million USD in highly liquid assets. It's going to cost what $60k a year to get a great communications person?
They don't have to expose attack surfaces in order to be in conversation with the rest of the platform on a daily basis.
Also @Ned, I dropped $60k USD of my own money into Steemit because I believe in it...and NOTHING in the last year has made me feel more hesitant it's future then the way you just responded to Luke Stokes, who was giving you some pretty solid feedback.
@bellyrub loves you!!
Sheesh keep up the great work, it shows you are devoted to steemit!!
This platform will continue to thrive.
@ned, @lukestokes said nothing rude to you. I'd like to know what you took offense to.
I haven’t taken offense. I am pointing out how inefficient, arguably inconsiderate and difficult it is for a platform to respond to incredibly myopic, incessant arguments. Refutation of myopic arguments requires myopic counter arguments and it’s a terrible use of resources. It’s the ultimate squeaky wheel problem and I explain this below.
A new SMT has been created in your name: ExcuseCoin care of SteemIt Inc. (STINC)
Hey @ned Can you please delegate me some power.
Good manners have much to do with emotions, to make them ring true one must truly feel them, not merely exhibit them.
Unprofessional response, clearly missing the point.
Unprofessional response, clearly missing the point.
This looks a lot like an insult, but I hope it isn't.
@ned Fixed up the Downvote for you.
accidental double post, flagging for visibility
And that reply from @ned says it all. We're all an experiment for a bunch of Douschebags...
👍👍👍
I'll try to say this as respectfully as I can...
Perhaps this may partially be attributed to the fact that so many could not get a post or vote through, and after numerous attempts many people likely just gave up.
On the plus side, there are a bunch of alternative condensers (STEEMIT's front-end) that have popped up as backups, so if there are continued problems with STEEMIT's website, at least people have the choice of still interacting with the STEEMIT blockchain via the interface they're already familiar with.
If you do use an alternate condenser however, make sure it's hosted by a community member you trust. And as an additional precaution, it's best to sign in using only your POSTING key.
And with that, please pass the Bordeaux! :)
Exactly! I started here a couple of months ago and I've become thoroughly addicted! Lately, this platform has been unusable!
I'm sure the Devs are working hard to resolve the issues, but is it too much to ask for one "State of the State" post?
thank you
Thank you for the update. I've been anticipating 0.19.x for a while now - hope to run it soon. I see active development on SMT / 0.21.x. However, any update on HF 20?
Everything you have pointed out about Communication is obvious. The important thing, however, is to keep pointing out the obvious. People are emotional and impatient, particularly for a platform they see significant potential in. Although it would seem logical to "shut up and ship", devote every ounce of attention to engineering, this is an aspiration to be a massive, decentralized, open social network, and people come first. Account for ill logic and irrational behaviour. Keep everyone updated, keep sending the same message every week, or any reasonable timespan, keep reminding people that you're working hard on it. Whilst it would be folly to reveal intricate details of your work for an abuser to exploit - and indeed, almost all of your audience doesn't care about these under-the-hood workings - it is most essential that they are reminded time and again that work is under progress. Even if it's a minor update, it's better than silence. Consider silence the devil's playground - emotional people will tap into their wild imaginations and spread FUD every which way. Communicate regularly, control the message.
Needless to say, again, all of this is the bleeding obvious, but I had to hammer it home, like you need to with us.
I completely agree with you here @liberosist! If I may quote my own post from just now...
People don't need specifics. They just need to know developers are aware of issues and are working on fixing them. Here on Steemit, it's hard to know where to get this reassurance, but it's so important to keep your members patient.
Perhaps when Community feature is implemented, we won't need such news flashes to clutter our home screens. Anyone interested can get that news by going to the respective community, I guess. But yeah, we do need some solution for the problem you have pointed.
I think any community site like this could benefit from a specific page with some updates. Especially when things aren't running very smoothly like lately. Ofcourse, we don't need many details mostly. I think any specifics would do fine in a dedicated community like you said :-)
Agree!
This...
Communication is key; Also giving timelines is key! Forget all the details, we are not interested in all the complexities. I get the impression product management is not in place at all; No experience with service evolution definition and planning is in place! I've been managing complex telecom and IT software products for years, from initial design to first delivery as a marketing tool, field test versions, to high performance versions and all the versions in between matching the life cycle of the product. I never ever had to give my customers the messages as expressed in this post. I never ever had to do that, since the next version addressing exponential growth was always addressed in time! FYI, I had to deal with the smallest to the largest customers you can imagine, and in the end all were addressed with the some software architecture and code; But with cleaver design it was possible to keep everybody happy while also the operating costs were according to the budgets of my customers. How can a service in the field for more than a year, still not have the code line with a high performant architecture. And how can a company address us with the statement "we know what is happening, we know when the service crashes" while the service is underperforming for many weeks already?...Fact is: the service is far below minimum expectations from performance point of view. Dealing with an open community, many messages are made public by the community that will harm Steemit and therefore also Steem. There must be a reason why many Alt coins are doing pretty well, even when BTC is flying high, but Steem is devaluated on a daily bases. Start becoming professional in product development and product management. Read about "crossing the chasm" by Geoffrey Moore if you will, it may open your (STINC) eyes.
It's rather easy - and seemingly fashionable currently - to critique development work by Steemit Inc, but the question is - do we really understand the challenges it takes to build a social network and frontend on a blockchain? I certainly don't, and I haven't seen a better alternative or solutions suggested. There's probably a good reason there have been a dozen or more social blockchain platforms announced, yet Steem is still the only working one I'm aware of. There have been platforms announced over 2 years ago now! If there were a direct competitor to Steem that were outperforming it, I'd agree, of course. As it stands now, I'm not even sure a blockchain social platform is a good idea. As flawed though it may be, nothing offers a superior experience than Reddit as a social platform for me, currently. On those grounds, criticism is certainly valid - Steemit is a sub-par social network experience; though de-facto still the best blockchain driven one.
Looking at Github, I see plenty of evidence that the team is working flat out, with new developers added. Occam's razor, they are simply facing difficult problems that take time to solve; rather than incompetent developers lazing around doing nothing. Vitriol doesn't seem to be warranted in this case.
I don't agree that publishing timelines are key for experimental projects like Steem with no real precedent. I believe it was a mistake to list out timelines in the 2017 Roadmap. Predictably, they have missed most of them as other priorities come to light. History has taught us an obvious lesson time and time again - schedules and budgets are never, ever met for innovative work of any kind. There are unforeseen and complex challenges of all sorts that will only be apparent during the development process. Technology corporations know this and keep their cards close to their chest, only announcing products and timelines with manufacturing set to go. (Yes, I'm aware competitive advantage is an important part to this, one that's not relevant to Steem. Still, I hope you see my point.) Products that have been in development for 5 years are announced 3-6 months from release.
As for price, it's completely irrelevant to technology - crypto traders appear to be irrational lunatics. Steem has a poor reputation in the crypto community; there are plenty of other coins which have far superior hype drivers. How many of them have a product that is in active use? Doesn't matter.
Sure I see your points, but still I think my criticism is allowed and correct. I dealt with experimental projects myself as I dealt with products that made my employer market leader by far, meaning nobody was able to do the same since we did it in a different way and outperformed everybody by far. I know all about what, when and how to communicate these things. I know all about how the plan these things and keep the 'customer' engaged even when timelines are not met. And sure, any IT/software project have the tendency to take longer then expected, but that is anticipated and reserved for. I'm in favour of mentioning timelines in roadmaps, and features per release. I'm also not scared to change those, features and timelines, but it all boils down to communication and keeping the customer engaged and knowing how to do that. Communication is horrible, in frequency as well as in content. Frequency is way to low, and content is way to detailed; Thats is were they loose their customers, us, the community who work hard to make a success out of Steemit.
Okay, let's have a set of amorphic promises with no due date and hope someone will do something next year. Or dosen of years.
To have commitment is tough call. Really tough. But it is the only way the team could move ahead.
Agree
We're not customers.
Interesting, what are we then?
BTW, the definition of a customer: "A party that receives or consumes products (goods or services) and has the ability to choose between different products and suppliers." reference: http://www.businessdictionary.com/definition/customer.html
Authors, readers, investors.
Paid beta testers?
Although the Steemit UI states "beta" and STINC didn't take Steemit UI out of beta, doesn't mean we are beta testers.
You realise how much money is going through the system? And how much money is distributed every single day? And how much money STINC owns through its Steemit account and others?
Yes we are authors and readers as these are the activities the service allows us to do. And we are also investors, since we invest in the currency Steem. In addition we are also curators, and partially decide who gets rewarded for their activities. But we are for sure consumers of the Steemit UI and Steem blockchain (as per definition I gave you in earlier comment), since we consume the service STINC develops, making STINC the supplier and us, the users, consumers.
Keep in mind when we walk or drive the streets, we are also consumers of the road we use, wether we directly pay for usage, or indirectly pay for it.
When we for instant are connected to an energy grid and in addition buy solar cells, put them on our roof and supply unused energy back the the energy grid, we change from consumers, to prosumers - meaning we become producers when we supply our solar energy back tot he grid.
Consumer =/= customer.
If I order flowers for my wife, she's the consumer, but I'm the customer.
I'm paying, therefore I'm owed customer service.
I've never paid a cent to STINC, so Ned doesn't owe me customer service.
A customer doesn't have to pay for a service in a direct manner. We actual create (partially) a value for Steem and Steemit UI, we pay with our time spend, with our knowledge we share in the community; With our activities/projects we define and execute using the Steem blockchain and services around it.
Using your explanation, anybody who offers service without a direct payment involved from the user (the customers and consumer) towards the service owner (the supplier) does not have any obligations. Facebook therefore does not have any obligations to its users? Skype doesn't have any obligations to its users? WhatsApp doesn't have any obligations to its users? WeChat doesn't have any obligations to its users? Whole businesses are build around these free services. These service owners are earning a lot of money because of all these free users. With the third industrial revolution, producers and consumers, suppliers and customers definitions and models changed.
Yes, agreed. More information is better.
that would be great and calming if honest and genuine
that would be great and calming if honest and genuine
agreed ;)
We all know you guys are busting tail, but I agree with @liberosist, it's nice to hear from you guys fairly frequently.
Just one little point before I go...
Right at the top of the page... left hand side, under the Steemit logo is the word 'beta'
To me, that says the team are working on it ALL and patience is not only a virtue, it's a necessity.
I'm saddened at how many people are sending barbed comments, little snippy jibes and seem to be hoping for failure. (Not here, on other posts)
I know this platform isn't 100% but I also know this too will pass.
Tempers getting frayed because of deadlines being missed, things not going to plan and other human and computer errors are expected and should be forgiven.
I'm sitting back and waiting for the exciting stuff to arrive because the team has gone quiet and that to me says they're busy.
Yeah, I know... happy happy, joy joy but I've been a beta tester in my time. "Give it here, I'll break it" let the people get on with it, be patient and put up with a few glitches for a little while. I know it's going to be well worth it.
I understand you point and I'd like to feel the same way, but you have to admit there has been a lack of communication.
When I drive to work, I know I'm going to sit in traffic. I endure the slowdown because I know it's there and I cannot avoid it. I accept it.
It's the traffic stoppages that I don't expect or know the reason for that infuriates me. No explanation, no reason, just stoppage.
On projects like this, I always expect a lack of communication because, as I said, people will be getting things sorted out for deadlines and time goes past way too fast when you're under the sword of Damocles of a deadline.
I may be way too laid-back or I may have too much confidence in the Steemit Team but I don't think so.
Like a duck, things may be quiet from our (the user) point of view, but one glimpse under the surface, you can see those legs milling like mad.
Have you ever tested a product that was called beta for 18 months?
Have you ever been involved with a project like Steemit?
This is the first.
But if you're unhappy, I'll be willing to give you the money you paid for your account and you can leave.
Thanks for the offer, but no need for refund.
I brought a bunch of $$ to the platform with boundless initial enthusiasm.
The more I learned the more I became disturbed.
Powering down and getting out as fast as I can. Hope to make it out without too much $$ destroyed.
Good wishes for the future. Everyone goes through a learning experience with first product.
And some get a chance to work on 2nd and 3rd and 4th and eventually get a home run.
Well, this update is much welcomed. Looking over the comments on this thread, it seems that people's responses to the recent situation on Steemit are highly varied. I find it fascinating because it reminds me we are always in control of our attitude in life.
The frustration felt by the Steemit community has been tangible, and I assume the Steemit team is aware of this and shares the sense of urgency and love for the potential of this platorm which is the catalyst for this feeling.
We are very grateful for all your hard work. If I can provide a couple of constructive criticisms to both users of Steemit at large, and then the team:
Users of Steemit:
Please help spread this knowledge. There is never any reason to complain about slowness/failed transactions as you can access the same mechanics instantly on busy.org.
For the team:
I know this was a long message, but I've been waiting for this update for a couple weeks, and I just want to continue the conversation and be a conduit for the continuing improvement of this platform and community. Much gratitude and respect to everyone, both the team and all Steemians! Looking forward to what's in store.
excellent comments, excellent overview and expressed in a very clear and concise and readable way for the average person, like me, to read and understand. thanks @d-pend
Glad it was helpful for you @watertoncafe!
Thank you very much for this update! I'm happy to see you're all working so hard on improvements and I do feel the site is slowly coming out of the the troubles we've had these past few weeks. I couldn't be happier to have such a dedicated team working on keeping this awesome place up and running!
About the communication. I completely understand you can't keep us up to date about all the things that are going on. I think many of us don't really need to know many details to feel at ease. Even a small notification like "We know we are experiencing problems and we're working hard on fixing them." will be enough for most. The biggest issue I think is that most of us aren't really sure where to get this tiny bit of reassurance. To be honest, I received a link to this post, otherwise I probably would have missed this update aswell.
I would love to see a little 'News flash' link in the top-right menu (you know, with the Welcome, FAQ, Currency Market, etc.)
It could contain tiny updates like "Made a few fixes to the site", "Got a new server running", "Updated the code", "Nice weather, huh?" well... anything really to show a sign of life. I think regular tiny updates like that would calm people's nerves a lot and would stop the complaining about communication. Everyone will know where to go for info and everyone can see a sign of life every week or so.
Comment's long enough now! Thanks for all your hard work! :D
Thanks for the feedback! I think that's a great idea!
Community Liaison, Steemit
I'm glad you like the idea. I really hope we'll see something like this in place eventually. Just one clear spot for people to go to for some site updates :-)
Absolutely a great idea. We need this, a kind of notice board. Something others and myself and also you are asking for. I certainly hope we will get this soon.
Thank you. Yes, I hope we'll get something like this soon. We just need a clear place to get our information :-)
Truthfully, 99% of this is over the average layperson's head.
I'm really thankful for posts from fellow-Steemians (@lexiconical comes to mind) who are quick to offer a simpler explanation for informative posts such as this.
Today, I've noticed that my replies are duplicating, with an illusion that it's in the perpetual loop.
Btw, without proof, I wonder if Steemit is under another attack. Today, I powered up using blocktrades, and last check over four hours ago, continue to wait for the coins to post in the SP account.
At least, we can be encouraged that the developers are working hard behind the scenes. I can't imagine what running something this large and growing must be like.
Peace.
It's great to know ppl are working on improvements and not to be negative but..the site's not been working properly for weeks now. This very comment may not even post, I may have to try three times and spend 120 seconds watching it try..is anyone working on this?
Absolutely. We are working to get these issues fixed as quickly as possible.
Community Liaison, Steemit
keep on fighting
Oh man, I had forgotten about this film!! The eyes on the fingers!
Greatest sequel that never happened.
@kevinwong OPEN THE MOUTH.
Good
Thank you very much for the communication! Good to know you are working on improvements and letting people know what is happening.
Did you write this for them? Seems like I have seen this letter before... ;-)
Thank you for Your Good work