Help us test new performance optimizations for steemit.com!
We've been working hard on improving our infrastructure over the last 6-7 months and many improvements have already been made. In early February we migrated to AWS and now run our entire stack with Elastic Beanstalk in docker containers. Everything is now infinitely scalable and has moved from a traditional best-efforts hosting solution to high availability infrastructure. Our traffic has multiplied at an exponential rate. But, there's still room for improvement.
Join us in testing these optimizations!
We'd like you to join us in helping test a new feature which will bring some great performance optimizations to steemit.com. In the interest of catching potential bugs before they get rolled out to production, we'd like to get as much real-world exposure to these changes as possible. You can now experience these changes live at https://steemitstage.com.
Keys
Our staging environment includes the same security measures taken for steemit.com and is considered pre-production, it's designed to be as close to the 'real' steemit.com as possible in order to fully vet new code before it makes it to steemit.com. If you're still worried about using your keys, you're welcome to just use your posting key. Even if you don't login, just browsing the site without logging in and providing feedback to us is still useful.
What's changed?
We're implementing a new custom designed proxy and caching layer in front of steemd, which is the back end blockchain service that the front-end website talks to in order to perform all actions on steemit.com. Most read-only requests to steemd will now be cached in an elasticache/redis database and served directly from it instead of having to communicate with a steemd cluster for requests that are already cached. This will most likely improve performance of general site usage dramatically.
In addition, historically websocket technology has been used by the front-end in order to speak to steemd. Websockets certainly have their advantages over traditional HTTP, but when it comes to large-scale infrastructure it's much more difficult to properly load balance and distribute active connections. Further, traditional HTTP/JSONRPC calls are more common and generally easier for 3rd party developers to use and develop with. Because of these reasons, our proxy/caching layer will still talk to steemd using websockets but everything else (the front-end plus new services being developed) will talk to the proxy/caching layer now using HTTP/JSONRPC. In order to make this possible some new updates were made to the steem-js javascript library to be able to utilize HTTP/JSONRPC in condenser (front-end app that is steemit.com).
If you're still curious, since our code is open source, you're always welcome to look over it yourselves. The proxy/caching layer is called 'jussi' and is available at https://github.com/steemit/jussi. Steem-js is at https://github.com/steemit/steem-js and the front end (steemit.com) is at https://github.com/steemit/condenser.
Feedback
Please leave any feedback about your experience using steemitstage.com on this thread. Visually, the site will look identical - these are changes that are invisible to you visually, but you should feel an improvement in speed while browsing around and utilizing the site. If anything is broken, please let us know. You're also welcome to reach out to me (@justinw) on steemit.chat if you'd like.
Justin Welch (@justinw - Steemit DevOps Engineer)
disclaimer: steemitstage.com is not setup for high availability and could be broken or completely down at times
This type of transparency is excellent.
Is there any benefit for users to be testers on steemitstage.com? Maybe some kind of acknowlegement would get you more testers.
Check out the performance tools in the Firefox Developer Edition. They are very useful.
Thanks! I'm not sure what you mean by 'benefit' and 'acknowledgement' in this context. Can you elaborate?
Sure. I mean would the the users of steemitstage.com earn rewards or be acknowledged in another way for reporting issues, bugs, suggestions, etc. It could be as simple as listing the contributors?
benefit is blessing the platform we love, and advance knowledge of new features before they go live.... it sounds pretty beneficial
Just trying it out right now, and the difference in speed is DEFINITELY noticeable! All the basics I tried out so far are working good, so... fingers crossed!
okay, I commented on a recent article, and the comment did go through... but the page kinda hanged without updating itself when the comment got posted.
Thanks for the feedback - I haven't experienced any lag in comments yet. Has it happened again?
just made a new comment, and it went up smoothly
Seems way faster, but I had a problem replying to a comment as well. "This page doesn't exist" was the response.
Thanks for communicating this and inviting testers. Great idea!
All this HF and optimizing will pay off in the long run .. Steem to the MOON
well done ur trying to ahead of performance issues. I will be blogging and doing my own testing to help out. I am a 15 years software developer in test. Thank you Thank
Thanks for the information sharing. I started some developments on current api, but these new developments will enhance further. I also use cloud in MS Azure. Now I'm using json-rpc api to scan changes, but block processing will make my solution much faster.
Sounds awesome! I think my only concern is you guys becoming too dependent on some of AWS's proprietary features, although I'm sure you've considered everything before making the move. When I joined this site, I thought it wouldnt have a lot of growth, seeing as it is a niche inside a niche, but I was pleasantly surprised when I saw news of 250000 users within months of 200k. For that kind of growth, good infrastructure is doubly important, so I wish you luck in maintaining and developing the site!
Fear not; we don't rely on any proprietary features of AWS, and we don't intend to. :)
The setup process for condenser is tedious and the vagrant setup is not working. I think if You can fix it more developers will start spending time on the project. I would like to improve the follower page to begin with
I submitted a PR today to get the README up-to-date. It includes instructions for using docker to both build and run condenser, in addition to corrections for building/running it the 'traditional' way. The
Vagrantfile
should be easily modifiable to mimic theDockerfile
. https://github.com/steemit/condenser/blob/readmeupdate/README.mdAwesome. Good to know. I will try it out again this week.
Nice post!
Sure