[COMPLETE] Maintenance of chainBB - indexes missing
UPDATE: This maintenance is now complete and the forums are fully populated again.
Just as a heads up - the chainBB forums are looking pretty empty because the DB got wiped within the last 8 hours. Not sure how/why, could be that the DB was accessible somehow and someone dropped them, or that the beta environment running within docker just took a dump. I haven't spent a lot of time with the infrastructure yet because this is still one huge test, but this highlights the need for a little of my time towards it.
Luckily, nothing of value is actually stored in these indexes or the database!
I have a huge EC2 instance on standby that needs to sync the blockchain and then rerun the indexer, then just import it over into the beta server, and we'll be right back to where we started. The beta forums should be back and 100% running again within the next few hours.
On a separate topic, I'm hoping to add "Sticky" posts at some point this week, and hopefully add a "Decline Payouts" button to the post creation process. Both of these options would be very well suited for posts like this.
What a troopper! Great work, thanks for the major update on this one. Namaste :)
Why you are not using Steem to store all the information in chainBB? I thought it's already like that in your previous post. There are so many flexibility in using specific tags combined with custom meta data to format and sort content differently.
The user data, posts and replies are all stored in Steem, but I maintain the forum indexes in an external database for raw speed and scalability.
Steem also doesn't offer API's for retrieving content as the forums would need it either. The forums themselves pull based on category (steemd pulls on tags) and then sorts them by when the last response occurred. It would be possible to build a custom plugin for steemd that achieved this, but that's a lot harder than using more traditional web technologies.
There's also the small matter of a steemd full node requiring upwards of 40GB of RAM to run smoothly, where as this forum index I'm running requires around 2GB. Adding a custom plugin to steemd could reduce the 40GB, but probably not a lot, as most of the normal plugins would still be required to maintain the proper data.
It's just not possible to efficiently pull the data needed with steemd at this point, so the indexes are in a database :)
Hope that helps explain why I went the route I did!
I understand the problem and I think keeping the indexes in a database was a good choice. There are ways to implement it over blockchain with a nodeJS app as a middleware to provide a different API using steem-js but it also involve some hacks and caching which is not necessarily easier.
However, since you are looking for a better database solution I recommend you to take a look at Gundb as it could provide a decentralized solution with an API as good as firebase.
I'll have to check it out. Right now I think I'm at my capacity for "new technologies being learned" though.
I think what happened was the docker networking was configured to open the port to mongodb, and someone just connected and dropped the data. Like I said, I didn't work too hard on it's first beta deployment :)
I have a large EC2 instance running steemd + mongodb + python chewing through the blockchain to rebuild those indexes right now. It's taking a little longer than I expected, but instead of deleting the ec2 instance (like I did last time), I'm just going to deactivate it. That will prevent a 2-3 hour full blockchain sync and then 11 million blocks of data processing. Next time (if there is one) I can just start it up and sync where I left off.
If I can get this hosted in a real production environment (as opposed to the VPS its on now), I'll make sure there's redundancy and backups to prevent all of this anyways :)
Good job @jesta
I like it.
I like your way!!!
Congratulations @jesta! You have completed some achievement on Steemit and have been rewarded with new badge(s) :
Click on any badge to view your own Board of Honnor on SteemitBoard.
For more information about SteemitBoard, click here
If you no longer want to receive notifications, reply to this comment with the word
STOPIf you want to support the SteemitBoard project, your upvote for this notification is welcome!
This maintenance is now complete, forum indexes restored, snapshot saved (with todays date for backups), and the big EC2 used to regenerate the indexes is on standby for another occurrence of this.
I really like all of your blogs
and i am also following you
can you please follow me back :D
And i am new here so can you please
upvote some of my post please
Thanks!