what if mongodb?

Hi there!

For those who have checked out the repo of the steem smart contracts tool (SSC) you may have seen that the storage layer is powered by LokiJS, a Javascript in-memory database. This library is great, the performances are awesome but that's thanks to its nature of being hosted in RAM... but we know what happens when we store data, too much data in RAM... it becomes expensive to maintain nodes and solutions need to be found in order to cut hosting costs.

When I started to work on SSC, I was just trying to "copy" ideas from different "blockchains" and integrate them into the tool, while I saw having a in-memory database was a must have to ensure performances I then figured out that, well, is that really necessary? I saw what Steemit Inc. had to do to solve that problem and now I don't want to run into that issue in the future so I decided to work on replacing LokiJS.

LokiJS is a no-sql database that uses an API that is very similar to what we can find on the very well known MongoDB database engine, so guess what? I decided to change the storage layer of SSC to MongoDB, that wasn't really painful... it took me a few days to figure everything out but it's working and it's working great!

One cool thing with MongoDB is that you can use different storage engines with it, WiredTiger, the default one that's shipped with MongoDB, but other companies worked on integrating other engines and so you can for example find RocksDB as one of the engines shipped in the fork of MongoDB maintained by Percona (they also have the in-memory engine if you really want to keep your database in RAM...)

One major drawback with moving to MongoDB is the replay time, I didn't calculate exactly but that's wayyyy longer...BUT you don't need to replay when you use MongoDB because you can simply get a dump of the database from another node an import it into your database...then you just need to run your node and it will only have to process the blocks that weren't included in the dump.

Moving to a well know database engine also comes with benefits, MongoDB offers solutions for scalability (replica sets, sharding) and depending on the performance you want to achieve with your node, you can choose the storage engine that best fits the hardware you have; you have a crazy server with tons of RAM because you know your node is going to be highly solicited then go with the in-memory engine, you're short on RAM, go with WiredTiger or RocksDB, that's up to you!

I've fired up two nodes to test this migration out, you can interact with these them via these two links:

• https://harpagon210.github.io/steem-engine : MongoDB 4.0 with WiredTiger
• https://harpagon210.github.io/steem-engine-rocksdb : MongoDB 3.6 with RocksDB

You can try to browse around and check the data against https://steem-engine.com/ and it should (surprisingly?) be the same... (except for the block hashes which are calculated based on the database hashes which are different between LokiJS and MongoDB)