EOS transitioning: getting into details

 A lot of users have both questions and doubts regarding the variety of directions related to the possible scenario of Golos Сore transition to the EOS code base. This post is not an attempt to cover all of those  questions, it serves as one of the introductory parts. The possible quantity of parts is difficult to predict at the very moment since the number of questions may grow as new details of the move to the new platform emerge.
It is important to point that now we are at the end of research of the EOS code base. In this post we will elaborate on technical issues.

Transitioning to EOS (Part 1: Research)

To study the options for further development of Golos project we initiated a research on EOS platform. The original goal was to study EOS in order to be able to import technical improvements into the Golos codebase. 

During the study process it became quite clear that the best way could be to fork EOS, introduce certain changes and move Golos to completely new blockchain with absolutely new opportunities for our community. Golos Core in this case could focus on blockchain development rather than being forced to make changes for the end clients of the blockchain.

For research purposes only, we have divided current Golos architecture into the following subsystems and analyzed the intersections with the EOS platform:

1. Built-in database in shared memory
2. Network subsystem
3. Transaction verification subsystem with hierarchical authorization system
4. Bandwidth subsystem
5. Add-in over database with routines and consensus indices
6. Set of operations and operation handlers 
7. Set of plugins for additional processing of information entering the blockchain
8. API for issuing information from a block and from the database of all blocks signed

All the operations above as well as their handlers we divided into the following subgroups:

1. Accounts (profile, keys, recovery, authority)
2. Posts and comments (creation, editing, deleting, beneficiary system)
3. Voting for posts
4. Market (transfers, conversions taking into account the restrictions imposed on transfers between 3 types of assets: GBG, delegation, etc.)
5. Witnessing system (voting, changing parameters, mining)
6. Transaction proposals
7. Custom operation for plugins

We have identified the following subsystems for analysis in the EOS codebase:

1. Database in a shared memory similar to Golos
2. Network subsystem
3. Transaction verification subsystem with hierarchical authorization system
4. Bandwidth accounting subsystem
5. Block manufacturers
6. Virtual machine WebAssembler for smart contracts
7. Business logic layer linking accounts, bandwidth and smart contracts in a virtual machine
8. Smart contracts library

During the analysis we figured out which systems could be inherited by the new Golos blockchain (Golos on the EOS codebase) - for further details, see our previous post.

To make your life easier we will list them again. When migrating to the new Golos platform the following subsystems from EOS can be inherited:

1. Built-in database (explanation below)
2. Network subsystem
3. Adapted account system to the requirements for migrating Golos users
4. Transaction verification subsystem with the hierarchical authorization system and with the necessary changes for Golos platform
5. Bandwidth accounting subsystem
6. Block producers with the changes, some of which were mentioned in the post of Nikolai Stefan
7. Subsystem market in smart contracts
8. Transaction proposal in smart contracts

The current Golos economy could be implemented in the form of the following smart contracts:

1. User profiles
2. Create / edit / delete posts / comments, vote for posts
3. Rules for converting a token into a vest, token issuance

In addition to that the following smart contracts can be added into the new blockchain:

1. Application moderators - those who set the rules for the application tokens issuance and distribution, the rules for bandwidth distribution among application users
2. Application workers - those who develope application codebase

In this post we will go deeper into the changes we want to implement. According to our estimates, the Golos Core team of 7 C ++ programmers can develop the proposed solutions within 6 months.

Technical drawbacks of Golos

This post is also intended to provide answers to the questions that are left after our first post. As it was stated before, a task of transition is to create a new competitive advantage for the Golos.

When considering the possibility of an integration with the new platform, third-party observers may have an improper impression that the following implementation requires an immense amount of code to be transferred but, fortunately, this does not match the reality, and one of the goals of this post is to outline all the obvious reasons in detail.

Burdensome heritage 

First of all, the current Golos codebase seems to be overloaded with conditional operators necessary to support economy transition from one state to another. All the past Golos hardforks were inherited from Steem and left traces in the code. These conditional operators make it difficult to add new functionality. They can be reworked (the famous refactoring) but it will require loads of resources without any functional bonuses.

There are different ways that could help to get rid of the inherited burden. And each has their pros and cons.

The transition to any new platform allows you to preserve the current Golos economy, but get rid of all discrepancies that had been pending since the times of Bitshares and move faster in adding new functionality.

What are the advantages of switching to a system built on the EOS codebase (we’d like to underline that it's not about switching to EOS directly)?

The technology of smart contracts lets free the code from storing of conditional operators, as loading a smart contract into a blockchain is a standard operation as well as data loading. The smart contract does not need to store all the conditions of economic transition because it will not receive and process the old data.

This significantly reduces the amount of code that we have to move to the new platform.

Inefficient database

Secondly, Golos daemon currently performs a lot of tasks: from synchronizing data on the network to issuing information on requests from blockchain clients. Each layer uses its own set of technologies. The issue that we would like to focus on is the storage subsystem that allows storing the system state (in fact, the database inside the daemon). The implementation built on boost :: multi_index immensely simplifies the business logic inside the application. But the database is not only a convenience in the implementation of business logic. For this database to be efficient it has to have several different subsystems.

At the moment, these subsystems are not present in the daemon. With enough RAM the daemon has direct access to the data without any middlemen.

But do not forget the details. In order for a daemon to have the ability to issue information and not just modify it, there is a single mutex in the code that allows a lot of read requests and completely blocks access to data when it needs to be changed. This leads to the fact that read requests block the execution of transactions from the newcomer blocks. When there are new transactions for changing the data there is a complete blocking of the read requests. The set of databases that exist in the IT world has a lot of strategies for blocking both individual tables and point locking of individual records. In Golos daemon only one strategy is implemented: global blocking of access to all data.

If you look at the history of commits, you can find out that Golos Core has recycled the system of locks, introduced new mechanics to improve “daemon sensitivity”. The result was not only a growth in daemon responsiveness, but also greater data integrity. For comparison, in the Steem code, and even in the EOS code, there are still bugs that lead to data corruption under certain conditions. Perhaps some of you remember an error when at the start of the daemon you encountered a locked state. Or that to solve this problem it was required to delete the file shared_memory.meta (which now by the way is not there at all).

The daemon work is directly related to the responsiveness of the system as a whole. To address the issue of responsiveness all the existing Golos clients export data to external databases and use their capabilities to respond to queries. And that is not all. Everyone is accustomed to doing replay when a new version of the daemon is released, although only a couple of fields in the non-consensus data component have changed.

All this makes us wonder is there a real need or necessity to store the system state in the format that is used now?

The inflexible API

The next point is the need to implement your API layer inside the daemon as there is no other way to get data from the state, because only the daemon knows how they are stored. This API is a rigidly fixed set of functions with a set of arguments.

You might remember hot dicussions around API cchanges as the functionality of clients is strictly limited by the set of functions and the slightest changes in the API lead to the need to rewrite the clients.

To implement the API layer in the daemon, there are many plug-ins, each with its own business logic, which does not participate in consensus at all. Any new client for the blockchain has to adjust to the current API. This is nonsense in the modern IT world. As a result Golos blokchain has long been a blockchain of one application golos.io.

The Golos Core team spends a lot of time processing and fixing bugs in APIs and plugins, which, by and large, is done to maintain the functionality of one client. And for the implementation of some new functionality, the client - golos.io - is forced to wait for the release of a new softfork.

We would like to expand daemon capabilities and add to it the flexibility in query processing by implementing GraphQL or any other mechanism for processing incoming requests and generating responses to them. Is this reasonable? This will make the blockchain itself almost a database, although it will still be a bit far from it.

From what is mentioned it’s clear that Golos (from a technical point of view) is all about distributed database with a very thick and very tough API layer. Getting rid of the API will significantly reduce an amount of code that has to be transferred to the new platform.

Improving imperfection

What captivates us in EOS? First, it has the same storage system as Golos and Steem. But there is a significant difference which is EOS RAM. The cost of storing data in EOS is such that moving Golos on it as an application becomes extremely expensive. There is one large hash table in EOS for data access, the keys are a username, a name of the smart contract, a table name, a key id in the table. As a value a row of the smart contract table is used. This is what allows you to keep track of user-consumed RAM.

External database

The solution we are proposing is to export of data to an external database and a transformation of the new blockchain into a controller over the database.

The Golos code has a plugin that can export data to MongoDB, however, this is not a solution that will make data access flexible. The plan is to integrate more deeply blockchain and the external database and it consists of the following stages:

- Create an access interface from the boost :: multi_index library to an external data source

- Create a data cache layer in the daemon's RAM

- Write a driver for access to MongoDB

The blockchain will store its state in the external database and not in the internal data format. EOS RAM in our blockchain will cease to exist in its EOS terms, but will become a data caching layer, the bandwidth to which will be accounted similar to the rules that are defined in EOS for bandwidth accounting for a network subsystem and a CPU subsystem.

Data storage setup of several layers enables an easy addition of drivers for other types of databases that can be better suited for different types of nodes. For example, using RocksDB or LevelDB as databases can allow a node to be held without external dependencies with less requirements on system resources. This type of repository may be better suited for seed or delegate nodes.

This approach has a number of advantages:

1. Flexible access to data thanks to the power of modern databases
2. High speed processing of read requests 
3. Availability of tools for data backup
4. Availability of tools for expanding the data structure without a need for a replay 
5. Ability to scale data storage using modern databases

Technical difficulties

As mentioned earlier, not all technical issues are crystal clear. There are a number of possible technical barriers that have to be investigated.

One possible problem is an increased access time to data since this is an external storage. This point is critical since a blockchain node has to be able to process the blocks. And this is superimposed on the fact that the time between blocks in the classic EOS is only half a second. The situation is a little bit improved by the fact that a node while storing data in an external database stops processing requests for information, and it only needs to do several requests for a change when processing a transaction. Even if this is not enough in the new EOS-based blockchain we can increase a processing time for each block.

The next problem is data integrity when a node is stopped. It is not critical as a blockchain has a built-in session mechanism for rolling back the blocks for forks and for rolling back of transactions when a new block is received. This mechanism is perfectly integrated with the database transaction mechanism. As we have a chain of blocks with transactions, we can restore the system state from any stop point.

And perhaps the last technical issue is a  trust factor. A node has to trust to the data stored in the external storage. To be frank, trust to a system state exists in the classical Golos as well as it is difficult to verify data integrity if they are issued on demand. To solve this problem one can add a private-public key pair for each node of the blockchain. So, a node will be able to sign the exported data with the key and check their integrity when downloading from the repository. Similarly, a client will be able to verify data integrity by checking downloaded data.

Theory and practice

The community may have a feeling that the ongoing research and preparation for the move can freeze the development of Golos blockchain. But thanks to the similarity of projects, prototyping of the external storage - if desired by delegates and community - can be implemented on Golos platform (taking into account the benefits that external storage can bring to the system, it seems a reasonable action). As storage implementation for two blockchains is the same, this will allow us to test the theory by implementing this technology in the current Golos.

Below is a plan we have for transferring data to the external database.

1 stage of implementation:

- To implement interface at the boost :: multi_index level to access the external data store

- To implement interfaces for integrating chainbase session objects into an external data store transaction

- To develop a driver for storing data in MongoDB

1 stage of testing:

- To measure time required for a complete replay of the blockchain with the same set of plug-ins on the same hardware with internal and external data storage

- to write load tests with a heterogeneous set of transactions in blocks

- To measure time necessary to perform different types of transactions

2 stage of implementation:

- To add data verification mechanisms to the external data store

2 stage of testing:

- To re-measure time with the data verification mechanism turned on

3 stage of implementation:

- To add a layer of data caching in RAM from the external storage

3 stage of testing:

- To re-measure time with data caching enabled and with the data verification mechanism turned on / off

+ Release of the product version of Golos blockchain with data storage in MongoDB.

All the processes listed above could be implemented within 2 months. After the 1st phase of implementation it will be possible to deploy Golos blockchain with an external data store, allowing the community enthusiasts to test the new technology.

During next stage the external database could be converted into the genesis of the blockchain built on a new platform.