The Gridcoin SuperBlock on Linux Is Coming!
Hey everyone, I go by the name BullShark and I’m one of the volunteer developers working on The Gridcoin Project.
As you all know, there have been some delayed SuperBlock issues which could be due to the network having to rely on Windows nodes for producing SuperBlocks. It goes without saying that we need Linux nodes to contribute in creating them and so I have been working on porting the Neural Network to C# and .NET Core so that it can run on Linux.
In this post, I would like to highlight some of the future improvements and the current progress made so far.
Improvements
Removed High Magnitudes for Bad SuperBlocks
In the current implementation when the VB.NET Code fails to download stats for a project, it will not be included in the SuperBlock.
There is a bug which causes everyone's magnitudes that are contributing in the other projects to go up. This is because the calculation uses the total amount of projects that downloaded stats to divide by rather than the total whitelisted projects in the beacon data. This will be fixed in the new version so that if there is a bad SB other users will not be rewarded unfairly.
We still need to come up with a way to prevent users losing their mag for projects that are periodically missing.
No Team Requirement
We have already programmed removing the team requirement into the port so if the vote passes it will be a quick change. As you can see from the image below there are two columns in the Researcher's table with before and after magnitudes when removing the team requirement.
Data Storage
The current implementation stores everything using lots of different flat files inside the NeuralNetwork directory. This is a problem because it makes the development more complex and can lead to file locking issues. The new version solves this by storing everything in a SQLite database.
On top of this, it is also faster and easier to extend because the code is easier to read and third-party tools can be used to query the database.
Improved Syncing
startail ran some tests and found that some BOINC project servers update their stats more frequently than previously thought. It turns out some are updating up to every hour or so and in order for consensus to be formed we need to update local stats more often.
To solve this problem without putting too much load on the BOINC server we are reading the HTTP headers for the Last-Modified date and only downloading the stats file if it has been updated. This means we can update stats on each quorum interval.
Cross-Platform
Last but not least it works on Linux, OSX, BSD and any other UNIX compatible platform where .NET Core runs.
The data gathering and processing is written in C# using the open-source cross-platform .NET Core runtime. I know there have been some people concerned about using .NET and would like to see the full implementation in C++ but it has taken a long time to port to C# and it would take much, much longer to do it all in C++.
The reason, is because this kind of work is much better suited to a higher level language. The UI will still be written in C++ and it will still be fast. Only the data processing and querying the local database will be done in C#.
Current Progress
Most of the VB.NET code has been ported to .NET Core and Ravon has built a bridge in C++ so that the existing wallet can integrate with the new implementation on Linux.
With the bridge he has built we now have a few Linux nodes trying to stake a SuperBlock on Linux. Ravon, TheCharlatan and myself are currently testing the Neural Network port on Linux in the hopes for staking the first Linux SuperBlock. Once we know it works and the port is stable we can start developing the new UI in Qt and C++
Note: we can release the Linux port without a UI in the beginning and then once the UI is built we can start phasing out the old NN code.
It shouldn’t be long now :-)
You can find the .NET Core in my fork of the GitHub repository and the work done on the bridge inside ravon's fork.
This news in huge in so many ways. It solves two major issues with the current implementation; portability and team agnosticism, both of which are major milestones. This is going to both extend the number of clients eligible for producing superblocks AND opening up for removing the dreaded team requirement.
If there is a friday to celebrate it is this one.
The removal of the team requirement would need to be accepted by the community - or by Rob - and therein lines the problem: we had a vote previously and it failed last minute because a big whale or two swooped in and voted no. It's high time to get rid of that requirement, but I'm not too enthusiast about it happening until Rob comes on board and says it will happen.
Rob has already embraced it. It's now up to flipping a switch.
Do you have that in writing somewhere? He didn't vote for it in the poll under his real name - perhaps under one of his investor accounts.
https://gridcoinstats.eu/poll/technical_poll_:_what_should_we_do_regarding_the_mandatory_team_requirement?
It's on cryptocointalk, I shall try to find it :)
Now it is just a fight to see who gets the first SuperBlock :-)
I see a new competition coming up; Most Superblocks per team
🤣
Excellent work @bullshark and it looks to be a great collaboration between the whole development team.
I run a self compiled headless Raspian-lite node on a Pi3, if I can help with running a test node let me know where to get the source code. Are there any special dependencies?
I have a decent GRC stake so it could help your odds in getting a superblock.
It has been a joy working with the team. A lot of great people are working behind the scenes on this project. @ravonn, TheCharlatan, iFoggz, startail, @cm-steem and too many more to mention.
Regarding running it on Raspbian. If .NET Core is able to run on it then it will be possible to build the future wallet to support it. From what I can remember during the BETA stages of the .NET Core development it was possible but I'm not sure it is supported.
I think if we can get it running on Linux and OSX it will be a big step forward though.
great! thanks for this very informativ post and the things you guys working on. :)
we are witnesses of a revolution for gridcoin.
nice to see this . ty for all this work
Dude, I've got to ask: how did you get the blockchain in SQL? I would love to be able to play around with the block in SQL but I haven't been able to figure out how to do this.
The NeuralNetwork does not store anything in the blockchain. Only the SuperBlocks are stored in it which are generated by combining all the researcher mags and CPIDs into a compact file.
So my thinking is there is a JSON for each block and that JSON includes a bunch of good information (one or more transactions, etc.). I 'm not really good at working with JSONs, so that why I wanted to find a way to be able to work with the data from the blockchain in SQL.
Looking forward for this to be implemented. I'm glad that you addressed the issues and offers solution,finally.
Super awesome work! Interesting to see that the code is already in place to remove the team requirement.
Nice one dude.. I follow u.. I'm buying this coin
Upvoted, resteemed, promoted... now excuse me, I am off to place some large GRC orders :)
panic hodls
Thanks, much appreciated :)
What platform do you use to buy GRC? I tried OpenLedger yesterday and was very surprised at the requirement to buy BTS to pay for the fees. Ended up using Bittrex instead. I probably won't use OpenLedger again unless it's to sell off some GRC since I noticed the exchange rates there were better for doing so.
Openledger runs on the bitshares platform and fees for that is BTS, nothing weird with that.
Only issue we have today is that the volume isn't that large. Poloniex still dominates that, but they have their own issues as well, not to mention they are centralized.
Took me quite a while to figure out that I have to buy it in a separate transaction, rather than just pay for the equivalent in whatever currency I'm trading in. More than just inconvenient, it makes it slightly harder to figure out how much I'm paying in fees. Bittrex at least says that it has 0.25% commissions on exchanges and how much I'm paying for withdrawals directly.
I was looking around a little and it is possible to buy GRC on the BitShares market and pay the fees with a different currency than BTS, but only with GRC.
Under the field for the Fee on the purchase or sale order you can switch BTS for GRC.
I know this doesn't solve your initial issue with buying the GRC in the first place without any BTS, but you can now after your first purchase pay the fees in GRC.
Fee Pool
How it works is that the Issuer of an asset has a fee pool that pays the fees for you at a current exchange rate equal to that in BTS. This is up to the issuer to have.
Source: https://steemit.com/bitshares/@xeroc/bitshares-fee-schedule
Thank you for the update! GRC dev team is on point, as always.
You are doing some great work in the branding department as well :)