EOS BP Security Statement
This article was written in collaboration with Eric Björk of Sw/eden, Eugene Luzgin of EOS Tribe, Jae Chung of HKEOS, Igor Lins e Silva of EOS Rio, Michael Yeates of eosDAC, and @jemxpat.
Motivation & Concerns
The EOSIO software is both groundbreaking and revolutionary. It has the potential to bring a new era of blockchain applications and create new paradigms in blockchain usability, scalability and governance. Thousands of people, including the people involved in this release, have invested huge amounts of time and money to ensure this potential is realised. This is all culminating in an expected launch of an EOS mainnet on June 2nd/3rd 2018.
However, the EOS mainnet is a unique target for attack. With a year long ICO and so much money raised, large community involvement, huge expectations, it is reasonable to assume that there are malicious detractors who wish it harm. We feel it prudent to consider the nature of possible attacks, and the defenses and procedures expected to be in place against such attacks.
The bootstrap methods implemented in the eos-bios program, result in severe vulnerabilities, allowing any attacker to damage or destroy producer nodes or the entire blockchain. The excessive information about the producer network published directly on the blockchain will allow all types of attack, from protocol injection up to volumetric DDoS attacks.
While it is never possible to perfectly secure any system, it is irresponsible to not create some reasonable defenses. In this message, we detail some of the attacks possible, demonstrate some tested attacks against the insecure boot chain, and then suggest our preferred method of bootstrapping in a more secure manner.
We deeply regret any discomfort and angst caused by this. We have tried to explain the problems and suggest solutions in public and private Telegram chat rooms for over two months. We have encouraged modifications to the proposed automatic bios code. Alas, changes to the security model have not been implemented, and have been actively resisted. We feel that we are left with no choice but to publish a few precise and demonstatable methods of attacking and damaging such a blockchain, and show how to mitigate such risks. If we do nothing, the risk of failure is too large to accept.
We did not publish these concerns earlier because we needed to investigate them extensively and verify that they were not just speculations before sharing them to the community. The writing that follows consists of facts supported by evidence.
Current insecurities discovered in the eos-bios boot process.
Much of the functionality of eosio is provided by plugins. They allow the system to produce blocks, connect to other nodes etc. The base eosio software includes many plugins. A block producing node should not run any unnecessary plugin on a live network, because some of them allow the chain to be hacked. Some of these plugins should never be exposed on a publicly accessible node.
One particular plugin we have noticed is the net_api plugin, which provides an api which can be used to control the plugin remotely. If this API is exposed on a public network then anyone with the ability to connect to this network can tell the block producers to disconnect from, or connect to, any other machine. This does not require anything close to a botnet. Without even being a part of the eos-bios network, a mac mini could disconnect every peer in the network.
Allowing this vulnerability on the live network could mean that an attacker can stop the chain from function properly and even partition the nodes so that eventually they stop. The Ghostbusters team noticed the vulnerability early on in the design of eos-bios and reported it to EOS Canada but no changes were made. This is one reason that we cannot support the launch using eos-bios.
How to hack a chain knowing one peer node running net plugin
Run command:
cleos -u [peer-url] net peers
Returns you a list of responses like following for each node:
{
"peer": "",
"connecting": false,
"syncing": false,
"last_handshake": {
"network_version": 1206,
"chain_id": "0000000000000000000000000000000000000000000000000000000000000000",
"node_id": "b92b2b7d8835e46e2fed97f5eebda31faea63fd07cc40ad52f132254f22cac8e",
"key": "EOS1111111111111111111111111111111114T1Anm",
"time": "1527171594944050635",
"token": "0000000000000000000000000000000000000000000000000000000000000000",
"sig": "SIG_K1_111111111111111111111111111111111111111111111111111111111111111116uk5ne",
"p2p_address": "reach.me.example.com:9876 - b92b2b7",
"last_irreversible_block_num": 57,
"last_irreversible_block_id": "000000391a9439aaa2864d9807965bc84865cdd15c4a0a3d0ae3c7e54a85a38f",
"head_num": 300,
"head_id": "0000012c9826a4f37db5e5c8b4790acad70c01188aee8e7b7330790937a93cdd",
"os": "linux",
"agent": "\"EOS Example\"",
"generation": 2
}
Next get a list of peers only:
cleos -u [peer-url] net peers | grep peersAnd disconnect each peer:
cleos -u [peer-url] net disconnect host:portOne could eventually force the connection to any other peer using, also increasing risks of network instability:
cleos -u [peer-url] net connect host:port
The Recommended Solutions
So should we be panicking and throwing out all of our tokens? Definitely not. As much as we have investigated and tested the potential security risks associated with launching the network, we have also experimented and discussed about solutions to the problems above.
Wireguard Private Mesh Security
Given the fact that EOS software has not been securely vetted and BP node stores sensitive producer private keys, we have to take extra measures for securing those producing nodes. Ideally no public Internet access should be allowed to BP nodes while allowing meshing between BP nodes. Hence we propose using secure peer to peer communication between BP nodes via point-to-point secure tunnels using the open source WireGuard kernel based VPN software.
Each BP node will have 1 or more full nodes that it establishes secure connection to. Those full nodes will be exposed on open Internet protocol and available for access for overall Blockchain network. BP node should not expose it’s non-VPN IP address to the open Internet access. Also for fault-tolerance BP node should establish several trusted connections to other nodes.
- Layer 1 (Block Production Layer)
- 2 Producers: 1x Producer node & 1x Stand-by node
- Producer Control Switch: a machine to monitor and enable failover switching
- Layer 1 nodes could be connected to other trusted producer or full nodes via VPN (WireGuard)
- Nothing extra installed other than Producer API, whose access is restricted to the producer control switch
- Layer 2 (P2P Layer)
- Full nodes to relay blocks
- Connected to the Layer 1 nodes via direct tunnels (WireGuard)
- Securely meshed to trusted BPs via P2P VPN (WireGuard)
- Only uses History API and Chain API, restricted to the proxy servers on layer 3
- BPs are encouraged to make public full nodes available for external access for example, exchanges, portals, new BPs…
- Layer 3 (API Layer)
- Web servers to support HTTP endpoints
- Layer 3 nodes should be pure web firewalls and have no blockchain information
- Connected to our Layer 2 Relay nodes via HTTP (wireguard)
- Uses Patroneos to prevent against basic DDoS and application layer attacks
- Layer 4 (Public Layer)
- Global BP Load Balancer (all traffic goes into this single point)
- Volumetric Attacks gets handled at ISP level
- Routes all HTTPS (SSL) traffic to our Layer 3 web servers
- Configured to DNS (optional)
- L4 can be as simple as DNS round robin with short TTL.
Conclusion
We are 100% committed to launching, and supporting, a stable and resilient EOS mainnet. Given the facts stated above, and insecure nature of eos-bios boot process, our community of BPs is strongly advocating against using at as standard launch procedure as it exposes a complete chain to an attack by anyone.
We, as a community, consider it our duty to ensure that the EOS mainnet is secure and resilient. Using the eos-bios process will create unnecessary risks for the EOS Blockchain launch and ultimately all EOS token holders. Also, any negative press on insecurities in EOS Blockchain launch or failed attempt to launch the Blockchain will have a negative impact on EOS price and reputation.
Therefore we believe it will be in all stakeholders interests to bring down any insecure and unstable network, using above methods we have highlighted, before any malicious third party takes advantage of such insecurity. We will simultaneously endeavour to introduce security improvements into launch and operational processes.
The security imperatives cannot be ignored and, given the stakes, must viewed as the paramount consideration.
- BP Teams Involved:
- Sw/eden
- EOS Tribe
- HKEOS
- EOS Rio
- Block Matrix
- eosDAC
- Eosmeso
- AcroEOS
The eosio::net_api_plugin was made optional for this very reason. No producer or public endpoint should be running with this plugin and it is a simple config file change.
I feel like the intro to this post is overly alarmist way to support an otherwise worthwhile effort at securing block producing nodes.
Sorry guys but eat some humble pie on all the judgments that some put on this group for trying to bring all this up prior to the launch. No one listened.
Now here we are 2-3 days later and BOOT BIOS didn't deliver. All that hype. All that political BS, pomp and circumstance, etc... and in the end, they feel flat on their face.
Funny the difference that just a few days makes. ;)
In the end, the very people you put in question ended up being the talent that will launch the chain.
Are you aware of the Ghostbusters project?
here's where Alex explains that net_api module was added for testing only and he warns it has to be taken out:
Thanks for the update, heard about this but hadn’t looked into it yet...
Thanks for the information! Launching the EOS mainnet will require a great deal of coordination between Block Producers. This involves configuring secure connections to prevent attackers from scanning the entire network topology looking for nodes to shut down.
We will need to plan, communicate and work closely together with other Block Producers to ensure the network is secure and efficient.
We have identified this requirement in our Technical Roadmap
The vulnerability you are pointing to is in the net_api plugin, not in the EOS-BIOS software.
I'd rather see a transparent, consensus oriented, well orchestrated launch, than one obscured by a VPN system among a group of BP's.
I've never been an advocate of "Security by Obscurity" especially when BP's are suppose to act with complete transparency.
But I would wait to see EOS Canada's debunking of this issue before making further comments.
The plugin in question in fact had some dependancies at the time of announcement though. A time when they were only days away from launch.
The Ghostbusters/ Trinity group had serious concerns that this could slow down the launch of the chain ultimately - which ended up actually happening. Now the Ghostbusters/Trinity group are the CORE group launching the chain.
That says everything you really need to know about all this.
End of story.
why I am not surprised. first thing they have to do is to behave like professionals and act securely. bps must realise they are not 'talking' to the community only, nor the other bps; they are talking to their future hackers. bring down the youtube videos sharing screens and commands please and stop being careless. if you care about the community be secure. and if you care more, be more secure. you'll never tweet, share your name or your email because that would be insecure. and you would reset your router each day like a pro.
Relying on hackers not knowing how to hack your system (also known as "security by obscurity"), doesn't work. You can safely assume hackers to know and understand your system well. Hence you can not hope for hackers not attacking your system.
What this group have published is nothing secret but standard functionality.
Besides these concerns were raised in smaller group conversations for some time now but no actions were taken to remedy security concerns.
And while marketing was winning technical side of this argument - we had to make these facts known to a wider community while keeping in mind it may make some people unhappy.
Ultimately our goal is successful launch of EOS Blockchain and a duty to EOS token holders.
Really is a breath of fresh air to see a talented community with a clear. The combination of this security bug and the consensus algorithm flaw i think it's safe to say mainnet isnt ready for launch and the fact that the community is aware of that and not trying to rush it is a great sign.
Great work Eugene and team!
It's a good article. You should edit it to take out the alarmist parts now that you know...
But that's the thing - the Ghostbusters/Trinity group was right about all their assessments. The BOOT BIOS group was not able to stand up the chain because of some of the very reasons the Ghostbusters/Trinity group outlined. And the plugin in question ended up having some dependancies and wasn't as easy to use without them - so some people in the BOOT BIOS group couldn't hang when it was game time.
Not trying to be confrontational, but that's the final word on all that.
Fair enough then if the tone of the article was about right in the end. To me it initially sounded overly political. Always good if you can end up coming back and say "I told you so"! Good work ghostbusters
Go get em son we aint no verge
@remind.bot 3 hours
Mentioning and replying to you here on Mon, 28 May 2018 21:46:10 GMT to remind you of this post.
Please consider upvoting this comment to keep this service running.
Reminding @cmichel of this post as requested!
Please consider upvoting this comment to keep this service running.