Updated: April 13, 2018

Author:@xebb82

contributors: @summerskin, @michaelyeates, @zaratustra, @nsjames, @v2c, @marcinup

Outline

• Intro and purpose
• Installation and configuration
  • Build dawn-v3.0.0 on Ubuntu 16.04, 16.10 or 17.10
  • One way to run multiple testnets on the same server
  • Connect to the Community Testnet
  • ... or start your own testnet
• Making our setups a bit more realistic
• Load testing
  • Hardware setup
  • Configuration
  • Results
  • Conclusions
• What's next?
• Links

Disclaimer:

This post outlines one way to run multiple testnets on the same server. Several excellent posts which details how to get the software up and running already exists: Bohdan, Eugene, Marcin, Michael. However we still get a lot of questions. This post describes how I do it. This is not the only way. This is not even the best way. But it is my way :)

Intro and purpose

Technically versed people that have either been playing with the eosio software or who understand how to use and follow the official instructions located here can save some time and jump straight down to: Making the setup a bit more realistic. Everything in this post is subject to change as more information is released and as new tools come out.

This post covers how to download, build and configure the software to connect to a testnet, as well as how to create your own testnet. It also goes through some technical aspects to consider when planning how to run your multi-node setup. Lastly, we detail some performance tests that we have run on a testnet using multiple different setups.

We believe some of the community will enjoy the information provided in this post and therefore, without further adieu, here it is.

Installation and configuration

In this part we are assuming that the reader has a basic understanding on how to read and execute bash scripts using a Linux environment. The scripts provided have been tested on Xubuntu 17.10 but they should work on Ubuntu >= 16.04.

Since early January we have had the Community Testnet more or less stable. We were however forced to reset the chain as we upgraded it to the branch that we thought would become dawn3 - but instead it turned out to be SuperDawn (the producers in the Community Testnet are named after superheroes and we got a tag named after us!).

$ ./client.sh get block 1 | jq -e ".timestamp"
"2018-02-03T21:46:00.000"

Our first block was created a little over than 2 months ago. During that time:

$ du -h --max-depth=1     
2,1G    ./blocks
1,1G    ./shared_mem
3,2G    .

... the directory holding the blockchain has grown to 3.2G in total.
As long as the chain is made up of mostly empty blocks it does not consume much disk. And neither does the software:

$ du -h --max-depth=1
1,5G    ./eos
1,5G    .

One assumption I'm making throughout this post is that disk can be seen as "free". To someone running their machines in the cloud this assumption is probably not correct (you might want to use one directory where you build the eosio software, and then only keep the binaries), but for those running on dedicated hardware it is probably true enough.

Build dawn-v3.0.0 on Ubuntu 16.04, 16.10 or 17.10

If you want to do it in a standardized and simple way you can just:

$ cd && git clone https://github.com/eosio/eos --recursive
$ cd eos && git checkout tags/dawn-v3.0.0
$ ./eosio_build.sh

... or you can:

One way to run multiple testnets on the same server

One way to run multiple testnets on different versions and setups on the same machine while lowering the risk of mixing them up is to do something like the following:

$ cd && mkdir testnets && mkdir testnets/[testnetName]
$ screen -SU [testnetName]
$ cd testnets/[testnetName]
$ mkdir wallet data-dir config-dir 
$ touch wallet/start.sh start.sh cleos.sh
$ git clone https://github.com/eosio/eos --recursive
$ cd eos && git checkout tags/dawn-v3.0.0
$ git submodule update --init --recursive
$ ./eosio_build.sh

Output from the build process should look like this:

1. We start by creating one directory (testnets) under which we create one directory per testnet.
2. A tip that helps me keep track of my different testnets: I start a screen session for each and everyone of them. These run in the background and then can easily be attached as you log back on or switch between the different sessions. I personally let nodeos run in one window (mostly because I like the colored output). Then I use another window where I call the client (note that we in a real environment will want to log to disk).
3. Clone and build one copy of the eosio software per testnet. This makes it quick and easy to upgrade one specific testnet with limited impact on the other.
4. Create three scripts per testnet;
   • start.sh - start the daemon
   • cleos.sh - call the client
   • wallet/start.sh - start the wallet daemon

Add something like the following to start.sh (we will modify this one later):

#!/bin/bash
./eos/build/programs/nodeos/nodeos --data-dir data-dir/ --config-dir config-dir/ $@

cleos.sh:

#!/bin/bash
./eos/build/programs/cleos/cleos -H localhost -p 8888 --wallet-host localhost --wallet-port 8000 $@

wallet/start.sh:

Here we bind the wallet (keosd) to localhost so no other system can access our private keys. You do not want to run the wallet plug-in with your nodeos.

#!/bin/bash
../eos/build/programs/keosd/keosd --http-server-address localhost:8000 $@ &

Next step is to make cleos.sh and start.sh executable, start the wallet daemon and import the keys:

$ cd && cd testnets/[testnetName]/wallet && chmod u+x start.sh
$ ./start.sh
$ cd .. && chmod u+x start.sh && chmod u+x cleos.sh

# ( !!! make sure you save the password from the following command !!! )
$ ./cleos.sh wallet create 
$ ./cleos.sh wallet import <private key>

# (show all private keys in the wallet)    
$ ./cleos.sh wallet keys

Other ways

There are multiple ways of doing this same thing and it is important to find your own specific flavor.
A lot of good inspiration can be found in a tool written by zaratustra, intended to make it easier to launch and manage multiple EOS.IO chains.

Eduardo also provide some docker images.

Connect to the Community Testnet

With small modifications the information below can be used to connect to any and all of the different testnets available. First we need to get the genesis.json and the config.ini:

$ cd && cd testnets/[testnetName]
$ wget https://raw.githubusercontent.com/michaeljyeates/10mill/master/genesis.json
$ wget https://raw.githubusercontent.com/michaeljyeates/10mill/master/config.ini

Certain things need to be modified in config.ini

genesis-json = /home/<user>/testnets/[testnetName]/config-dir/genesis.json
http-server-address = 0.0.0.0:8888
p2p-listen-endpoint = 0.0.0.0:9876
p2p-server-address = <domain_or_ip>:<port>
agent-name = "<username>"

Setting http-server-address to 0.0.0.0:8888 means that it will listen on all network interfaces, on the port 8888. If you have something else running on this port you will receive an error when trying to start the daemon, it will say something like: "bind: Address already in use"

The p2p-server-address is afaik not used for anything other than identifying your node. It should to be set to something unique.

To start syncing the Community Testnet blockchain:

$ cd && cd testnets/[testnetName]
$ ./start.sh

It is normal if it looks something like the following:

1. look at our monitor (handled by @marcinup, built by @cryptolions) to see which block we are currently at

2. execute cleos.sh to see which block you are at

   $ cd && cd testnets/[testnetName]
   $ ./cleos.sh get info

Once they match you are fully synced.
Come join the fun at https://t.me/CommunityTestnet . Unfortunately we don't accept new producers until we have BP voting functionality.

Once you are up and running

Once everything is working as it should it might be a good idea for you to start running nodeos in the background and re-direct the output to a log file on disk instead of logging directly to the window in your screen.

If you edit your start.sh to look something like this:

#!/bin/bash
./eos/build/programs/nodeos/nodeos --data-dir data-dir/ --config-dir config-dir/ > data-dir/log.txt 2>&1 &

... then you will redirect the output from nodeos to go to a text file. You can do something like this to keep an eye on the last 30 lines:

$ watch tail -n 30 data-dir/log.txt

In the future you will want to add some automated way to keep track of what is going on in this file. If you want you can add: "echo $! > data-dir/nodeos.pid" to start.sh, which will save the pid in a file (data-dir/nodeos.pid). This pid can be killed (use -2 och -15) if you need to restart nodeos.

Connect to one of the other community driven testnets

Under the Links section we have included information to get in touch with all community driven testnets that we are aware of. Both the Scholar testnet and Arrowhead are currently accepting new producers.

... or start your own testnet

First we need to create a genesis file. We recommend placing this at: testnets/[testnetName]/config-dir/genesis.json but it can be anywhere you want. The following genesis is what we used when we started the Community Testnet on dawn-v3.0.0 (a gold star to the first one who decodes the chain_id).
Make sure to generate and use your own initial_key (./cleos.sh create key is your friend).

{
    "initial_key": "EOS55xgWhiYohaHtFTxcPMorB9MzwBfe3JgUuQEujrurmHLHiXmna",
    "initial_timestamp": "2018-04-06T00:00:00",
    "initial_parameters": {
        "maintenance_interval": 86400,
        "maintenance_skip_slots": 3,
        "maximum_transaction_size": 2048,
        "maximum_block_size": 2048000000,
        "maximum_time_until_expiration": 86400,
        "maximum_producer_count": 1001
    },
    "immutable_parameters": {
        "min_producer_count": 42
    },
    "initial_chain_id": "00000000000000000000000000000000003c332053555045524845524f203c33"
}

Boot the chain

What is written under this topic will probably be replaced by Alexandre Bourgets EOS Bios Launch in the near future. Until then, this will serve:

Michael put together a bash script that can be used to help with the following steps of the boot process:

1. Set the eosio.bios contract
2. Create producers based on a csv-file
3. Set the eosio.system contract, issue 1000000000 EOS and fund each producer with 1 000 000 EOS (we have a problem with showing balances and possibly setting the eosio.token contract fixes this - but it has yet to be confirmed (thanks @cryptolions))

Download the script from our highly official github.

$ wget https://raw.githubusercontent.com/michaeljyeates/10mill/master/boot_chain.sh

Some things in boot_chain.sh needs to be edited to fit your environment:

#!/bin/bash
CLEOS=/home/<user>/testnets/[testnetName]/eos/build/cleos/cleos
HOST=localhost
PORT=8888
WALLET_HOST=localhost
WALLET_PORT=8000
EOS_BUILD_DIR=/home/<user>/testnets/[testnetName]/eos/build
EOSIO_KEY=EOS7rgcwNPrpgd9FSGAyg6Qn5pHE3sV1nYhiXDRDB2rtVkoh7BX7H

Create the file producers.csv (producer_name,public_key) and put it in the same directory as your boot_chain.sh. It could look something like this: (make sure each producer have the matching private keys to the public keys you are using):

firestorm,EOS5cMGmNturu4C1R4T8dB2LVx5SMNoiL9uhYdNKJVxmdLuAnEqo1
ironman,EOS59vUXs8FXvZshSCMBgzseDpu7jcnzWmyv29XXyH3gqUtZ84t41
aquaman,EOS64Vgd3PVJfoTvB19K8fShiy9e1ZGo1iN33PgLrJfUDeTXSmSuG
phoenix,EOS62ZdqFkz4yWhdF3vE2AzMLDSRgEqSfLm5NESkabGY6V4HQ8PkB
batman,EOS5wEcKNZ8YdVnCSCnGmwpWLwb2cCqP5FSyWoW1vBASr78KKu8G3

Create a config.ini. Make sure:

1. you are not connecting to any peers: leave p2p-peer-address commented out (and use a port that noone will connect to)
2. the private-key used as initial_key in the genesis.json is included as: private-key = ["EOS...", "..."]
3. producer-name = eosio

Then start the chain. Validate that eosio is producing blocks, import the private key for eosio and then run the script:

$ cd && cd testnets/[testnetName]
$ ./cleos.sh wallet import <private_key>
$ chmod u+x boot_chain.sh
$ ./boot_chain.sh

At this point you can use the steps detailed in Connect to the Community Testnet to launch another instance of nodeos to start producing with your new producers (firestorm,ironman,aquaman,phoenix and batman)

Import the ERC20 distribution

As far as I know there is currently no software available from block.one on how to import the ERC20 token distribution. Therefore I quickly whipped together two bash scripts that can be used for this (found at the ScholarTestnets github).

You need a snapshot of the distribution. Either you can create one yourself, or you can cheat and download a snapshot made by Robert Allen (found here).

The script I wrote loops the csv-file, extracts the amount and the public key. It then randomizes an account name, creates the account and transfers tokens to it. It had at least two small bugs that the Scholars might have fixed by now:

• a short sleep could be added after every 300-600 transactions, else you'll get the block exhausted allowed resources-message and your transactions will not go through (if you are lazy you can just stop the script, and then run it again after waiting a few seconds)
• "0000" needs to be added to the amount to make sure the correct distribution is sent to each user

Making our setups a bit more realistic

As we are getting closer to launch it is probably time to start taking our setups more serious. In this post we decided to test a few slightly different setups and document how they each perform under load.

1) Single producer

Incoming traffic

The firewall forwards incoming traffic on ports 9876 and 8888 to the producing node which provide users and other producers access to the blockchain.

Outgoing traffic

None.

2) Shared firewall

Incoming traffic

The firewall forwards incoming traffic on ports 9876 and 8888 to the full node which provide users and other producers access to the blockchain.

Outgoing traffic

Both the full node and the producing node will establish multiple connections to the nodes making up the blockchain.

3) Hidden producer

Incoming traffic

The firewall forwards incoming connections on ports 9876 and 8888 to the full node which allows other producers and users access to the blockchain.

Outgoing traffic

The producer is connected only to the full node which in turn handles the connection to the multiple nodes making up the blockchain.

4) Two full nodes, one balancer and one hidden producer

The initial tests with one hidden producer and one full node (2) seemed to indicate that the amount of transactions included in each block went down slightly as the producer was not connected to the sender, therefore we added a test with dual full nodes. More on this under results.

Incoming traffic

The load balancer forwards incoming connections on ports 9876,9776 and 8888,8788 to the full nodes which allows other producers and users access to the blockchain.

Outgoing traffic

The producer is connected to the two full nodes.

5) Six nodeos on two full nodes, one balancer and one hidden producer

This is a variation of 4) where we run three instances of nodeos per full node.

Incoming traffic

The load balancer forwards incoming connections on ports 9876,9866,9856, 9776,9766,9756 and 8888,8878,8868, 8788,8778,8768 to the full nodes which allows other producers and users access to the blockchain.

Outgoing traffic

The producer is connected only to the full nodes.
The full nodes are not connecting to any node.

Load testing

Hardware setup

We decided to use three different machines in these tests, they are all running Ubuntu 17.10, are connected via 1 gbps network and have the following hardware specifications;

1. AMD Ryzen 1800x, 32GB, Samsung 960 PRO m.2 512GB
2. Dual Xeon X5670 6 core @ 2.93 GHz, 64GB, 4x148GB 10k rpm raid5
3. Dual Xeon X5670 6 core @ 2.93 GHz, 128GB, 2x240GB Intel SSD Pro raid1

One thing we wanted to investigate was how the different servers stand compared to each other. To do this we have been moving the producing accounts (six in total) between the different servers: de-facto turning producers into full nodes and vice-versa.

Since I only have one machine running on AMD; all AMD - AMD tests are done using the same server. For Intel I have made sure to use both machines; one as producer, one as full node.

Configuration

This script test far from all aspects in need of testing.
We decided to focus this article mostly on running this script on multiple machines, with different configurations and with small variations to the script (such as sending from multiple accounts / adding a & to the end of the line starting with ./cleos.sh... starting a new thread for each round in the loop and so forth):

#!/bin/bash
x=1
while [ "$x" -lt "601" ] ; do
    ./cleos.sh transfer eosio storm $x test -f 
    let "x++"
done    

Results

This section can seem a bit spammy but I want to include as much data as possible. It is the only way I can see to make sure we will receive good input regarding the (lack of) conclusions drawn below.

I have left the first line as is, and then cleaned the rest to fit the logging output into fewer lines

1) Single producing node

The client run cleos which connects directly to the producing node to push the transactions into the queue.

400 transactions - single process

Producer: Intel

Client : AMD

storm generated block 69d80e1a... #936319 @ 2018-04-12T09:18:52.500 with 32 trxs, lib: 936272
storm generated block eae7e8e2... #936320... 38 trxs, lib: 936272
gargamel generated block 4b8bd1ba... #936321... 38 trxs, lib: 936284
gargamel generated block b8f5ff12... #936322... 38 trxs, lib: 936284
gargamel generated block 1b3011a4... #936323... 38 trxs, lib: 936284
gargamel generated block bacca3fd... #936324... 39 trxs, lib: 936284
gargamel generated block 9a7f3f63... #936325... 38 trxs, lib: 936284
gargamel generated block 75b86a35... #936326... 37 trxs, lib: 936284
gargamel generated block c2b90933... #936327... 39 trxs, lib: 936284
gargamel generated block e083987a... #936328... 38 trxs, lib: 936284
gargamel generated block bd3483fa... #936329... 25 trxs, lib: 936284

Accepted transactions: 400

Client : Intel

eosio generated block 02416a28... #934068 @ 2018-04-12T09:00:06.500 with 11 trxs, lib: 934029
eosio generated block b0064143... #934069... 23 trxs, lib: 934029
eosio generated block be9d02de... #934070... 23 trxs, lib: 934029
eosio generated block 7f49eb8c... #934071... 23 trxs, lib: 934029
eosio generated block ca3e3016... #934072... 24 trxs, lib: 934029
eosio generated block adee9e8d... #934073... 23 trxs, lib: 934029
eosio generated block f282b9bd... #934074... 24 trxs, lib: 934029
eosio generated block d39450d5... #934075... 23 trxs, lib: 934029
eosio generated block 8d9d0280... #934076... 23 trxs, lib: 934029
eosio generated block 038c7c13... #934077... 24 trxs, lib: 934029
storm generated block 13be901d... #934078... 25 trxs, lib: 934041
storm generated block 601f8483... #934079... 23 trxs, lib: 934041
storm generated block 4cf71e87... #934080... 24 trxs, lib: 934041
storm generated block 83458b1b... #934081... 23 trxs, lib: 934041
storm generated block 02777f90... #934082... 24 trxs, lib: 934041
storm generated block 587224ae... #934083... 23 trxs, lib: 934041
storm generated block e720dcf3... #934084... 23 trxs, lib: 934041
storm generated block f1d46148... #934085... 14 trxs, lib: 934041

Accepted transactions: 400

Producer AMD

400 transactions - multiple threads

Client : AMD

batgirl generated block bbc4465f... #938719 @ 2018-04-12T09:38:52.500 with 51 trxs, lib: 938672
batgirl generated block bdcc5bd9... #938720... 148 trxs, lib: 938672
noheart generated block 51f744e2... #938721... 67 trxs, lib: 938684
noheart generated block ccd7271f... #938722... 123 trxs, lib: 938684
noheart generated block c45f4dae... #938723... 8 trxs, lib: 938684
noheart generated block f7e15287... #938725... 3 trxs, lib: 938684

Accepted transactions: 400

Client : Intel

noheart generated block 047016a3... #940382 @ 2018-04-12T09:52:44.000 with 12 trxs, lib: 940340
noheart generated block 6cc4ea81... #940383... 98 trxs, lib: 940340
robin generated block 916c95c0... #940392... 59 trxs, lib: 940352
robin generated block 4fb18ebf... #940393... 76 trxs, lib: 940352
eosio generated block a37dd8d0... #940402... 64 trxs, lib: 940364

Accepted transactions: 309

Producer Intel

Client : Intel

eosio generated block e42da57c... #944230 @ 2018-04-12T10:27:07.500 with 39 trxs, lib: 944189
eosio generated block 89bfd68b... #944231... 183 trxs, lib: 944189
eosio generated block 36ad237d... #944232... 36 trxs, lib: 944189
eosio generated block 15ceb810... #944233... 133 trxs, lib: 944189
eosio generated block 70a091b8... #944234... 9 trxs, lib: 944189

Accepted transactions: 400

Client : AMD

noheart generated block 58359209... #944996 @ 2018-04-12T10:33:30.500 with 197 trxs, lib: 944957
noheart generated block ab52eddb... #944997... with 2 trxs, lib: 944957
noheart generated block 6c3fc2bc... #944998... with 194 trxs, lib: 944957
noheart generated block e7fa13fe... #945000... with 7 trxs, lib: 944957

Accepted transactions: 400

2) Shared firewall

400 transactions - single thread

Producer AMD

Client : AMD

eosio generated block 238d0f5b... #474716 @ 2018-04-09T14:33:43.500 with 17 trxs, lib: 474675
eosio generated block b9346754... #474717... 37 trxs, lib: 474675
eosio generated block 3bb1c121... #474718... 35 trxs, lib: 474675
eosio generated block 7fe86788... #474719... 37 trxs, lib: 474675
eosio generated block 9df12ec0... #474720... 36 trxs, lib: 474675
eosio generated block 2b673f6f... #474721... 37 trxs, lib: 474675
eosio generated block 57ef6011... #474722... 37 trxs, lib: 474675
eosio generated block 1b413e1f... #474723... 37 trxs, lib: 474675
storm generated block 7fb9b53a... #474724... 37 trxs, lib: 474687
storm generated block 7263eb30... #474725... 35 trxs, lib: 474687
storm generated block ccd4d263... #474726... 37 trxs, lib: 474687
storm generated block 28870f83... #474727... 18 trxs, lib: 474687

Accepted: 400

Client : Intel

storm generated block 7b75f0ea... #974247 @ 2018-04-12T14:37:25.500 with 7 trxs, lib: 974206
storm generated block 3508866e... #974248... 23 trxs, lib: 974206
storm generated block ce1c9b79... #974249... 23 trxs, lib: 974206
storm generated block cf16b535... #974250... 23 trxs, lib: 974206
storm generated block d11ab6bd... #974251... 23 trxs, lib: 974206
storm generated block 86096b8c... #974252... 24 trxs, lib: 974206
storm generated block d74e0c97... #974253... 23 trxs, lib: 974206
storm generated block 546e1dc6... #974254... 23 trxs, lib: 974206
gargamel generated block 4534f0bd... #974255... 23 trxs, lib: 974218
gargamel generated block 9eae1196... #974256... 23 trxs, lib: 974218
gargamel generated block 46b0e19d... #974257... 24 trxs, lib: 974218
gargamel generated block 5db95d6b... #974258... 23 trxs, lib: 974218
gargamel generated block c2e6c8e0... #974259... 24 trxs, lib: 974218
gargamel generated block 0ff75754... #974260... 23 trxs, lib: 974218
gargamel generated block e2ea9cbc... #974261... 24 trxs, lib: 974218
gargamel generated block 64b8cb62... #974262... 23 trxs, lib: 974218
gargamel generated block 2094d945... #974263... 23 trxs, lib: 974218
gargamel generated block f42f750e... #974264... 21 trxs, lib: 974218

Accepted: 400

Producer Intel

Client : AMD

eosio generated block e14aa8f5... #961940 @ 2018-04-12T12:54:47.000 with 5 trxs, lib: 961892
storm generated block ded09ff3... #961941... 39 trxs, lib: 961904
storm generated block 57b182da... #961942... 38 trxs, lib: 961904
storm generated block 42bd953a... #961943... 38 trxs, lib: 961904
storm generated block bfe23d4f... #961944... 39 trxs, lib: 961904
storm generated block dd81dee3... #961945... 38 trxs, lib: 961904
storm generated block 6896bda7... #961946... 38 trxs, lib: 961904
storm generated block 42f2dd71... #961947... 39 trxs, lib: 961904
storm generated block 0ae1c8ea... #961948... 38 trxs, lib: 961904
storm generated block 38d85fdd... #961949... 39 trxs, lib: 961904
storm generated block 2f679555... #961950... 39 trxs, lib: 961904

Accepted: 390

Client : Intel

gargamel generated block 742491b3... #962173 @ 2018-04-12T12:56:43.500 with 25 trxs, lib: 962132
gargamel generated block 8c4d8053... #962174... 26 trxs, lib: 962132
gargamel generated block 25df3125... #962175... 28 trxs, lib: 962132
gargamel generated block 09724881... #962176... 26 trxs, lib: 962132
gargamel generated block b9205c5f... #962177... 26 trxs, lib: 962132
gargamel generated block 38010e88... #962178... 27 trxs, lib: 962132
gargamel generated block 13b11d88... #962179... 27 trxs, lib: 962132
gargamel generated block bc023a92... #962180... 27 trxs, lib: 962132
batgirl generated block d16e54af... #962181... 27 trxs, lib: 962144
batgirl generated block 4fc23dd6... #962182... 27 trxs, lib: 962144
batgirl generated block 6730e0bb... #962183... 25 trxs, lib: 962144
batgirl generated block f43079b3... #962184... 27 trxs, lib: 962144
batgirl generated block 1d0e376f... #962185... 26 trxs, lib: 962144
batgirl generated block d6414136... #962186... 27 trxs, lib: 962144
batgirl generated block d0be491f... #962187... 27 trxs, lib: 962144
batgirl generated block cd3893e5... #962188... 2 trxs, lib: 962144

Accepted: 400

400 transactions - multiple threads

Producer AMD

Client : AMD

storm generated block 59c6f1ae... #974609 @ 2018-04-12T14:40:26.500 with 4 trxs, lib: 974566
storm generated block 60533d50... #974610... 148 trxs, lib: 974566
storm generated block efd26313... #974611... 24 trxs, lib: 974566
storm generated block 0ca9b026... #974612... 162 trxs, lib: 974566
storm generated block 45141188... #974614... 1 trxs, lib: 974566
gargamel generated block 03a3d02a... #974616... 61 trxs, lib: 974578

Accepted: 400

Client : Intel

batgirl generated block 79ba6f1f... #986730 @ 2018-04-12T16:21:28.500 with 12 trxs, lib: 986683
batgirl generated block ab749e72... #986731... 206 trxs, lib: 986683
noheart generated block ca968e1e... #986732... 8 trxs, lib: 986695
noheart generated block 9a9aeda9... #986733... 153 trxs, lib: 986695
noheart generated block 665c9fa4... #986737... 11 trxs, lib: 986695

Accepted: 390

Producer : Intel

Client : AMD

noheart generated block 19e82a4f... #987164 @ 2018-04-12T16:25:05.500 with 3 trxs, lib: 987127
noheart generated block 46a45013... #987165... 208 trxs, lib: 987127
noheart generated block a40159bb... #987167... 185 trxs, lib: 987127
noheart generated block dbc9579f... #987169... 2 trxs, lib: 987127
noheart generated block 770f4636... #987171... 2 trxs, lib: 987127

Accepted: 397

Client : Intel

gargamel generated block b2388402... #986996 @ 2018-04-12T16:23:41.500 with 178 trxs, lib: 986959
gargamel generated block 85c48888... #986997... 56 trxs, lib: 986959
gargamel generated block 686602cc... #986998... 166 trxs, lib: 986959

Accepted: 400

3) Hidden producer

The client run cleos which connects directly to the full node to get the transactions into the queue.

400 transactions - single thread

Producer AMD

Client : AMD

noheart generated block 6135feb2... #990185 @ 2018-04-12T16:50:17.500 with 21 trxs, lib: 990148
noheart generated block f945bc8e... #990186... 36 trxs, lib: 990148
noheart generated block e202dbb0... #990187... 36 trxs, lib: 990148
noheart generated block b3f42ccb... #990188... 36 trxs, lib: 990148
noheart generated block 469f55f6... #990189... 35 trxs, lib: 990148
noheart generated block 4bdcdcbc... #990190... 37 trxs, lib: 990148
noheart generated block 83cc0ca1... #990191... 34 trxs, lib: 990148
noheart generated block bb4b7291... #990192... 36 trxs, lib: 990148
noheart generated block acac2070... #990193... 36 trxs, lib: 990148
noheart generated block 076ce3f3... #990194... 36 trxs, lib: 990148
noheart generated block 8eb40b5d... #990195... 36 trxs, lib: 990148
noheart generated block 2c47515d... #990196... 21 trxs, lib: 990148

Accepted: 400

Client : Intel

robin generated block c089fac4... #975381 @ 2018-04-12T14:46:52.500 with 24 trxs, lib: 975334
robin generated block 18c3d110... #975382... 26 trxs, lib: 975334
eosio generated block 0d7ef01a... #975383... 25 trxs, lib: 975346
eosio generated block 1708ad64... #975384... 27 trxs, lib: 975346
eosio generated block 261799dc... #975385... 26 trxs, lib: 975346
eosio generated block 00df4a86... #975386... 25 trxs, lib: 975346
eosio generated block 48118069... #975387... 26 trxs, lib: 975346
eosio generated block 3dc8603d... #975388... 26 trxs, lib: 975346
eosio generated block afb66413... #975389... 25 trxs, lib: 975346
eosio generated block db60d0c9... #975390... 27 trxs, lib: 975346
eosio generated block b9290be7... #975391... 26 trxs, lib: 975346
eosio generated block 84719f28... #975392... 26 trxs, lib: 975346
eosio generated block fad10576... #975393... 25 trxs, lib: 975346
eosio generated block 82243480... #975394... 27 trxs, lib: 975346
storm generated block c25f7cb3... #975395... 26 trxs, lib: 975358
storm generated block 64eed439... #975396... 13 trxs, lib: 975358

Accepted: 400

Producer Intel

Client : AMD

gargamel generated block 8f40d25c... #977215 @ 2018-04-12T15:02:10.500 with 18 trxs, lib: 977168
gargamel generated block 1e44eb1c... #977216... 36 trxs, lib: 977168
batgirl generated block c078a0a5... #977217... 34 trxs, lib: 977180
batgirl generated block e92eed1e... #977218... 34 trxs, lib: 977180
batgirl generated block 91f32026... #977219... 37 trxs, lib: 977180
batgirl generated block 81187f25... #977220... 36 trxs, lib: 977180
batgirl generated block 4cc9b790... #977221... 37 trxs, lib: 977180
batgirl generated block c20d3a35... #977222... 36 trxs, lib: 977180
batgirl generated block ed8d3a54... #977223... 37 trxs, lib: 977180
batgirl generated block 116323be... #977224... 36 trxs, lib: 977180
batgirl generated block c9d763d0... #977225... 35 trxs, lib: 977180
batgirl generated block b8c9284f... #977226... 24 trxs, lib: 977180

Accepted: 400

Client : Intel

gargamel generated block 1317f42b... #975994 @ 2018-04-12T14:51:59.000 with 13 trxs, lib: 975946
batgirl generated block b30ce053... #975995... 24 trxs, lib: 975958
batgirl generated block d167c02a... #975996... 24 trxs, lib: 975958
batgirl generated block f905a24d... #975997... 24 trxs, lib: 975958
batgirl generated block 5dc01ddf... #975998... 23 trxs, lib: 975958
batgirl generated block b0cb5f06... #975999... 24 trxs, lib: 975958
batgirl generated block cb75ed06... #976000... 24 trxs, lib: 975958
batgirl generated block 11a2bd13... #976001... 24 trxs, lib: 975958
batgirl generated block 5b13e951... #976002... 23 trxs, lib: 975958
batgirl generated block 0de45bf3... #976003... 22 trxs, lib: 975958
batgirl generated block 66287858... #976004... 24 trxs, lib: 975958
batgirl generated block 96733ea0... #976005... 22 trxs, lib: 975958
batgirl generated block 90f284ea... #976006... 24 trxs, lib: 975958
noheart generated block 7dfa12ab... #976008... 24 trxs, lib: 975970
noheart generated block e9908e45... #976009... 24 trxs, lib: 975970
noheart generated block 83dad3bf... #976010... 24 trxs, lib: 975970
noheart generated block 159ca01a... #976011... 23 trxs, lib: 975970

Accepted: 400

400 transactions - multiple threads

Producer AMD

Client : AMD

gargamel generated block 4a5a65c0... #987934 @ 2018-04-12T16:31:30.500 with 168 trxs, lib: 987895
gargamel generated block d425df94... #987935... 55 trxs, lib: 987895
gargamel generated block 7541a6bc... #987936... 169 trxs, lib: 987895
gargamel generated block df27c1b1... #987938... 8 trxs, lib: 987895

Accepted: 400

Client : Intel

noheart generated block 2bf9aca9... #975512 @ 2018-04-12T14:47:58.000 with 15 trxs, lib: 975466
noheart generated block 89eb0170... #975513... 191 trxs, lib: 975466
noheart generated block 3ad8c5c4... #975514... 5 trxs, lib: 975466
robin generated block 65afde20... #975515... 165 trxs, lib: 975478
robin generated block 3050f077... #975517... 13 trxs, lib: 975478
robin generated block 064b05d6... #975519... 11 trxs, lib: 975478

Accepted: 400

Producer Intel

Client : AMD

batgirl generated block 22c8e863... #989741 @ 2018-04-12T16:46:35.500 with 63 trxs, lib: 989704
batgirl generated block 7e37c62f... #989742... 144 trxs, lib: 989704
batgirl generated block ace33dd9... #989743... 73 trxs, lib: 989704
batgirl generated block 2f0b5b5b... #989744... 112 trxs, lib: 989704
batgirl generated block 40c63b87... #989745... 8 trxs, lib: 989704

Accepted: 400

Client : Intel

robin generated block 00888535... #989918 @ 2018-04-12T16:48:04.000 with 7 trxs, lib: 989872
robin generated block 0dac6abf... #989919... 197 trxs, lib: 989872
eosio generated block 55c4c6bf... #989921... 189 trxs, lib: 989884
eosio generated block ca1684f6... #989923... 7 trxs, lib: 989884

Accepted: 400

4) Two full nodes, one balancer and one hidden producer

The client run cleos which connects to the load balancer and the connection is then redirected to one of the full nodes using the round robin algorithm.

400 transactions - single thread

Producer Intel

Client : AMD

robin generated block 3e54884c... #1009891 @ 2018-04-12T19:34:52.500 with 5 trxs, lib: 1009844
robin generated block b327b7fe... #1009892... 31 trxs, lib: 1009844
eosio generated block b6a13e6c... #1009894... 29 trxs, lib: 1009856
eosio generated block d2b78eea... #1009895... 29 trxs, lib: 1009856
eosio generated block 2780d072... #1009898... 29 trxs, lib: 1009856
eosio generated block 9890a2e0... #1009899... 31 trxs, lib: 1009856
eosio generated block 525b7610... #1009902... 29 trxs, lib: 1009856
eosio generated block 83e6b94b... #1009903... 31 trxs, lib: 1009856
storm generated block 9dc05f72... #1009905... 6 trxs, lib: 1009868

Accepted: 220

Client : Intel

robin generated block 4dbd01f5... #1009162 @ 2018-04-12T19:28:48.000 with 23 trxs, lib: 1009124
robin generated block 51b1642e... #1009162... 23 trxs, lib: 1009124
robin generated block 43074497... #1009164... 23 trxs, lib: 1009124
robin generated block 76b37ce9... #1009165... 24 trxs, lib: 1009124
robin generated block f82ec33c... #1009166... 22 trxs, lib: 1009124
robin generated block 47efb29e... #1009167... 23 trxs, lib: 1009124
robin generated block 26e96977... #1009168... 22 trxs, lib: 1009124
robin generated block dc71cfdb... #1009169... 24 trxs, lib: 1009124
robin generated block d17920bd... #1009170... 23 trxs, lib: 1009124
robin generated block ecc93fc3... #1009171... 23 trxs, lib: 1009124
robin generated block 95cff703... #1009172... 23 trxs, lib: 1009124
eosio generated block 564990e9... #1009173... 22 trxs, lib: 1009136
eosio generated block f71e4cba... #1009174... 24 trxs, lib: 1009136
eosio generated block 5ecaf0cb... #1009175... 22 trxs, lib: 1009136
eosio generated block 51dda329... #1009176... 23 trxs, lib: 1009136
eosio generated block 2faa2d7e... #1009177... 23 trxs, lib: 1009136
eosio generated block 3b3c968d... #1009178... 22 trxs, lib: 1009136
eosio generated block 9f436a6c... #1009179... 11 trxs, lib: 1009136

Accepted: 400

400 transactions - multiple threads

Producer Intel

Client : AMD

gargamel generated block 76088e6e... #1011221 @ 2018-04-12T19:45:57.500 with 23 trxs, lib: 1011176
gargamel generated block 22c47c4c... #1011222... 29 trxs, lib: 1011176
gargamel generated block 270fcab9... #1011224... 29 trxs, lib: 1011176
batgirl generated block 28294504... #1011225... 29 trxs, lib: 1011188
batgirl generated block 1e07003b... #1011227... 29 trxs, lib: 1011188
batgirl generated block 9979bfe0... #1011229... 28 trxs, lib: 1011188
batgirl generated block 82a924cc... #1011232... 29 trxs, lib: 1011188
batgirl generated block ad943338... #1011233... 28 trxs, lib: 1011188
batgirl generated block fb482b21... #1011234... 26 trxs, lib: 1011188

Accepted: 250

Client : Intel

noheart generated block 865c19c4... #1010166 @ 2018-04-12T19:37:10.000 with 38 trxs, lib: 1010120
noheart generated block 16f541d6... #1010167... 266 trxs, lib: 1010120
robin generated block f0d55775... #1010169... 96 trxs, lib: 1010132

Accepted: 400

5) Six nodeos on two full nodes, one balancer and one hidden producer

The client run cleos which connects to the load balancer and the connection is then redirected to one of the full nodes using the round robin algorithm.

400 transactions - single thread

Producer AMD

Client : AMD

Note that the machine connects to the load balancer instead of "localhost" as with the other AMD - AMD tests.

batgirl generated block c86380e6... #440608 @ 2018-04-09T09:30:23.500 with 13 trxs, lib: 440571
batgirl generated block 9e3be2ea... #440610... 28 trxs, lib: 440571
batgirl generated block 2a7839d6... #440612... 29 trxs, lib: 440571
batgirl generated block 1fe1e4c1... #440613... 29 trxs, lib: 440571
batgirl generated block 696635e5... #440615... 29 trxs, lib: 440571
batgirl generated block a169f276... #440616... 29 trxs, lib: 440571
batgirl generated block bba16c71... #440618... 29 trxs, lib: 440571
batgirl generated block 4ac5e117... #440619... 29 trxs, lib: 440571
noheart generated block 2f7b65bf... #440620... 29 trxs, lib: 440583
noheart generated block 5ba2e2f8... #440621... 29 trxs, lib: 440583
noheart generated block ae11e657... #440622... 4 trxs, lib: 440583

Accepted: 277

Client : Intel

eosio generated block 48f9e16d... #440935 @ 2018-04-09T09:33:07.000 with 18 trxs, lib: 440895
eosio generated block d92bc787... #440936... 23 trxs, lib: 440895
eosio generated block 6115797e... #440937... 24 trxs, lib: 440895
eosio generated block 8e01a0e7... #440938... 23 trxs, lib: 440895
eosio generated block 3c7c8dc1... #440939... 22 trxs, lib: 440895
eosio generated block 20d96484... #440940... 23 trxs, lib: 440895
eosio generated block ede12df7... #440941... 23 trxs, lib: 440895
eosio generated block 9c246a10... #440942... 23 trxs, lib: 440895
eosio generated block 96cc48aa... #440943... 23 trxs, lib: 440895
storm generated block 0722e343... #440944... 23 trxs, lib: 440907
storm generated block 0d234a47... #440945... 23 trxs, lib: 440907
storm generated block 11313d91... #440946... 23 trxs, lib: 440907
storm generated block 81c2ec3e... #440947... 23 trxs, lib: 440907
storm generated block fd9f6558... #440948... 23 trxs, lib: 440907
storm generated block 03ad3c89... #440949... 23 trxs, lib: 440907
storm generated block e1970063... #440950... 23 trxs, lib: 440907
storm generated block 5c4bcda4... #440951... 23 trxs, lib: 440907
storm generated block 7cdc16c8... #440952... 14 trxs, lib: 440907

Accepted: 400:

400 transactions - multiple threads

Producer Intel

Client : Intel

gargamel generated block 93595d2e... #1076472 @ 2018-04-13T04:50:07.000 with 164 trxs, lib: 1076432
gargamel generated block 5a1b8b9e... #1076473... 171 trxs, lib: 1076432
gargamel generated block aef2734b... #1076474... 65 trxs, lib: 1076432

Accepted: 400

Increasing about of transactions to 800

noheart generated block 5ab7f541... #1077963 @ 2018-04-13T05:02:54.000 with 99 trxs, lib: 1077925
noheart generated block 27d98fa3... #1077964... 267 trxs, lib: 1077925
noheart generated block 51ffd3b9... #1077965... 65 trxs, lib: 1077925
noheart generated block a151fc51... #1077966... 301 trxs, lib: 1077925
noheart generated block 15ed9bf6... #1077968... 9 trxs, lib: 1077925
noheart generated block aa95cbc1... #1077970... 59 trxs, lib: 1077925

Accepted transactions: 800

Increasing about of transactions to 1200

batgirl generated block 75b7305d... #1078526 @ 2018-04-13T05:07:35.500 with 62 trxs, lib: 1078489
batgirl generated block 4f9507b5... #1078527... 248 trxs, lib: 1078489
batgirl generated block 2050f7bb... #1078528... 129 trxs, lib: 1078489
batgirl generated block 8ed44119... #1078529... 287 trxs, lib: 1078489
batgirl generated block 118f12fc... #1078530... 253 trxs, lib: 1078489
batgirl generated block a74369b3... #1078531... 91 trxs, lib: 1078489
batgirl generated block 813a3d6d... #1078532... 1 trxs, lib: 1078489
batgirl generated block 7b145dcb... #1078533... 2 trxs, lib: 1078489
noheart generated block c3f2c546... #1078547... 34 trxs, lib: 1078501
robin generated block 7c3edb6d... #1078561... 7 trxs, lib: 1078513
eosio generated block e6740849... #1078562... 12 trxs, lib: 1078525
eosio generated block 27829485... #1078570... 1 trxs, lib: 1078525
storm generated block 1ee016b4... #1078576... 4 trxs, lib: 1078537
gargamel generated block df538f17... #1078592... 27 trxs, lib: 1078549
gargamel generated block a61bcaaf... #1078594... 3 trxs, lib: 1078549

Accepted transactions: 1161

Got some messages like this (which I believe accounts for the 39 missing transactions)

Please increase the expiration time of your transaction!
Error Details:
Transaction is expired, now is 2018-04-13T05:32:24.500, expiration is ${trx.exp}
Error 3030022: Expired Transaction

6) Bonus

600 transactions - multiple threads

Here I ran both the producing nodeos and three nodes accepting incoming transactions on the AMD machine. All nodes connected to each other and the client connected through the balancer.

eosio generated block c5460277... #441227 @ 2018-04-09T09:35:33.000 with 29 trxs, lib: 441183
eosio generated block d0f96953... #441228... 31 trxs, lib: 441183
eosio generated block 66358b58... #441231... 29 trxs, lib: 441183
storm generated block 82329467... #443403... 284 trxs, lib: 443355
gargamel generated block 3d505b1e... #443404... 114 trxs, lib: 443367

Sum: 487

The producer forked off and two of the full nodes got stuck with different errors:

Bad file descriptor

Block not accepted

The only way I found to restore the chain was to shut down all nodes except one, and then resync the other. I believe that this has been addressed here and hopefully it is no longer a problem in master.
We did however find some other issues when running on master in the Jungle testnet, so I will have to wait to re-test this.

Conclusions

It is way too early to draw any conclusions from these tests. Especially regarding what kind of load an eosio blockchain will be able to handle. This article has not even started to scratch the surface. What I have done here cannot be seen as anything more than extremely limited testing. However, I will not let that stop me from speculating:

It seems that my AMD machine might be responsible for the dropped transactions reported above. I do not know why - but when I exclusively use the Intel machines the chain seems to be more stable and I no longer get dropped transactions that I cannot account for. I will try to find time to re-install that machine and see if I can get it to work better with a more clean setup. It does seem to work when I run AMD - AMD, meaning it could possibly be something with the network interface.

Once I started playing with the different timeout settings in the balancer I could increase the amount of transactions put into the blocks quite heavily. This needs to be more thoroughly investigated and I will try to prioritize it as we move forward.

The following screenshot was taken while sending a few hundred tx/s to the four instances of nodeos running on that machine.

Before running these tests I thought that the limiting factor on how many transactions we could fit into each block would be the single core performance of our producing machines. This might be true at a later stage, but not exactly in the way I have seen it before. To get to that point seems to require a more advanced setup than I had previously assumed.

Each individual instance of nodeos acting as a full node and accepting incoming transactions is quite easy to spam down, whereby it stops accepting transactions for a few milliseconds. It seems that a possible work-around here is to run multiple instances of nodeos on our full nodes, and balance the load between them. I will test this hypothesis with Charles.

What's next?

More than anything, I hope this post will be seen as an invitation to come join the community driven testnets. Please pick a testnet and help us test both performance and stability, as well as our setups. Some suggestions:

Running in multiple locations

The blockchain doesn't care if a block producer is replaced by another. But to us as individual producers it is very important that we make sure our setups are as safe and secure as we can make them. Producers looking to run on bare metal will probably want to run producing nodes at multiple physical locations to decentralize within our own operations. However, we can only allow one producing machine to produce each block. Currently within the eosio software there is no functionality to handle this.

Handover functionality

We are in discussions with block.one regarding them helping the community to build some sort of fail-over mechanism. There are multiple ways of doing this. If we haven't either received something or built something ourselves by mid May: I am planning on putting together something simple to help us at least automate the process a bit.

Removing the cap

Error 3020004: block exhausted allowed resources
Error Details:
Block has insufficient cpu resources

When pushing transactions it is very easy to hit the a cpu limit. We believe this can be configured by setting the max_block_cpu_usage on the eosio_parameters type on the eosio.system contract.
We have not found time to investigate this further. Instead I started spamming transactions from multiple users.

DDoS protection

Charles (@ankh2054) and I are planning to start looking at different solutions to help protect BPs from incoming DDoS attacks. Stay tuned for that article.

Virtualization

How big of an impact (if any) on performance does running both clients and servers as virtual machines have? I'm planning on finding time to start testing this sometime during the coming weeks. If you have some ideas or want to help, please get in contact with me.

Docker

Bohdan of @cryptolions will look into testing performance and load using a dockerized setup.

Keep track of resources

It would be highly appreciated if someone could help run more advanced performance tests and also keep a close eye on the load on the servers and clients involved. I have some ideas that could possibly help if someone wants to run these tests but don't know how; feel free to contact me on telegram if you need input.

Links

Community Testnet (https://t.me/CommunityTestnet)
Jungle testnet (https://t.me/jungletestnet)
Scholar (https://t.me/ScholarTestnet)
Arrowhead (https://t.me/joinchat/GjJY7BJrnbIW_S67Zchexg)