Incident Report

This evening at 7:45 pm local time, I was notified by Telegram bot that my Peerplays witness node was missing blocks. I could not log in to my server, nor even get to the VPS company main page. The VPS reported a connectivity issue and was back online 5 minutes later.

I could connect to my node again, but it was still reporting missing blocks, even the logs were reporting pushing blocks. I noted that I was not getting blocks for other witnesses and decided my node had forked. I restarted and replayed the blockchain. The replay process took 5 minutes, and it took another couple of minutes to catch up missing blocks.

By 8:06 pm we were signing blocks again. In this time, we missed 30 blocks (with only 15 active witnesses at the moment, block slots are coming around more frequently).

I run a testnet node on a different server. I have now set up a backup main node on this server also. Switching will currently be manual, but will investigate automated options, such a port of jesta's script.