Roundup Round III

Over the last month and a half we saw vigorous ongoing research and development on all sides of the Ethereum roadmap, and progress is rapidly starting to translate into real results that can be run and verified inside of an Ethereum client.

On Metropolis:

Agendas for core dev meetings 15 and 16 here: https://github.com/ethereum/pm/issues/13 and https://github.com/ethereum/pm/issues/14
List of accepted EIPs here: https://github.com/ethereum/EIPs/blob/master/README.md (mostly agreed on, though we are still going back and forth on details such as gas costs)
Most of the EIPs have been implemented in C++ and Python and are being implemented in other clients; tests are being actively written and added here https://github.com/ethereum/tests/tree/develop/GeneralStateTests (see the various branches for different tests)
Clients that have not implemented GeneralStateTests could provide a docker image for Hive testing (contact Martin Swende). All GeneralStateTests are being converted into BlockChainTests and run on Hive.
A release date has still not been finalized; the general consensus is to wait until all tests are passing on major clients before setting one. Because of the rapid increases in block difficulty, the ice age continues to be delayed; current prognosis assuming no further increases (arguably a very pessimistic estimate) is that block times will not exceed 20s until July 12, and will not exceed 30s until Sep 12.
The Ethereum blockchain has hit several new all-time highs:

Difficulty (450 TH) and hashrate (28.5 TH)
Transactions per day (187115, or ~2.16 per sec)
Gas usage per day has not yet reached the all-time high of Jun 18, when the blockchain was heavily spammed as part of the DAO attack and various counterattacks, but is nearing it with an 11-month high of 10.7 billion per day. That’s 1991878 gas per block, or ~45% full blocks for the day (reminder: gas limits are dynamically adjusting, so congestion with rapidly increasing fees is not likely) . Uncle rate on that day was only ~7.4%.
On various side projects:

ENS has been deployed, and auctions are ongoing.
Whisper is getting a proper API, which is in alignment with our general RPC, the API should be ready soon and a workable whisper version will be released.
Swarm has made a number of significant improvements, including (i) supporting directory upload and download via the http interface, (ii) full FUSE support, (iii) a new protocol pss for node-to-node messaging, (iv) replacing the chunk hash with a Merkle tree hash to enable more efficient data inclusion proofs. Progress toward POC3 is going at full steam.
Pyethereum development has picked up quickly:

Jan Xie and his team have successfully synced a pyethapp node to the most recent block on the mainnet.
Several bugs in the implementation have been fixed, and the client is now passing all state tests and most pre-Metropolis blockchain tests. Work to find the remaining issues is ongoing.
Most Metropolis EIPs have been implemented, including the four new precompiles.
The tester module has been revamped so that it is fully based on the Chain module, and a new and more convenient interface has been added, including functionality such as creating state tests.
Casper research is now in the process of fine-tuning the incentives for liveness, and implementing the logic inside of pyethereum. This includes:

The validator daemon
The hybrid Casper fork choice rule
Casper contracts including tests
Signature validation code, including for ECDSA and quantum-resistant hash-ladder signatures
Other research stuff:

Substantial work has been done on sharding, and specifically the dreaded “data availability problem”; see here: https://github.com/ethereum/research/wiki/A-note-on-data-availability-and-erasure-coding
The Sharding FAQ has been updated
Greg Colvin’s “EVM 1.5” proposals are now EIP drafts 615 and 616
Geth development:

Removed the artificial 20 shannon minimum gas limit from the transaction pool, enabling Geth nodes to accept and relay arbitrarily priced transactions. This should help enable a real gas market.
Identified a sync regression for HDD users, resulting in a heavy rewrite of fast sync code, making it much more robust and stable.
Identified and fixed a few EVM bottlenecks, causing complex contracts to execute up to 60% faster doing up to 95% less memory allocations.
Polishing the light client and its mobile bindings in collaboration with Status, Walleth and others. Preliminary proof-of-concept done for light client event retrievals.
Working on a new filtering mechanism, with very promising results, reducing the time required to filter the entire blockchain for contract events from minutes to seconds.
Investigating a new mining strategy, which should move transaction processing for miners completely parallel to mining, hence reducing the overhead of transaction inclusion to 0. This should permit miners to lower fees without transactions impacting the probability of finding blocks and/or having them included.
Slowly adding minor polishes to Puppeth, Clique and Rinkeby, such as a tiered faucet for Ether withdrawals, configurable gas dynamics for private network miners and automatic ENS integration.
An external security-audit has been performed by Truesec AB. No critical issues were found.
C++ development:

the monitoring node achieved a full sync to the mainnet
metropolis EIPs are almost complete
work on fast sync and snapshot sync has begun
Testeth tool has new option –statediff for debug information on a praticular test case as well as new test format GeneralStateTests (enhancement of the old statetests) [4074]
Remix:

almost finished a new static analysis module that can detect reentrancy bugs (externally contributed by “soad003”) [508]
added a folder view to show open files inside folders – files imported via e.g. import “github.com/ethereum/solidity/std/StandardToken.sol”; are now visible in the folder view [449]
debugger can now display mappings [498]
rearranged tabs to provide better overview [496]
Solidity:

we released the unified standard json interface to interact with the compiler [1639]
added “interface contract” feature[1688]
added some more safety checks: statements without effect, and unary plus and unused variables [2139, 2152, 2199]
further work on the new intermediate language including web assembly and EVM1.5 backends [2129 and lots more]
almost finished work on exporting and especially also importing the AST (this will allow mutation testing and lots of other extensions) [1810 and others]
Mist:

Decouple geth specific hard-coded node and network handling, allow for arbitrary nodes and networks defined by local or remote clientBinaries.json
Refactor of the settings backend in preparation for a settings UI
Added macOS and Windows Code-Signing, preparing github/swarm based auto-updater
Many of the audit findings will incorporated in the coming releases
Changes to the provider will make Mist and other Projects like MetaMask and status.im more future proof, as developers don’t need to depend on high level libraries not changing over time. There will be a few announcements concerning the new provider. Discussion about that is happening at https://github.com/ethereum/interfaces/issues/16

Web3.js

We are currently adding generation and signing with private keys right into web3.js. So that you can easily create wallets and sign messages in your dapps. You can find the current experimental docs here: http://web3js.readthedocs.io/en/1.0/web3-eth-accounts.html
The next steps will be adding @maiavictor’s swarm library and the new whisper API and the new web3.js should be ready for test drive by the community.