The ethereum 2.0 testnet, Medalla, has not been finalizing for about 1,000 epochs, with participation dropping to as low as 1.5%.
There may be a fork as the crash happens while processing block 515616 (5d6fc4e3).
Apparently people left to Zinken, the ethereum 2.0 dry run, with Preston van Loon, a Prysm dev, stating:
“We enjoyed a successful dress rehearsal launch with the short lived Zinken testnet and now it is time to bring our attention back to Medalla.
If you were running a Medalla validator and turned it off, please turn it back on. This network hasn’t been able to finalize since Zinken started and we suspect that this is due to users abandoning Medalla for Zinken.
If you don’t intend to return to Medalla, please complete a voluntary exit so your validators will no longer contribute to participation issues.”
There are no real money at stake here, so node runners have no incentive to politely get out of the network. Thus they have caused problems for everyone. However Raul Jordan, another Prysm dev, says:
“This incident has been really good for us to see more opportunities for optimization and small bugs that don’t manifest except during times like this. Like for example, certain caches not working.
Nishant [Das, a Prysm dev] also discovered the root cause of the sync loop bug some stakers are seeing recently [pictured above].
There are only 3 more features to do before Prysm is feature complete for mainnet. Peer scoring, common slashing protection export format, and aligning to the eth2 api standard. Then, only security + improvements till launch.”
This matter was discussed in an eth 2.0 devs meeting, with the notes stating:
“Medalla testnet is not v1.0 conformant. We also expect most people to turn off their nodes when mainnet comes. Therefore, we are considering a new long-running v1.0 testnet/devnet without such wide participation.
There have been suggestions to upgrade Medalla to v1.0 spec. [Preston] Even though we might see a big drop in participation, we will still be able to gather syncing data etc and Medalla can still serve a purpose.
As of today, participation is low, but worth running for a few weeks to see the inactivity leak in action and otherwise stress-test clients.”
The version one of the spec is the final version prior to the mainnet launch with candidate 0 released already.
Medalla therefore needs an upgrade, or a new testnet needs to be launched to be kept running effectively indefinitely as people test their staking setup or other aspects even after the mainnet is live.
The problem is to resurrect medalla might take quite some time with an eth dev stating:
“It takes 21 days of non finality to leak 50 % of the balance of a validator at which it will be exited. So I think around November 3rd Medalla will spring back to life. We have seen this with Altona already.”
November 3rd is close to the expected mainnet launch, but if the issue is indeed that validators turned off just to go to Zinken or because they thought their job was done or because they were bored, then the issue is more that this is all with fake eth, rather than any protocol related matter.
However as Jordan says, the “innocent” situation has brought up client issues with Prysm now apparently needing to do some further work.
One of the issues appears to be that a node runner says he “restarted my node to install some updates, and have been locked out of participation for a few days now.”
The suggestion was to “run off changeCacheKey,” with a new client release expected soonish.
All of this does reveal certain weaknesses in ethereum 2.0 as the action of each validator affects all validators and the network.
In some ways that’s the case in bitcoin too. If 30% of the mining hashrate leaves, then it is more difficult for the remaining miners to find a block, thus block times might rise to an hour or two until difficulty adjusts every two weeks.
In ethereum however difficulty adjusts by the block. So the algo in ethereum very quickly responds to miners’ actions with the end result being that what other miners do does not affect the remaining ones very much.
In bitcoin too there have been situations when hash has dropped even by 50%, like during Black Thursday, and the network has still kept running. More slowly, but moving.
Here, one little bug and you might need two weeks for all to get back up. In addition, the security threshold is much weaker as you need just 33% of validators to corrupt the network or halt it. While in bitcoin you need 51% and even then all you can do is double spend your own coins if others are impatient enough to not wait for the situation to resolve in these sort of circumstances.
That differentiates bitcoin and ethereum quite considerably, with the former generally safer while the latter is still a bit experimental.
