EOS Mainnet Faltered For Five Hours

Block.one successfully released version 1.0 of open source EOSIO blockchain software on June 2. EOSIO is an operating system on which scalable DACs (Decentralized Autonomous Communities) can easily be built, launched, and governed.

However, it took more than a week for tests to be conducted on mainnet, before block producers took it live. However, the EOS mainnet faltered 48 hours after it went live. The EOS software “paused” the mainnet to prevent a fork.

As transaction began to freeze on the mainnet at 9:56 UTC (~11:00 am London time), one of the EOS’ block producer candidates reported that the EOS network monitor showed multiple BPs in “red”. In other words, this indicated issues with the application programming interfaces (APIs). Some of the members of the EOS community even raised concerns of a possible DDoS attack.

EOS (EOS), which is the highest-funded initial coin offering as of now, raised approximately $4 billion before its launch. The EOS community considers the blockchain as a big competitor to Ethereum, which is the uncrowned king in hosting DApps. EOS (EOS) has claimed to offer features such as inter-blockchain communication.

An urgent meeting was held online when it was decided that all standby block producers would pause their nodes and backup data that could assist in identifying the root cause of the problem. The EOS developer team identified a bug related to the handling of deferred transactions. A patch was (v1.0.5) issued to rectify the error. Almost five hours later, the EOS mainnet went live. It is now up and running, although EOS team has cautioned not to transact for a few more hours. The mainnet is currently under close supervision by developers and block producers.

The entire time line of events, from the time mainnet froze to the time it went live again, is given below:

“AT 9:56 UTC the EOS Mainnet halted. At 10:01 UTC Block Producers and many Standby Nodes joined together on an international conference call to identify and fix the issue.

At 10:57 UTC The choice was made that all Standby Block Producers would stop their nodes and backup any information that could helpful in diagnosing the problem.

At 11:02 UTC, a method to unpause the chain was formulated and is currently underway. Normal EOS chain functions should be available within 3 to 6 hours from the time of the publication. Incoming connections to the network are not being accepted while the work is taking place. Please wait for an update from the Top 21 Block Producers before attempting EOS mainnet transactions.

At 13:03 UTC we identified the root cause and are working on a fix. A full release outlining the root cause will be issued once we’ve completed the work.

At 13:09 UTC BlockOne is issuing a patch in tag 1.0.5 which will be rolling out in the next few hours. The following steps will be taken:
A nodeos fix will be released.
The block producer nodes will be upgraded.
Block producers which had locally produced blocks will need to resync up until block 1027926, with production paused, and enable-stale-production disabled.
All nodes will upgrade to 1.0.5
Block producers will sync to each other to resume production
Chain will be opened up for public access.

At 13:28 UTC The root cause was due to an issue with how deferred transactions were handled.

At 13:36 UTC Pull Request 4158: https://github.com/EOSIO/eos/pull/4158
The EOSIO software is designed in such a way that when an error of this type occurs the chain will pause to prevent a hard fork. All block producers were halted when they detected a break in consensus. A fix was written, and is being tested and will be rolled out by the block producers community.

13:36 UTC — Update #4: Pull Request 4158: https://github.com/EOSIO/eos/pull/4158

At 14:13 UTC Tag 1.0.5 was issued by Block.One: https://github.com/EOS-Mainnet/eos/tree/mainnet-1.0.5

At 14:48 UTC the network went live. The Block Producers and Standby Nodes responsible for responding to this fix were able to work with Block.One, identify the issue, and merge a fix within 5 hours’ time.”

The EOS developer team should be commended for providing a live commentary of what was going on. This avoided a panic situation. By and large, the EOS token remained steady