Secured #3: Security Teams | Ethereum Foundation Blog

Over the past year, the Ethereum Foundation has significantly grown its team of dedicated security researchers and engineers. Members have joined from a variety of backgrounds ranging from cryptography, security architecture, risk management, exploit development as well as having worked on red and blue teams. The members come from different fields and have worked on securing everything from the internet services we all depend on each day, to national healthcare systems and central banks.

As The Merge approaches, a lot of effort from the team is spent analyzing, auditing and researching the Consensus Layer in various ways as well as The Merge itself. A sample of the work is found below.

Client Implementation Audits 🛡️

Team members audit the various client implementations with a variety of tools and techniques.

Automated Scans 🤖

Automated scans for codebases aim to catch low hanging fruit such as dependency vulnerabilities (and potential vulnerabilities) or improvement areas in code. Some of the tools being used for static analysis are CodeQL, semgrep, ErrorProne and Nosy.

As there are many different languages used between the clients, we use both generic and language specific scanners for the codebases and images. These are interconnected through a system that analyzes and reports new findings from all tools into relevant channels. These automated scans make it possible to quickly get reports about issues that potential adversaries are likely to easily find, thus increasing the chance of fixing issues before they can be exploited.

Manual Audits 🔨

Manual audits of components of the stack are also an important technique. These efforts include auditing critical shared dependencies (BLS), libp2p, new functionality in hardforks (eg. sync committees in Altair), a thorough audit into a specific client implementation, or auditing L2s and bridges.

Additionally, when vulnerabilities are reported via the Ethereum Bug Bounty Program, researchers can cross-check issues against all clients to see if they are also affected by the reported issue.

Third Party Audits 🧑‍🔧

At times, third party firms are engaged to audit various components. Third party audits are used to get external eyes on new clients, updated protocol specifications, upcoming network upgrades, or anything else deemed high-value.

During third party audits, software developers and our team’s security researchers collaborate with the auditors to educate and assist throughout.

Fuzzing 🦾

There are many ongoing fuzzing efforts led by our security researchers, members of client teams, as well as contributors in the ecosystem. The majority of tooling is open source and runs on dedicated infrastructure. The fuzzers target critical attack surfaces such as RPC handlers, state transition and fork-choice implementations, etc. Additional efforts include Nosy Neighbor (AST based auto fuzz harness generation) which is CI based and built off of the Go Parser library.

Network level simulation and testing 🕸️

Our team’s security researchers build and utilize tools to simulate, test, and attack controlled network environmets. These tools can quickly spin up local and external testnets (“attacknets”) running under various configurations to test exotic scenarios that clients must be hardened against (eg. DDOS, peer segregation, network degradation).

Attacknets provide an efficient and safe environment to quickly test different ideas/attacks in a private setting. Private attacknets cannot be monitored by potential adversaries and allow us to break things without disrupting the user experience of public testnets. In these environments, we regularly utilize disruptive techniques such as thread pausing and network partitioning to further expand the scenarios.

Client and Infrastucture Diversity Research 🔬

Client and infrastructure diversity has received a lot of attention from the community. We have tools in place to monitor the diversity from a client, OS, ISP and crawler statistics. Additionally we analyze network participation rates, attestation timing anomalies and general network health. This information is shared across multiple locations to highlight any potential risks.

Bug Bounty Program 🐛

The EF currently hosts two bug bounty programs; one targeting the Execution Layer and another targeting the Consensus Layer. Members of the security team monitor incoming reports, work to verify their accuracy and impact, and then cross-check any issues against other clients. Recently, we published a disclosure of all previously reported vulnerabilities.

Soon, these two programs will be merged into one, the general platform will be improved, and additional rewards will be provided for bounty hunters. Stay tuned for more information on this soon!

Operational Security 🔒

Operational Security encompasses many efforts at the EF. For example, asset monitoring has been setup which continually monitor infrastructure and domains for known vulnerabilities.

Ethereum Network Monitoring 🩺

A new Ethereum network monitoring system is being developed. This system works similar to a SIEM and is built to listen to and monitor the Ethereum network for pre-configured detection rules as well as dynamic anomaly detection that scans for outlier events. Once in place, this system will provide early warnings about network disruptions in progress or coming up.

Threat Analysis 🩻

Our team conducted a threat analysis focuse on The Merge to identify areas that can improved with respect to security. Within this work, we collected and audited security practices for Code Reviews, Infrastructure Security, Developer Security, Build Security (DAST, SCA and SAST built into CI, etc.), Repository Security, and more from the client teams. Additionally this analysis surveyed how to prevent misinformation, from which disasters may strike, and how the community might recover in various scenrios. Some efforts related to disaster recovery exercises are also of interest.

Ethereum Client Security Group 🤝

As The Merge approaches, we formed a security group that consists of members of client teams working on both the Execution Layer and the Consensus Layer. This group will meet regularly to discuss matters related to security such as vulnerabilities, incidents, best practices, on-going security work, suggestions, etc.

Incident Response 🚒

Blue Team efforts help bridge the gap between the Execution Layer and the Consensus Layer as The Merge moves closer. War rooms for incident response has worked well in the past where chats would spring up with relevant people during incidents, but with The Merge comes new complexity. Further work is being done to (for example) share tooling, create additional debug and triage capabilities and create documentation.

Thank you and get involved 💪

These are some of the efforts currently taking place in various forms, and we’re looking forward to share even more with you in the future!

If you think you’ve found a security vulnerability or any bug, please submit a bug report to the execution layer or consensus layer bug bounty programs! 💜🦄