Decentralization opens more opportunities for bugs to occur between components of the system, rather than just bugs within a centralized component. While modern development tools can handle these individual components (or even small sets of integrated components), the continuous testing of entire testnets is a relatively unsupported workflow.
For Avalanche, these holistic integration tests are critical to ensure any potential errors in implementation are caught before they hit deployed networks.
Today, we’re proud to introduce Kurtosis, a complete, end-to-end testing platform dedicated to helping developers build antifragile technologies on Avalanche, even as the network continues its rapid pace of decentralization.
Its team believes that unexpected events–so-called “Black Swans”–are the largest threat to delivering innovative software as networked systems become increasingly complex. They stress the importance of shifting whole system testing earlier and earlier into the development cycle to find and address bugs before they can cause issues on a scale big enough to damage the whole project.
We know testing can be hard to wrap your head around, so here’s a case study of how Kurtosis helped Avalanche as we prepared for mainnet launch.
Avalanche is designed to support high-volume financial applications in production. We place a high value and priority on constant, system-wide testing. These exercises are time-consuming and expensive, as they require developers to deploy entire networks configured for test activity and extensive manual workflows.
The Kurtosis team first built tooling to handle test orchestration. Building on top of the Docker container system, the Kurtosis platform launches multiple lightweight, Docker-based testnets in parallel, one per test. Once testnets are up, Kurtosis runs one test scenario per testnet to guarantee the isolation of tests.
Kurtosis is designed to facilitate development workflows, so it runs locally on developer laptops as well as in continuous integration environments like TravisCI or CircleCI. Developers can catch errors as early as possible in the development cycle and the entire team has a shared source of truth about standards of whole-network functionality.
With Kurtosis, developers have complete control to define, reuse, and modify unique test environments. They can even create complex test scenarios like heterogeneous networks and networks containing a mix of honest and Byzantine (malicious) nodes.
To ensure that tests were catching real, relevant issues, the Kurtosis team worked with the core Ava Labs development team to deploy tests in continuous integration while they prepared the platform for release. This partnership guided some critical Kurtosis features, such as the ability to add and remove Avalanche nodes during the execution of a test.
One error seen early in Avalanche development was unexpected behavior when nodes entered the network and claimed the same identity as an existing node. Kurtosis aimed to guarantee that the network maintains connectivity on every change of code. By providing dynamic testnets, Kurtosis reliably guarantees that rogue nodes entering, exiting, and re-entering with new IDs do not perturb network connectivity.
Since deploying the Kurtosis platform, a set of 7 whole-system integration tests are running on the production mainnet codebase on every code change to test key components, scenarios, and standard user workflows.
Those tests have helped optimize the network activity, ensure safe releases, and hold promise for Kurtosis’ ability to enable rapid, safe developments on Avalanche for years to come.
Kurtosis helped Ava Labs solve the challenges of a platform that runs network-wide tests on every commit. With one success story at its conclusion, the team is ready to expand to the Avalanche community as the guardrails for developers to quickly and safely build everything from decentralized apps (dapps) to custom clients to new chains.
Kurtosis Technologies is now seeking a small number of high-engagement partners who will help shape the platform’s evolution as their individual needs are addressed; if this sounds like you, reach out to the team at [email protected] or kurtosistech.com.