Test FLARE (Test Flakiness Automated Reproduction and Explanation)

The cost of software failures is a huge burden to the worldwide economy that was estimated to be at least £1.3 trillion in 2017. Consequently, software testing, a vital defence against failures, contributes to a large proportion of software development effort and cost. Flaky tests are a particular strain on resources allocated to software development, because they intermittently pass and fail without changes to tests or project code, with often maddening, non-obvious causes. Flaky tests are tests that fundamentally do not always tell the truth: they can fail when code is working, and pass when it isn't. Because developers can no longer trust the results of their tests, they are unable to gain confidence that software is working correctly, potentially exposing end-users to the consequences of software failures. Flaky tests are a common occurrence in industry, significantly disrupting software development - even for companies with the greatest amount of resources to tackle them, such as Microsoft, Facebook, and Google.

A test can produce different pass/fail (i.e., flaky) outcomes because of differing, unpredicted ways that the execution environment in which it runs interacts with its behaviour and/or the code that it tests. For instance, a machine may be experiencing a heavy concurrent task load, causing it to execute tests slowly, sometimes triggering timeouts in the code under test, and sometimes not. Or, network access is erratic on the testing infrastructure, meaning the availability of network resources may be compromised. Or, a program under test's logic is time and date dependent. These are just a few real examples of the different ways in which tests can be flaky. For some environmental conditions, the test passes, but in an alternative context, the same test fails.

To remove flaky test behaviour, a developer has to modify test code or the code that it tests to control for aspects of its execution environment; i.e., the potential sources of its...