Dealing with Flaky Tests

Dealing with Flaky Tests

For the past day or so, I have had my head down, tear­ing my hair out, try­ing to de­bug a sig­nif­i­cant block caused by many en­gi­neers’ arch neme­sis: a flaky test.
In my case, it’s ac­tu­ally been sev­eral flaky tests.

A flaky test is any test for your code that only some­times fails.
It shows up when a part of your test re­lies on some­thing non-de­ter­min­is­tic.
Non-de­ter­min­is­tic com­po­nents can in­clude:

Fetching re­sources over a net­work con­nec­tion.

Reading from a ran­dom num­ber gen­er­a­tor.

Using a non-de­ter­min­is­tic asyn­chro­nous sched­uler.

Animations (often use the GPU ).

…and many more.

If your ap­pli­ca­tion in­cludes code that looks any­thing like these, you are at risk for flaky tests.

My use-case hap­pened to be es­pe­cially v…