you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 years ago (4 children)
  • [–] 7 points 2 years ago*

    All the time. Causes include:

    • Test depends on an external system (database, package manager)
    • Race conditions
    • Failing the test cleared bad state (test expects test data not to be in the system and clears it when it exits)
    • Failing test set up unknown prerequisite (Build 2 tests depends on changes in Build 1 but build system built them out of order)
    • External forces messing with the test runner (test machine going to sleep or running out of resources)

    We call those "flaky tests" and only fail a build if a given test cannot pass after 2 retries. (We also flag the test runs for manual review)

  • source
  • parent