34
How much flakiness do you tolerate in end to end tests? (programming.dev)
submitted 2 years ago* (last edited 2 years ago) by to c/experienced_devs@programming.dev
22 comments fedilink hide all child comments
 

End to end and smoke tests give a really valuable angle on what the app is doing and can warn you about failures before they happen. However, because they're working with a live app and a live database over a live network, they can introduce a lot of flakiness. Beyond just changes to the app, different data in the environment or other issues can cause a smoke test failure.

How do you handle the inherent flakiness of testing against a live app?

When do you run smokes? On every phoenix branch? Pre-prod? Prod only?

Who fixes the issues that the smokes find?

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 years ago (2 children)

I'm a fan of randomizing the test order. That helps catch ordering issues early.

Also, it's usually valuable to have E2E tests all be as completely independent as possible so it's impossible for one to affect another. Have each one spin up the whole system, even though it takes longer. Use more parallelism, use dozens of VMs each running a fraction of the tests rather than trying to get the sequential time down.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] [S] 1 point 2 years ago

    Wherever possible, this is a good idea. The campsite rule - tests don't touch data they didn't bring with them - helps as well.

    However, many end to end tests exist as a pipeline, especially for entities that are core to the business function of the app. Cramming all sequentiality into single tests will give you all the problems described, but in a giant single test that you need to fish out the result for.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 2 years ago*

    The problem with randomising the test order is that it compromises the reproducibility of results. If there are ordering issues, then your tests will sometimes fail and sometimes pass, but will developers look at that and think "ah there must be an ordering issue" or will they think "damn these flaky tests, guess I'd better rerun the pipeline"?

  • permalink
  • fedilink
  • source
  • parent