34
How much flakiness do you tolerate in end to end tests? (programming.dev)
submitted 2 years ago* (last edited 2 years ago) by to c/experienced_devs@programming.dev
22 comments fedilink hide all child comments
 

End to end and smoke tests give a really valuable angle on what the app is doing and can warn you about failures before they happen. However, because they're working with a live app and a live database over a live network, they can introduce a lot of flakiness. Beyond just changes to the app, different data in the environment or other issues can cause a smoke test failure.

How do you handle the inherent flakiness of testing against a live app?

When do you run smokes? On every phoenix branch? Pre-prod? Prod only?

Who fixes the issues that the smokes find?

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 years ago

It depends, but generally as little as possible - nobody has time to inspect flakiness cause on a frequent basis when mindset is to move fast. Within team we identified components and degree of control we have over them to control flakiness

  • permalink
  • fedilink
  • source
    •