The evolution of acceptance test automation, or at least my experience of it has been something like this:
2005 - 2008 : QA runs automated test on their machine pior to raising deployment ticket
2009 - 2012 : CI runs automated tests - downstream, everybody ignores them
2012 - present : Continuous Delivery…UI tests broken, no deployments…now we have your attention!
So hopefully you have arrived at stage 3 and have done the following to ensure your UI tests are as resilient as possible:
- Followed Martin Fowler’s test pyramid and not implemented every conceivable system test via Selenium
- Implemented sane retry logic to find UI elements
- Isolated your UI using stub backends to guard against unexpected data
- Added spoonfuls of helpful debug logging to highlight issues
But perhaps a couple of your tests still appear now and again be nondeterministic and people are getting frustrated.
This can happen, UI testing is hard and no matter how much you defend against it peculiarities of the runtime environment can conspire against you.
In this instance it might be handy to have ‘one more go’ when you get some failures and Cucumber’s rerun formatter allows this.
Caveat - generally I would say a nondeterminstic test should either be fixed for good or deleted however sometimes the world is just not that perfect, maybe for example you aren’t able to improve underlying infrastrucre issues etc etc. Hence this may still be a valid course of action.
This is about the simplest example I could come up with.
The test will fail 50% of the time, allowing you to see the rerun kicking in on selected runs.
You can run the test using
bundle exec rake
The test should be run for a second time if it fails and should terminate with an appropriate exit code.
You can see some examples of this working in the project’s Travis build.
Another useful thing might be to run any failing tests from previous CI runs first and you could probably adapt this approach to do just that!
I just thought I would illustrate with an example what others have already highlighted: