Test Automation has always been a challenge when it comes to running your suites within the time frame especially in an Agile mode with more builds coming the testing way.
So here is a quick strategy that we tried and seems to be working well for now:
What it used to be:
- Entire test automation runs for 8-10 hours
- One big test suite or a few smaller test suites
- Reusable code being used a lot (which was good , but also had a repercussion of redundant flows)
- Robust suites which broke less but after complete runs the failures where not always application bugs but automation errors.
- The whole idea was how much test coverage do we have
- Metrics were based on number of tests run/passed/coverage and so on..
What it is now:
- Continuous Integration Builds
- Test suites do not run that long instead there are dedicated 1 hour suites (To keep the suite below the 60 minute time frame and also have the apt coverage , these would be targeted suites for a particular functionality/stories and not cover anything that the previous suite did)
- The idea is that these one hour suites would be targeted suites and at any cost will not run over an hour on a single automation box
- The one hour suites will not have any redundant flows nor will there be at any time more than 5 such suites in our latest test set.
- The one hour suites will have less than 5% automation defects and should be minimized to almost zilch.
- Metrics are much better now and the tests actually fail only when there is a genuine defect and not an automation failure.
Comments and suggestions are always welcome , until then Happy testing!