Test Variations - ABAKAS

Once you have a base of test automation, it becomes easy to do variations on a test. What if I ran test X with changed configuration Y? What if I ran it with different hardware Z?

These are one of the benefits of a good test automation stack; it becomes fairly easy to run lots of variations on the same thing, and that can result in a lot of learning about how system behavior changes as various things flex. For example, I recently took a performance test we run and tweaked the setup to make it run with different memory allocations - and in not too long I had a much better understanding of how system throughput changes as a result of adding or constraining that particular resource.

This isn't the kind of test i need to run frequently. I run it every once in a while to make sure the system behavior in this area hasn't changed, but it's overkill to run it every single night with a lot of our automation.

So here's the dilemma: Do I check in the test with my modified configuration?

Pros to checking in:

  •  Next time I run it, I'll be sure I have the same setup, so my results are comparable
  • Someone else could run the same test and gather their own information from it (or a variation on it)
 Cons to checking in:

  •  Checked-in code that doesn't run is likely to get stale and stop working as other code changes around it. This goes for test code, too!
  • Someone might try to run it more frequently, and that will take up a lot of machines for very little benefit
There's no particular right answer here. Which one you choose will depend on how hard it was to set up, how much consistency matters, whether it's really likely to stop working, and how frequently you do want to run it. Pick the one that works for you; just make sure it's a choice you make, not something you fall into.

Source: Test-variations - ABAKAS

Comments

Post a Comment

Popular posts from this blog

Software Testing @ Microsoft

Image
In continuation from my previous  series of posts  targeting testing in top few companies from the technology space. After  Software Testing @Facebook  and @Google  we shall now look at Microsoft: Testing@Microsoft Software Development Engineers in Test (SDETs) are usually just called Test and sometimes Software Testing. SDETs are responsible for maintaining high testing and qualityassurance standards for all Microsoft products. Software Development Engineers (SDEs) are often referred to as Software Development. SDEs write the code that drives Microsoft products and upgrades. Very few teams in Microsoft are still using SDETs to do significant amount of manual testing. Manual testing is mainly outsourced. Actually, as early as in 2004, for example, most of the manual testing of MSN Explorer was outsourced. Think about it this way: it just doesn't make sense for test manager to spend their headcount (SDETs) on something can be easily outsourced.  Until 2005, Microso
Read more

Trim / Remove spaces in Xpath?

Image
If you are wondering how you can trim the spaces in your xpath, this is how: The normalize-space function strips leading and trailing white-space from a string, replaces sequences of white-space characters by a single space, and returns the resulting string. Example : Which one is better? //td[starts-with(normalize-space(),'Text to Trim')] //td[starts-with(normalize-space(text()),' Text to Trim ')] The second option is better as it targets at the specific nodes More string functions in XPATH: Functions Description starts-with( string1 ,  string2 ) Returns true if the first string starts with the second string. contains( string1 ,  string2 ) Returns true if the first string contains the second string. substring( string offset length ) Returns a section of the string. The section starts at  offset (which is a number), and is as long as the value provided at  length. substring-before( string1 ,  string2 ) Returns the part of  string1  up unti
Read more