Header Ads

Shall We Execute Our Automation Testcases in Random Order??

When I automate test cases, I always try to reuse what is already available in my test inventory. Always follow do not repeat yourself technique.  If atall components (functions, sub,procedure,testdata etc) are not matching with my requirement, I would create a new one.  Even there is a tendency to reuse the bunch of steps or bunch of tests even testsuite itself. Like me so many like minded persons available in information technology industry for sure.. :D

But as per testing best practice goes, we really need to improve our way of test development. As per that the best practice, we need to make test case independent to each other. There should not be any sharing between two test cases including test data.

Let us see with an example,what I am trying to venture through this post.

An example..
Lets say, I have to update a previously created employee who is attached to an organization.

So what are the manual testcase steps:
  • Create an Organization
  • Search for the same Organization and Create an Employee
  • Find the employee and find the organization and attach Employee to Organization
  • Now Update the same employee 

When I will automate the same flow I will create the structure like-

Testcase1: Create Organization
Testcase2: Create Employee
Testcase3: Attach Employee to Organization
Testcase4: Update Employee

Are test cases written in Big Ball of Mud test pattern correct?
With the advancement of testing,This is one of the basic test pattern where test assets are heavily shared between the test flows. Master driver or test suite can be also shared along with test data.There is no formal boundary or structure maintained to distinct between components[may be testdata,sub,macro,code etc]. Assume a scenario where a component of a test asset is reused by some hundred of test flows.Now to accommodate a new test case or a set of test cases we need to change the component a little bit. From the structural point of view as master drivers or test suites are shared,so we really do not know the impact of changing one component. With this approach it is very tough to find out which components use what all components.The change might fail some unnecessary test scripts in other area. So If we need to randomize the test cases designed in this pattern ,this might trigger failures. It will be needed for patchwork and workaround.So not so efficient way to execute them in random manner.

Are test cases written in spaghetti test pattern correct?
We will bunch these test cases into a batch called test suite and try to execute the same.
For this case we can simply put everything in a flow. Record playback like mechanism. Or we can divide the entire flow to sub flows[test cases] and stitch them. With this approach, each flow heavily dependent on the previous flow. As the state of the application is used by the next test case,so we can not run them individually. They are tightly coupled together. In such case the design pattern is spaghetti test pattern. So all test inventory which is having record playback framework and test cases  are tightly coupled ,we can not execute them in random order.

Are test cases written in Chain Linked test pattern correct?
Now where is Create Organization test case in inventory?? It is inside Organization module. Same like Create Employee will be inside Employee module and updating an employee may be in transaction or some other module. These type of approach where we will have different flows coming from different modules are upgraded from spaghetti test pattern.Each test case is considered as chain and bunch of chain hardly share the test data. But still it depends on the execution order to get the same state of application for the test cases.From the structure it is different from spaghetti model. Here the test cases are shared but the master driver or the test suite or the driver is independent of each other. Maintenance of this type of automation asset and  test data is a pain area for a organization. Mostly keyword driven framework, data driven framework, Modular decomposition approach,Function driven approach,Data oriented function driven approach follow chain linked test pattern. So randomize the test case execution may not suite for them.

The above three approaches talks about more re usability rather than independence of test cases.
 
public class updateEmployee{
@Test public void Create_Organization()
{
//Creation of org code goes here
}
@Test public void Create_Employee()
{
//Creation of Employee code goes here
}
@Test public void Update_Employee()
{
//Updating of Employee code goes here
}
}
With these kind of the approach the Test1 becomes the precondition for Test2 and Test1 and Test2 will both become precondition for Test3. These test cases can not be picked individually rather they needs to be run in a batch.

Advantages
  • No need to repeat the code.Optimum writing but maximum usage of test code
  • Structured or semi structured so finding out the batch and execute them is easy.
  • Mostly follows "do not repeat yourself" logic hence become modular with great ROI of automation.
  • Less execution time.

DisAdvantages
  • These type of test pattern mostly fail to give correct failure count.Or it is a constant pain for the test engineer to keep track. How? Let me explain... In the above test flow say Testcase-1[Create Organization] passed but Test Flow-2 [Employee Creation] failed. As Test case-1 and Test case-2 are precondition for Testcase-3 ,Hence Testcase-3 also will fail. or may not get executed depending upon your test configuration settings. So Report may look like Test-3 Passed-1 Failed-2. But no way, we can run these 2 failures. We need to execute 3 testcases once again. In an organization, in test inventory,thousands of scripts run.And if we consider ten parentage failures.We will have issues in reporting how much did we run really to get all test cases passed. Engineers need to be extremely careful to report
  • There is no way you can pick the failed test cases and execute them. As a result execution time becomes high. It becomes unreliable too
  • There is a huge chance that the polluted or corrupted test data may enter into next flow and may fail the flow.
Are test cases written in Hermetic test pattern correct?
Hermetic test pattern put more focus on test independence rather re usability. As per this test pattern each test case should be independent of each other. No sharing of test assets between test cases batched even in single test suite. Let us see how it will look like when we design our test cases as per Hermetic test pattern.
Test Name-CreateOrganization
 
public class CreateOrganization{
@precondition
//precondition steps go here
@Test 
public void Create_Organization()
{
//Creation of org code goes here
}
@Cleanup
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
}


Test Name-CreateEmployee
 
public class CreateEmployee{
@precondition
public void Create_Organization()
{
//Organization Creation code goes here
//Create_Organization method of CreateOrganization testcase and 
//Create_Organization method of CreateEmployee testcase are different and has no link in between
}
@Test 
public void Create_Organization()
{
//Creation of org code goes here
}
@Cleanup
public void Delete_Organization()
{
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
}
}

Test Name-UpdateEmployee
 
public class CreateEmployee{
@precondition
public void Create_Organization()
{
//Organization Creation code goes here
//Create_Organization method of CreateOrganization testcase and 
//Create_Organization method of CreateEmployee testcase are different and has no link/relation in between
}
@precondition
public void Create_Employee()
{
//Employee Creation code goes here

}
@Test 
public void Update_Organization()
{
//Creation of org code goes here
}
@Cleanup
//Cleanup code goes here ...here we delete all the test data created in the test. 
//Application goes back to original condition.
public void Delete_Organization()
{
}
public void Delete_Employee()
{
}
}

Advantages

  • Fully Structured and independent to each other so no problem to pickup any testcase and execute.
  • No polluted or corrupted test data propagation between tests
  • Uniquely identify the issues with testdata
  • Easy reporting and rerun report generation.

Disadvantages
  • Lot of refactoring required if we have a large test inventory prepared in other model
  • Need to prepare test data for atomic test cases which is a hectic job.
  • Execution time increases by multiple times.
So we need to take a conscious call if we will follow the hermetic design or batch mode test design . Mainly if we test a flow based scenario it is always better to go for batch mode test design.Other wise we must follow atomic test pattern or hermetic test pattern.
With hermetic test pattern we will be able to randomize our test execution.I always try to make testcases independent. If not possible or if that does not make sense, I go for batch mode or bunch mode and tag them differently.Mostly Smoke testcases[a set of testcases running day and night to make sure build is stable and correct],I prefer them independent and execute them in random order.

image Credit:pro-it-service.com
Powered by Blogger.