We can automate every possible testcases, including low probability,low impact edge cases.More the testcases we need to run, more it triggers maintenance. Execution time will be higher . So it is better to cover high impacted testcases. The only thing we need to keep in mind that we should not create spaghetti model of testcases. Tests independent in nature gives more value in the long term.Also we can create a model-Do not Repeat yourself.A lean regression suite helps keeping maintenance low but gives lots of value.After doing so many things, our test code base goes high with the releases.The volume becomes heavy. As a result the execution time will become high too.
In a normal day Automation test engineer can execute say 5-6 test cases without manual intervention. We need to find out some alternative where we can run the testcases very fast. If we increase the hardware or computer for that person,he can execute all of them in 1 hour.So in a day 50-60 testcases can be executed without any issue. Now, the idea can be further modified. The idea to execute on multiple machine can be implemented via tool that can remotely control the process of receiving and managing scheduling the execution provided by user.
The remote control program is nothing but an agent and mater(sometimes said master slave) concept. Where the master runs in a separate machine and clients keep listening to the command from master. The master is responsible to schedule and dispatch the run command to the client,tracks the progress and update about the the job.
Automation testing suite is nothing but a collection of risk based suite covering the critical functionality that must work in order to effectively deliver services to customers. It can be based on the functionality change as well.Running them in different system without monitoring them brings the idea of continuous Integration . There are CI tools available in the market. they are -Jenkins,Hudson, QC,Cruise control etc. We can also make out own master slave to support Parallel testing.CI can execute the tests either on each check in by developers or testers or by default it can have a scheduler. The trigger process can be pure manual too.
As the design change takes place during software development , the ability of regression testing emerged.Automation becomes strength for test engineers to determine the changes. Regression runs with previously automated testcases,find the bugs and changes.
Total execution time: a+b+c+d
min(a,b,c,d) or maximum(a,b,c,d)
A parallel test takes more than 50% lesser time from a non parallel (sequential one).Rather than waiting for each test to finish execution we will be able to execute multiple tests in multiple systems in a shorter time of span.Multiple systems can be of different OS combinations,across various browsers. It is somewhat similar to distributed automated testing .A little modification is design will make it parallel running.
Parallel running done when we do -
- version update of a software-We run same testcases against two systems-old and new and verify output.
- Migrating from one system to another
- Database migration
In parallel running, we ensure that the new system is capable enough to handle all features available in older system and ready to replace the old one.
Consider the following equation:
if A=C,B=C then A*B=C*C=C to the power of 2
A=Test early and Test often
B=Maximize the benefits from automation
So introduction to automation in a very early stage of the software development and run them as much as possible will maximize the benefits from automation. With less manual effort to test regression,quick signoff sends the software to the production line.
Second important aspect of parallel testing is to test in cloud. Cloud is the big step forward.Many companies including startups are moving towards cloud.Automation team also needs to provide values while testing in cloud. In cloud while testing parallel testing comes handy.In cloud way of testing , Automation script will have a single copy but support n numbers of clients tried to access to it.It will reduce the throughput,execution overhead and cut down the overall testing cost.
- We need to re-engineer the test cases to support multiple client
- We need multiple VMs to support the parallel execution-take old hardware that are laying in the IT room that will mostly help in execution.
- It must have a BCP-Business Continuity Plan implemented.
- The code asset must be same for all client.
- A good CI-Jenkins,Cruse control,cruise control,or your own CI/schedular
- Need fledged environment that is running perfectly
- All pre condition scenarios should be available to test.
- While testing cross browsers or on multiple machines, test might fail due to platform issue. Automation engineers should not neglect those rather investigate the cause of failures.
- All test related settings should be coded in script and governed by script itself. In parallel testing script may run in a out of the box computer. Otherwise it will fail.All tab related settings,automatic update process,antivirus alert should be disabled before attaching to the master.
- TDD is hard to learn but all stakeholders will learn the process in a few weeks. A concise effort, by taking few extra step will surely make it possible for a long term success.
- Automation scripts also need constant refactoring to maintain its hygiene. This continuous process bring automation up to date and bring maintenance cost significantly down in the long run.
- While designing the regression testing , take incremental approach. Initial days ,automation looks tiny but as the days go,it will become huge.
- As parallel tests will run on a different systems, regular review of the automation tests to ensure they are still fit and serves the purpose.
- Metric is another important measure.The initial measure should be incremental growth of line of code. Then to check the automation effectiveness.We need to show the management so that the funding keep flowing.
- Making result visible is another form of publicity.That may create an awareness in all levels including developers.
- Faster feedback is the main outcome of this approach.
What kind of test to pick for Parallel testing?
Well, this space is purely governed by functional experts. Business analysts or business users can determine the critical business areas that needs to be tested. The only condition here, no two tests will have dependency in running or they need to be a single suite.
As in CI for parallel testing a very fast feedback is necessary to test the application,Smoke tests will be the first to pick up in this category.
In my view , the system should have a GUI where manual,functional or any stakeholder can pick and execute the testcases.
Selection of testing tool will also matter. A fitness tests provides faster feedback on GUI testbut it can be very slow when compared to JUnit tests. Canoo webTest,Selenium with Geb and spock or any open source tool are ideal for my way of testing. UFT is not free needs licenses to run in parallel.XML is the easy and effective way to provide data.Excel can also be used.
Parallel testing is most efficient when we do-
- Version assessment
- Localization testing
- Internationalization testing
- Linguistics view and content validation
- Cross browser testing
- Cross platform testing
- Simply running huge test asset into different systems.
Not all the testcases can be made parallel. The optimum balance should be drawn between coverage and speed.