Header Ads

How To Pilot A Test Automation Project?

Image Credit:Youtube.com
A pilot project is a case study on a given problem statement based on different tools, situations. Piloting a test automation project is to try out tools and find out the best tool or tools combination for automating different scenarios. The output come of a pilot project is a successful /unsuccessful conclusion. There is no such failure is a Pilot. It is all about learning the incompatibility to provide solution for a given problem. in that case we need to look for another tool for this automation.Sometimes despite of putting all effort,thing may go in wrong way due some factors on which we do not have control. Use this experience for future design.Based on the outcome of a pilot project, we get direction to use the tool for production.The real project starts.All findings,learning,experiences need to be put together to drive the real project.It is an iterative process to understand if the tool and the application is working fine together.
Guidelines for test automation :
  1. Identify a set of like minded employee to take up the Pilot project. Most efforts to do test automation is taken by testers and they take it as hobby. Their responsibility is something else. These efforts can not sustain in long run as lack of guidance.
  2. Automation needs to clear about the scope of testing. They might have a stable GUI application in place to kick start automation pilot. Based on the application certain tools are also needed to be identified. to carry test automation pilot.The scope can be enterprise oriented,product oriented or project oriented.
  3. All members in the pilot team need to take some responsibilities to carry out.
  4. Automation projects are not problem free,there will be lots of obstacles,So we need to have a plan in place. The plan must be agile in nature. Depending upon the condition it should get changed.Changes occurring dynamically will lead to a success.
  5. We need to have a constant focus and realistic goal to get good result from pilot.
  6. It is always better to build a bare basics and build more on top of that. Pilot projects help in this aspect.
  7. Scope determination is another important aspect of Pilot project. If you are having a very shorter duration release and the project is for very shorter time,automation is not for you. We must not go by the catch word “Automation”.Automation project should only carried out for such cases where it is worth automating.Invalid,outdated cases or low ROI or negative cases  should be out of scope.
  8. Test automation is not only for quicker way of execution. This could be a feature. Even small piece of code ,executing on a smaller area may help manual tester. Like data preparation by macro, Small AutoIT code for window automation or taking server backup,VBS code to take screenshots,starting ,stopping services could be in scope for automation. pay attention to the need of manual testers.
  9. Test code is similar like software code. So it might have some bugs as well. Implement a process for peer review or code review in the first step itself. Demoing to the stakeholders frequently can reduce the load of redevelopment later.
Now with these guideline the next question comes which should be considered to Pilot project?

In pilot project we mostly need to automate the smoke testcases. Make it as easy as possible, around the GUI aspect- dropdowns,popups,menus,lists,etc(controls), then Edges(Certain,definite,number of edges).Every edge connects two consecutive components.Every two nodes or controls of G joined by an unique path.
So control( C)=number of edges+1.
Total testcases could be covered=Controls-1.
In GUI we do not have statement coverage rather we have control coverage. Picking up easy staff to automate that provides value and quick feedback should be the first one to pick up. This will give all the stakeholders a comfort layer to think more.
Then try to make wrappers around those. Once they are done, pick up few functional testcases. Somebody may ask “why we need to work with controls?” The reason is simple. start with small step forward by targeting the only single important flow. Lean and mean approach to follow in pilot.But at the same time do not loose the bigger aspect of automation.

There are two aspects to it:
  • The manual tests tends to focus on the limited area and often to words business needs. They mostly miss to cover huge numbers of GUI components presents in an application.Automating these area will reduce the risk of undetected bugs around controls.Manual way of regression testing is a big failure due to various factors.More over, it is loss of time and man power.Introduction automation (if put correctly) will surely reduce the manual effort by 60%. That means manual testers or functional testers get more time to think and design new testcases.In today's rapid speed development,agile is the way to go and TDD or BDD can be an efficient manner to support automation Testing.
    Image credit: saucelabs.com
  • These become the basic building blocks for framework. They may be the low level functions or components that will be stitched together to make a business flow.

The tests which are repeatable in nature ,lengthy and boring should be the first choice. This cases are the pain key areas of manual process. Once automated, they will bring more value add. Next comes the complex cases.Introducing automation for a quick win may destroy automation for a long term. So be very careful on this.The only success criteria for a successful automation in Agile is “One Goal for All”. Everybody in the team(Dev,testers,business analysts,owners etc have some role to play.). Agile way of developing software is not not the only criteria.Clear communication,impact area analysis ,Relationship between team members,building automation script criteria are few notable discussion area.

In pilot project the tool sales person provide all necessary support. But while automating a piece , It is better  to take help from somebody who knows the tool or the language.Sales pitches are often misleading.
There is a big myth around automation that test automation is real easy and simple. This is due to super cool sales pitch from the tool's sales team.But all we need to understand all testers can not develop automation and it is as good as development task.
The second big myth or misleading information is "commercial tools are way expensive". To minimize the cost often organizations go for free tools and develop their own. Use scripting language like perl and ruby.Use shareware tools or do not go for automation at all. But in reality the cost of the tool is 3 to 10% of the total testing cost.

So the pilot project will be more favorable to reputable and established creators/vendors as they are less vulnerable to fad driven ,24*7 support availability , But open source products should also be considered.

Development team can play an instrumental role during pilot project. Also major success for automation depends on the relationship between developers and testers. Pilot Projects are right place to make this nexus. By educating development team about the automation requirements,this will reduce the rework viz regression development significantly.The cost of regression development is too high This can lead to low ROI.
Testcases should be outside of the test tool.Use of Quality centre,OATS or some repository for this purpose. In this case even your pilot fails the testcases can re reused for other pilot projects. Only the controls and wrappers need to change.With every successful testcases,mock the testcases via automation as far as they are realistic.Remember tests may be run over several other platforms.
While doing a pilot project , we are suppose to provide the best tool for our customer or for our organization. Test tool selection should be based on:
  • Robustness
  • Track record
  • Cost of the tool
  • Cost of implementation of scripts
  • Rolling out automation on the floor cost
  • Performance
  • Timely manner of execution
  • Less learning curve
  • Available technical support
  • features available etc.
So create a score card and provide number may be 0-10 scale and then compare which suites to your need.Benefits of using commercial tools:

  1. Customer support base is huge. open source or in-house tool do not have support base or having less support base.So research,development and up gradation time is saved.
  2. Commercial tools are being constantly updated to cope up with current technology trend.So it can support multiple technologies where as the opensource tools will have less or very specific support to a particular technology.
  3. Commercial tools will have huge support like forums ,blogs and how to guide where as opensource tools will have less forum support.
  4. Plug and play easy frameworks are readily available for commercial tools
  5. Commercial tools can be easily integrated with Test management tools which makes reporting and execution easy.

It is essential to keep the testers out of pilot as much as possible. Create distinct roles for automator and testers.
Documentation is another important aspect to look for. Tools documentation helps the test engineers to understand better.It reduces the training timeline by expediting the knowledgebase. FAQ page could be another great idea to understand the common pitfalls and how to get rid of those.
validate few data conditioning test while doing Pilot project.Date testing and screenshot capture testing should also be there in the pilot project checklist.
What is very important while doing pilot is to maintain stakeholders updated , to meet their interest and support with report.The pilot report should be publish as early as possible. Frequency matters!!!. So be prepare for sending report.
UI level tests are very unstable ,fragile,expensive to maintain. Hence automation must look out for an alternative to carry out in some other way. Like-API testing. While piloting a project,automators must look at the feasibility to introduce API testing.

From an automation stand's point, unit testcases are very important assets. They are very easy to develop and can execute in few seconds. While piloting, we must target such testcases , which will give not only speed but also confidence. But smaller GUI automation success , expectation and enthusiasm become high(uninformed optimism) with this mostly the initial scope is forgotten. The new scope become larger,technologically more complex with the initial timeline so pilot project engineers need to be very careful about the expectation.
While piloting a project there might be one more scenario that is migration (from one old tool to new one). It is always better to pick up fresh or new flows for piloting rather than picking up faulty flow. I encountered such a situation. What I did , I did not pick up the flows which were faulty or were running fine with old tool. I supported all new features and testcases with new tool and slowly migrated the old flows to new one. It is always said that don't change it until it is broken. I obeyed the same. My old testcases were running in old system where as the new testcases were started running in new tool.

For each approach we take to introduce automation needs an explanation to management. We need to be clear why I am choosing a tool, language,cost and most importantly the ROI. Always remember management support for automation is one of the prime factor for successful automation.A realistic objective , provided sufficient time and a appropriate team will surely achieve the desired ROI. Introducing automation to an organisation is great but always keep all the stakeholder updated about the fact that it is an automated way of testing not automatic testing. Scripts can only do any testing what is already coded but can not cover new functionality development.  
Powered by Blogger.