This post talks about different direct and indirect techniques to get better ROI.Calculating ROI for automation is a very subjective discussion. I tried to come up with several direct and indirect factors that affects ROI. But remember ROI calculation is notoriously hard to quantify.
Software testing via automation is a process or a series of processes ,designed to make sure computer code does what it was designed to do and that it does not do anything unintended. Software code should be predictable and consistent, offering no surprise to the end user.
While testing a set of test cases that satisfies the programming ,does not guarantee that all possible errors would be found. In an ideal world, testers want to test everything ,every permutation of a program. Practically , this is impossible to cover all testcases. It is impossible too.
Another misconception around automation is going towards automation will give direct benefit and ROI quickly. But this is a false statement. Automation may not give immediate value to money. But it may help you to achieve lot of tangible and intangible benefits. It not only offer support to scrum team but also provides a helping hands to the build team,developer team and finally regression testers.That is why it is very tough to get its ROI so easily.
A simple calculation is ROI=net Income/Investment.
ROI=(gain from investment-cost of investment)/cost of investment
ROI=(revenue-cost of goods sold)/cost of goods sold
As per best practice , there are two different ways we can calculate ROI..
- Efficient ROI calculation
- Risk estimation and reduction ROI calculation
As automation needs huge investment by purchasing modern commercial tools,hiring technical automators, version control tools for scripts,Test management tool. So as a manager or part of management, one should always calculate the ROI.
In a simple term ROI in automation=(cost of automation-cost of manual)/total cost
In most of the organisations automation cost is calculated as the cost to procure the tool.Many a time, I keep hearing that cost of selenium is zero , hence the automation cost should be very minimal.
Automation executes the test faster than manual QA. This statement is the tip of the iceberg.Actual story is far bigger than that.Implementation time of a script,execution monitoring ,automation failure analysis,re-execution of the failed cases are few hidden factors for automation.These are additional costs to automation but seldom factored when calculating the ROI. Benefits from automation and ROI are two different things.They should not be used as an interchangeable terms.Benefits are usefulness,but ROI talks about price.
What is cost of automation:
Factors affecting overall testing:
- Volume of testcases that can be automated directly without any technical or functional issues-Test coverage
- Number of releases in a year and number of cycle per year
- Number of platform supported(windows/linux/mac) vs number of browsers(IE,Chrome,Firefox,opera etc) supported.
- Cost of manual testing
Factors affecting automation testing:
- Cost of the tool(Selenium,protactor ect are opensource whereas UFT,Test complete,TOSCA are costly)
- Setup cost or building initial cost(framework development,low level or high level component or business and generic function building)
- Cost of automation testing itself.
Few points to keep your ROI high:
- Don't invest too much effort in only designing automation without testing it in a real scenario.
- A well planned project is likely to go near success.So for a project to success , prepare a solid plan.A plan is nothing but a guide.
- All plans should be agile in nature
- Team building is an important aspect of automation,especially when they are geographically divided.Rather experts, if a lot of interested people in automation will give you better result.also beware of the knowledge walking out of the door like contractors. it is better to involve the regular permanent employees rather vendors.
- There is no such tool which is perfect for testing the under laying application. But all the tools that are available in market needs someway or other improvement or tailoring to suite our need.So choose a which is cost effective and suites most of our need during automation. We should not go for very costly sophisticated tool or should not go for free tools. Actually the main agenda is if you don't spend anything, you will get nothing.
- skilled lobar cost or making them skilled cost--if you do not have skilled lobar it is very tough to get a successful automation.It is always better choose a tool, where you have greater knowledge base.A well managed,skilled knowledge team will surely give success and contribute ROI.
- A good training saves lots of research time.Which is termed as non productive hours in terms of management. Reinventing the wheel will always lowers the ROI.
- Architecture cost-If you don't have one,automation is bound to fail in the long run.A good architecture (technical)is important for framework development. This gives flexibility and adaptability in longer run. Minimizing the cost can only be achieved if automation reaches that height.
- Constructive use of time is always important between testing cycle.Always dream big and go for it as long as time permits. Slowly,we can get more more out of automation. Self motivation is a quality that we need to work on this.
- Creative way to resolve issue reduces the overhead which interns a cost to automation process. We must nurse the issue resolving culture so that the overhead cost reduces that in terns improve ROI.
- Organisational support is needed to get significant automation initiative started. Be sure we are choosing a champion who will stay on task and be the driving force behind automation initiatives.
let us come to the calculation part:
A manual testing can be dome maximum of 8 hours/day where as automation can run 24 hours.(This is not dream,some of the organization has already reached to that height) but majority falls between 12 hours to 20 hours of execution.
Now cost for manual testing=(total number of testcases execution effort/total testers effort)* rates of manual tester
total number of testcase execution effort=Data preparation+Execution+Analysis (in hours)
total tester effort=Total test engineers*8 (in hrs)
Now automation investment