Lets us talk about few myth first about automation testing.
- Test Automation is very easy and anybody can implement test automation strategy. Anybody can develop a script.Well,this myth is developed by the sales person of the company to make sure their tool gets sold out.While demoing their tool,they always follow the record playback strategy. just record a simple flow , enhance by using test data concept,run the script shows the result.With these sales technique we started feeling that any body can script.But reality is completely different. Automation is not all about record and playback.It needs to be designed,then developed and tested. Definitely test engineer needs to be from some programming language background or they need to learn languages,at least the basics.Automation development is not as complex as application development but these source codes must be saved as application source code.
- Tool design is better than evaluate tools.Mainly this concept came from the expensive tools available in market. Test engineers or organization try to avoid the license cost of such tool. They prefer to build their own tool. Or try to use free scripting or languages like VBA,macro ,perl,Ruby to design test.These thing can be considered as lean approach .But not Automation.See the below links for lean approach.
But the fact is when you use a proper tool to automate the productivity will be higher which is significantly less than the licence cost. Using other technique can be considered as research and development but not proper automation.
With a proper test tool with licence will give automation test engineer more command over the QA arena. Let us see how?
- Most advanced tools will have a proper encapsulation of programming. So less advanced coding required here.
- More the mature tool more great customer care.But opensource tool will have less or no customer support.
- Advanced tools will get constant update about technologies.
- Huge support base including different forums.
- Readily available framework and integration with other technologies.
- Advance tool will have more functionalities in different technologies covered.
Automation Framework:In general Framework is an integrated part of automation testing.This is an structured combination of ..
a.Testing best approaches
before going for a automation testing...nono....forget about automation testing prior to any testing, client and service provider need to sit together and determine the approaches...
let us understand the tool based testing...
Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may require additional effort to achieve real and lasting benefits. There are potential benefits and opportunities with the use of tools in testing, but there are also risks.
Potential benefits of using tools include:
- Repetitive work is reduced (e.g. running regression tests, re-entering the same test data, and checking against coding standards).
- Greater consistency and repeatability (e.g. tests executed by a tool, and tests derived from requirements).
- Objective assessment (e.g. static measures, coverage).
- Ease of access to information about tests or testing (e.g. statistics and graphs about test progress, incident rates and performance).Risks of using tools include:
- Unrealistic expectations for the tool (including functionality and ease of use).
- Underestimating the time, cost and effort for the initial introduction of a tool (including training and external expertise).
- Underestimating the time and effort needed to achieve significant and continuing benefits from the tool (including the need for changes in the testing process and continuous improvement of the way the tool is used).
- Underestimating the effort required to maintain the test assets generated by the tool.
- Over-reliance on the tool (replacement for test design or where manual testing would be better).
This shows in 2004 the automation coverage was 5% and in 2006 it was 20% but today it is almost 50%
Software product based companies looses $21.2 billion /year due to insufficient testing.
$931 million was invested during 1999 and in 2004 it went up to $2.6 billion.
The solution may be....
1.The scope and need of testing.They must not goahead and do a testing activity for the buzzword testing--QTP etc.We need to think what will be the scope of testing.Will it be Product Oriented? Will it be Company Oriented? or Project Oriented?
Second scoping should be what kind of testing we are going to perform? Functional Testing or Load Testing or Web Service Testing or API Testing?
Tool Evaluation2.The requirement gathering and identification of proper automation tool is one of the basic criteria before going to testing viz automation testing.We need to understand the nature of requirement , priority of such requirements and priority of them There are several cases where requirement gets changed in every alternate day(Agile Testing).
Now second step of this section is to evaluate the testing tool.The Major search about testing tool in a search engine as as follows:
for tool selection there are several charts are available..
Over 300 tools are readily available in market.
Java test tool-48
other Web Testing-76
So it would be a very tough choice of tools.We need to prepare a checklist for evaluation of such tool.Identify required resources in company.Now do prototype for few sample flows.Check for different actions,Validation and reporting feature.
Tool Evaluation Criteria3.Again if it passes the above stages then business people and technical persons are required to sit together to identify the scenarios to automate and most critical prior to automation testing.100% requirements can not be covered and is not required for any automation.
Here is a article that gives more insight of tool evaluation-http://www.mydigitalvoice.in/2016/10/how-to-pilot-test-automation-project.html
Some basic Tool evaluation criteria are:
- Easy coding and develop scripts
- Easy maintenance
- Easy and huge support base
- Less cost
- Available training doc or training materials
- Technologies support
- Execution in distributed mode.
ROI-The real power of AutomationClassic Way of ROI calculation:
Manual Cost=Development cost+Execution Cost+Maintenance Cost
Automation Cost=Price of Hardware+Price of Software+development cost+Execution Cost+Maintenance cost
ROI=(Manual Cost-Automation Cost)/Automation Cost
Now this calculation is one time calculation. If organization test multiple times ROI will be higher. So if you test your application multiple times for regression or for patch testing,It is wiser to move into automation.
More About Automation Framework
4.Then comes the framework.So it is based on the application and approach.
To my view point application comes first then the approach.
The main advantages are:
- Consistency -process repeat ability and resource independent
- Structured approach
- Maintenance becomes very low.
- Maximum Re usability
- Proof of adequate testing
- Unattended execution or nightly execution
- Maximum coverage of requirements.
- Common standard across projects
- Non Technical person can start design code
- The modification during scripting should be very easy.
- Data Orientation for execution over different data set
- Huge reduction in testing cycle time
- Reduce overall QA cost
- Eliminate tedious manual effort and manual error
Automation testing is lengthy process as well as very sophisticated approach.its life cycle can be written as requirement analysis --->test planning-->test design-->test development--->execution of test cases-->bug logging--->reporting ---> maintenance.
More over automation is used to downsize manual effort not to replace the same.
Types of Framework:
There is no hard and fast rule to design a framework.
a.Record and play back
a.Data Driven Automation Framework
b.Keyword Driven Automation Framework
c.Modular Automation Framework
d.Hybrid Automation Framework
e. BPT Framework.
For more details click here
Test execution toolsTest execution tools replay scripts designed to implement tests that are stored electronically. This type of tool often requires significant effort in order to achieve significant benefits.
Record and play back
This is simple record and then playback.the only enhancement is to parameretize the data that changes.Less reusuability. It is only driver and deliver concept.Capturing tests by recording the actions of a manual tester seems attractive, but this approach does not scale to large numbers of automated tests. A captured script is a linear representation with specific data and actions as part of each script. This type of script may be unstable when unexpected events occur.Most of the time scripts are not modified to check the change in software but a new script is build to test.It is often referred as Linear framework.
When you can go for record playback?
Mostly when your application is like mainframe where we can only tab around the application and send test data.So if any application only accept keystroke , then record and play back could be an option.
Script1+Test Data 1 +repository1.mtr--->i.e/any browser-->UAT
Script2+Test Data2 + repository2.mtr--->i.e/any browser-->UAT
Script3+Test Data3 + repository3.mtr--->i.e/any browser-->UAT
It is very easy way of developing and delivering script.find disadvantages here
Data Driven Automation Framework
Repeated use of Test Scripts with different inputs.A data-driven approach separates out the test inputs (the data), usually into a spreadsheet, and uses a more generic script that can read the test data and perform the same test with different data.Generally where the flow remains same and objects does not change, Data driven framework can be very usefull.Testers who are not familiar with the scripting language can enter test data for these predefined scripts.Input data is given a pool of data set.Even the output data comes out from the data pool.It helps us to reduce coading for a large dataset.Ease of Testing of Time consumption.It is best used in IBM-Rational Robo.Data is not hard coded here,instead data are stored and provided from data source like -XML,Excel,flat file,Database or combination of all.As data layer is separated from code , same scripts can be reused multiple time.
find disadvantages here
Keyword Driven Automation Framework
Steps and data are feed to the automation tool.Automation tool judges the steps and input data from a data pool,It gave the output data.In a keyword-driven approach, the spreadsheet contains keywords describing the actions to be taken (also called action words), and test data. Testers (even if they are not familiar with the
scripting language) can then define tests using the keywords, which can be tailored to the application being tested.It is very sensative towords data design.This is based on generic key word based.
The advantage is the setting of keywords can test window/web/oracle etc based application.
More details can be found here
Modular Automation Framework
In this approach,The whole scenario is divided into small business flows-called module.Those modules are written in a library file as a functions.
In the script level the high level business functions are called.They are linked with small module functions.more over this is a very simple approach over record and play back.Objective to make it modular is nothing but to make the script reusable.There are driver in the main script and modular functions are stubs.We create shared object repository for Modular Automation Testing.When Application is huge, Initial development may go high but later the development cost just go down. Maintenance is easy and straight forward.
For more details click here and here.
Problems with this approach can be found here.
Hybrid Automation Framework
This is a combination of all frameworks .They are used whenever necessary.So is a summation of Data driven--keyword driven--Module approach.The main advantage is this is logic driven.Keywords are logic based.
For more Click here
We generally carried by the buzz words in the name of the framework. So migrating to a good framework may not be necessary for your organisation if the current one is doing just great. So better not to reinvent the wheel or don't change it until it is broken.
few Guidelines while designing the framework:
1.DO version control(use CVS,SVN).
2. Try make the script Environment neutral.
Read More here for speed up automation with file size. Even Object Repository can be a nice part of Automation.How? Click here to learn more
Hoe to improve the framework? See here and here