Results for Framework

How To Create Test Data for Advanced Test Structure or Test Framework

August 07, 2015
In my last couple of posts, Structure of Automation Test Cases and
Best Test Structure::Hermetic Test Pattern , I have shown each testcase
should be independent to each other. But to make them real
independent it requires testdata manipulation. Testdata also need to
be independent ad unique in nature.

Why it is required?
It is required to stop data pollution.Stop using polluted data usage
inflow and outflow. 
I am writing few common methods to make testdata independent:

  1. Use testcase id or name at the end of the testdata like Cutomer_Name=”Rabin”+”Test123”;
  2. Use time stamp to make testdata unique like-Customer_Name=”Rabin”+”3103201515082015”;
  3. Use random number generator to create unique“Robin”+rand.nextInt(900) + 100;
I like the Random number generation method and if it is not possible to use, I generally use the testcase id.
More note on Random number generation:
RandomStringUtils.random(10, false,true);

you need to import - org.apache.commons.lang3.RandomStringUtils package
this is very useful when generating 10 digit random number.
AtomicInteger counter = new AtomicInteger(1);

This is also very useful to get unique number.

image credit:
How To Create Test Data for Advanced Test Structure or Test Framework How To Create Test Data for Advanced Test Structure or Test Framework Reviewed by Animesh Chatterjee on August 07, 2015 Rating: 5

Best Test Structure::Hermetic Test Pattern

August 07, 2015
In my last post-Structure of Automation Test Cases, I have shown how we can improve the test structure and make a better one.Too many tests will make feedback loop delayed and too less tests may not have enough coverage.Sometimes too many tests are added to a single E2E(End to End) script.This kind of test design will make it spaghetti. The goal is to design a significant approach that will lower down the feedback loop.
Now lets looks at the question below:

  • What is the biggest benefit or goal ? 
  • What is the cost ? 
  • How is the team? 
  • What training team needs? 
  • What is holding automation back? 
We have to prioritise the issues and resolve one by one. Now in this post I am trying to venture the pattern on which the structure is made. The test structure is made on Hermetic Test Pattern. As per the Hermetic test pattern each test should be completely Independent and should not depend on other test. So,It should be atomic in nature and self sufficient.As per the maintenance concern,each test case should be maintained concern,each testcase should be maintained separately and if any dependency outside of the test found(may be a 3rd party library,3rd party service) test engineer should cut the linkage with the service.
The best way to start is to look into smaller objectives but add test slowly to make it mature.
Advantages: Each testcases are atomic in nature so no dependency on anything. Test takes clean start as precondition creates all test data No polluted data inflow our outflow. We can execute the test in different order on need based. We can create more molecularity which increase code re usability. As each testcase is atomic,parallel execution is still possible.The deep advantages are:

  1. Increase confidence in software quality.
  2. Earlier time to produce the software to the market means more revenue.
  3. Reduce the cost in testing.
  4. Consistent repeatable testing
  5. Run tests unattended.
  6. Find more regression bugs
  7. Run tests more often
  8. Better quality software
  9. Coverage is high
  10. Find more bugs
  11. Test on different systems.

Disadvantages: Duplication of code increases if not created modular type. Design testdata for each testcase Execution time increases. High resource utilisation.
Best Test Structure::Hermetic Test Pattern Best Test Structure::Hermetic Test Pattern Reviewed by Animesh Chatterjee on August 07, 2015 Rating: 5

How to give test URL to my framework

January 26, 2012

This section is dedicated to my viewers.Here they can ask question and I will be happy to help or redirect to some page what is best suitable for them. 

Hi Animesh
How to change/give environment name dynamically -to my framework? .Right now we are hardcoding in our scripts.

Hi Veenetha,
Thank you very much for writting me!! I am trying my level best to solve your query.

Well veenetha,there are several ways how you can insert environment name into your framework:----
1)Ofcourse,I agree,the hardcoding into a script is one of the options.
 String  URL=""
2)The second improvement could be,you can create a dialog box(input to take the environment.This you will place master/global driver and the value you will pass to the child scripts.

Or,you get the URL from user and save it to the notepad and read it dynamically afterwords.
3)Create an environment file for providing URL.I found advantage on creating a separate file for environment details.Updation is very easy and effective.
Say active_URL=""
Tomorrow you can change it to

My personal suggestion is to go with either 2 or 3.
How to give test URL to my framework How to give test URL to my framework Reviewed by Animesh Chatterjee on January 26, 2012 Rating: 5

Features of today's Automation Testing Framework Part-3

January 08, 2012
Hope you have enjoyed Part one and part two of Framework Building. Now I can say It is bit updated framework. This post will talk more about the fine tuning your framework development.
Portability:When I said Distributed nature in part two What I mean that any test can be operated from any machine.Let us improve the framework little bit more. I can copy the master driver which will automatically create the folder structure add up the necessary dependent files and will be ready to run any testcases.
Ease of Functionality:  This can be achieved when we can give user a platform where user can plug in and plug out any extra feature he/she required/not required respectively.Like Manager , business analysts and functional people can control global testing without knowing the tool.This will truly become Skill Utilization too.
Capture all : Yes!!! that is the key for a successful  framework.Always capture Step information,log,Success criteria,Exit and entry criteria and Error Information.
Scoping and Descoping:   The more you make your framework user friendly , by providing Easy controlled and controllable actions ,More it will be a go to tool for all.This feature will allow business user to scope a set of testcases or scenario for a specific business flow.
Advanced Coverage: The more you fine tune your application the more the test engineers are going to use it.If your Framework supports Negative testing ,their is nothing like it.Manual persons also can jump into your framework to speed up their application testing.It is nothing but to increase your coverage i.e Automation Coverage This simply saving manual effort.
NO Object Repository:  Now a days testing demands very fast execution and report generation.Please refer one of my post on Having Repository.So now a days people like to have a pattern based testing where there will be no Object Repository.Pattern will detect the type of field and will do actions.
Hassle Free Integration: Framework should support a Hassle free way to integrate a newly build module. I suggest a GUI component to do this integration. By this an Automation test engineer or a functional test engineer can make a code and can submit for further use.Framework should not be so complicated that integration of modules will take lot of time and it requires a lot of effective testing time.
Implementation of Source control System : This is one of the basic criteria to make an advanced framework.Yes!!! Track your source code to make everybody understand what kind of change is applied upon the code.History will make sure if the piece of code is untouched or error prone and who has done what kind of change.There is always a chance that the new code is error prone and can create regression. For that case we can always go back to previous version.   
Recordings: New framework should support video recording of each and every testcase execution report. Based on the requirements, framework should support the pass video or fail video along with steps and logs.
Features of today's Automation Testing Framework Part-3 Features of today's Automation Testing Framework Part-3 Reviewed by Animesh Chatterjee on January 08, 2012 Rating: 5

How to design folder structure for Modular driven framework

September 28, 2011
How a modular driven framework should look like?
I am going to describe a general idea about the folder structure of module driven framework.


Folder Name Purpose
Root name should be the project Name or the client name what every is good to identify.
Common Here we will be placing all the common functions,common objects,Common Recoveries
------------Function Library This is the place where we will be placing all the common functions.Its extension will be .VBS
------------Initialization file This file will be placed under common folder and is required to save values during run time or to start a script. the extension can be .txt,.csv,.vbs
------------Recovery Scenario This is file where all the recovery specific to all modules can be defined. Ideally all known problems should be handled by code but this is a theoretical folder to have…Extension-.qrs,.xml
Modules All the modules and specific to modules cases will be kept here. Say i have a Module called customer
now under customer there can different functionalities…
1. Customer Registration
2. Customer Verification
3. Customer Activation
4. Customer Termination etc
----------- Object Repository This is a folder where module Specific repositories will be kept. Extension-.xml,.mtr….This Objects may be removed after use
----------- Script This is a folder where module Specific scripts will be kept. .vbs will be the extension
----------- Datatable This is a folder where module Specific Data tables will be kept. The initial of the data table will be the first two or three letters of the module.
like-CR-datatable.xls(used for Customer Registration)
----------- Module Specific Recovery This is file where all the recovery specific to a module can be defined. Ideally all known problems should be handled by code but this is a theoretical folder to have…Extension-.qrs,.xml
Master Driver This is the folder where all your main script,datatable,Object repository related to Master driver will stay
------------------Master Driver Script This is the Folder where the main driver will be kept
------------------Sequence File This file talks about the running sequence of master driver.Say you have 7 scenarios and you want to execute 3 of them ….Sequence file is the key to configure that.Please don’t be confuse with keyword driven framework's sequence file
------------------Datatable-Main This is a folder where Data tables will be kept. This datatable talks about the child scripts information and calling rules(say input,output)
------------------Object Repository This folder contains the most common objects required to run the script.Please remember this will not be removed any point of time during the execution
------------------Recovery Over all recovery and finally statement is written here
Technorati Tags: ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Windows Live Tags: folder,Modular,framework,module,Name,Purpose,Root,client,Common,Here,objects,Function,Library,extension,Initialization,script,Recovery,Scenario,problems,code,cases,customer,Registration,Verification,Activation,Termination,Object,Repository,Specific,Datatable,Data,Master,Driver,Sequence,File,talks,Main,rules,output,execution,Over,statement,Recoveries,modules,scripts,scenarios
WordPress Tags: folder,Modular,framework,module,Name,Purpose,Root,client,Common,Here,objects,Function,Library,extension,Initialization,script,Recovery,Scenario,problems,code,cases,customer,Registration,Verification,Activation,Termination,Object,Repository,Specific,Datatable,Data,Master,Driver,Sequence,File,talks,Main,rules,output,execution,Over,statement,Recoveries,modules,scripts,scenarios

Blogger Labels: folder,Modular,framework,module,Name,Purpose,Root,client,Common,Here,objects,Function,Library,extension,Initialization,script,Recovery,Scenario,problems,code,cases,customer,Registration,Verification,Activation,Termination,Object,Repository,Specific,Datatable,Data,Master,Driver,Sequence,File,talks,Main,rules,output,execution,Over,statement,Recoveries,modules,scripts,scenarios
How to design folder structure for Modular driven framework How to design folder structure for Modular driven framework Reviewed by Animesh Chatterjee on September 28, 2011 Rating: 5

How Object repository can affect execution performance?

September 22, 2011
There are several approaches in the market to generate your script and  the framework implementation. But we need to think a manner where script development time ,script maintenance time becomes less.Most importantly we need to find an approach where the execution time is also become less.
What is the importance of downing the execution time?
Well, i am inspired by a philosophy that automation test engineers should not loose time by executing test script. All the test script should be configured once and should be able to run unmanned manner. More over these scripts will run in a computer assisted environment. Computer is not that intelligent as we are.If you insert enough intelligence into script it might slow down the process and on the other side intelligence is less..there is a chance that script will error out. So what is the best approaches to automate your script that will serve long without issue is a question.

Role of Object Repository
Object repository is a place where your testing tool use to store information about objects under test for a flow. This may be .tsr file or a straight xml file.During run time script takes reference of  this file and identify the objects.
It is seen if the OR size is less that means test engineers are doing more descriptive programming.Descriptive programming is very powerful but it is a time consuming process.Hence it will slow down the process.
On the other hand if the OR file is big.The seek time will be high. There might be a scenario where object conflict will occur.This also slow down the process.
Here this picture shows if OR files grows high in Size how the response time of the scripts increase.
As per experience, An O.R file should be between 5 to 12 MB..Above that or below that performance is not as good as we expected to be.

Check out here how to speed up an automation framework
Technorati Tags: ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Windows Live Tags: Object,repository,execution,performance,Introduction,script,framework,implementation,manner,development,maintenance,Most,importance,philosophy,automation,computer,environment,intelligence,error,Approach,Role,tool,objects,reference,size,Descriptive,scenario,Here,files,response,Above,Check,scripts,engineers
WordPress Tags: Object,repository,execution,performance,Introduction,script,framework,implementation,manner,development,maintenance,Most,importance,philosophy,automation,computer,environment,intelligence,error,Approach,Role,tool,objects,reference,size,Descriptive,scenario,Here,files,response,Above,Check,scripts,engineers
Blogger Labels: Object,repository,execution,performance,Introduction,script,framework,implementation,manner,development,maintenance,Most,importance,philosophy,automation,computer,environment,intelligence,error,Approach,Role,tool,objects,reference,size,Descriptive,scenario,Here,files,response,Above,Check,scripts,engineers
How Object repository can affect execution performance? How Object repository can affect execution performance? Reviewed by Animesh Chatterjee on September 22, 2011 Rating: 5

How file size can affect execution time?

September 18, 2011

There are several approaches in the market to generate your script and  the framework implementation. But we need to think a manner where script development time ,script maintenance time becomes less.Most importantly we need to find an approach where the execution time is also become less.
What is the importance of downing the execution time?
Well, i am inspired by a philosophy that automation test engineers should not loose time by executing test script. All the test script should be configured once and should be able to run unmanned manner. More over these scripts will run in a computer assisted environment. Computer is not that intelligent as we are.If you insert enough intelligence into script it might slow down the process and on the other side intelligence is less..there is a chance that script will error out. So what is the best approaches to automate your script that will serve long without issue is a question.
Please read here for the best Approach for automation.
File Structure and size:
Once you finalized your approaches the next important aspect is the component development. Yes the most important component of any framework is File.
Now what ever framework we develop ,the structure always resides inside the Hard disk.Even if we came up with  high speed data bus,high speed data transfer mechanism,during execution it is seen the number of files ,structure and file size slow down the process.
  • if the file size is very less  that means we need to incorporate a lot of files….. the seek and latency time goes high
  • if the file size is high that means we are including every thing inside one library and seek time becomes high.

The guideline for file design is written below:
  • try to write same functions into a file but do not club all the function into a single file.
  • Number of files which will be loaded first (like-common functions –common to the entire flow..needs to be called almost every steps, path library) should be less 2/3.
  • library specific to a module will be loaded dynamically and will be released after use.
  • Garbage collector should be called after dropping one file . it depends on the scenario to scenario.
  • More over any point of time the files are handled can not exceed 3/4.
  • Try using lightweight HTML reporting instead of Excel Reporting.(excel-2000 and above takes heavy space in RAM….it is always greater than 2 MB.
  • Using of CSV as a input file is the best approach. One more best approach is to use XML.
How file size can affect execution time? How file size can affect execution time? Reviewed by Animesh Chatterjee on September 18, 2011 Rating: 5

What is Hybrid Framework?

September 16, 2011
Recording script using standard recording mode for any test tool is not enough for long term success for an automation.Also scripts needs to be developed by experts. Manual team can create the recording but proper way of coding and maintaining it should be automation expert's duty.Otherwise it will create a high noice and high maintenance.There are several way to code properly.Page object model,Wrapper function implementation (Control and Window) are very popular in today's automation. This post talks about the mixture of few approaches and try to find a definition for better automation development

What is it?: As the name suggests, It is a combination or mixture of either two or more basic frameworks.Say some parts of your application is best solved by keyword driven framework and some parts can be resolved by only record and play back. Then you can apply both the framework.
say we have scenario where you have to test a signature or do a signature on a paint window or draw a picture. All leading frameworks fail at that point. No framework will support draw graphics of a signature . Building such coding will be expensive. So the solution is to add record playback framework with your existing framework to do this part.
It can also be data driven but modular approach.Any framework which is a combination of two different approach can be treated as hybrid framework.
It is the most commonly implemented framework due to its easy implementation and extensible.
Mostly combination of different framework is made to mitigate their risks and disadvantages and extend their advantages depending on the project objective.
Common Frameworks used for building Hybrid:
  1. Data driven framework
  2. Keyword driven framework
  3. Modular framework
  4. BPT framework
  5. Record playback(Linear framework)
Theoretical concept:
The objective of building such framework is to maximize the strength of each framework and minimize the weakness. We are actually  hiding the weakness of one framework with another framework’s strength. One more intention to make such a mixture of framework is to make our run without any issue and manual intervention.

How to develop Hybrid Framework?
Test engineers checks the manual test scripts. Here we can have two conditions..
  1. The application is present
  2. The application is yet to come.
Based on the availability of the application test engineers may record the script .They can spy the object to create a repository or dictionary of keywords and objects.Or else they can write descriptive programming to proceed further. The code and flow should be easily understandable. The coding should be easy maintenable.
At this point test engineers will have to select set of approaches to be followed to automate a flow.After this point, there will be a brain storming process to detect the cost benefit analysis on different approaches. For each of the process the best yield results will be captured.
The set of best approaches will be summarised and published as Hybrid Framework.
image There is no fixed guideline while developing the hybrid framework.Anything that suits our requirement can be placed in hybrid framework. But as great freedom comes with great responsibility, Hybrid developers needs to careful and needs to have programming knowledge.

Proactively find issues in production is an important aspect of test automation.Automation runs best when you know the objective of it and guiding principles.
An important step to follow for using any framework is not to reinvent the wheel.Reuse whenever possible.Do not just automate in any one approach rather use something which is useful.This will give us greater value.
Tailoring the reporting will give more information than traditional reporting. There can be multi folds of reports.
1. Incident reporting- This talks about all the failures.
2. Escalation-This is refined for potential bugs.
Reporting also should talk about the type of bug,platform information,general statistics,detection rate,redundancy for tests.There are few good to have feature like-email triggering of the result,XML testdata file for light weighted testdata input,logfile generation etc. Sharing testresult across team is very important. It is having two way effect. One way development team will be extra careful next time onword about the product code at the same time test developers will be extra careful while designing the test.

It is very important to seek feedback for further improvement areas.Sometimes even the customer gives feedback about some corner cases which in house QA never thought off.For this purpose regularly observe the automation runs.Even the UAT testers can give lot of scenarios.It is always better to have existing automation scripts. But need to upgrade if some functionality is broken or needs up gradation.Automation execution can be really noisy for synchronization issues. Application slowness,test tools fastness contribute the most. There are third party software which also contributes to this. Be careful when using combinations of software or tool.By seeing the execution trends you can determine the same.

Sometimes we use by pass logic or work around to make a testcase pass. This can give a false positive result. Actual bug can be hidden behind this. So make sure you workarounds are limited and well tested.Discussing about the automation scripts ,scenarios and what automation is actually doing may give automation team a better idea and understanding.This is also reduce the pressure on automation team. It will clear the road blocks.So communication also plays a vital role. So don't be afraid of asking or questioning the area which you don't know.

A significant benefit of hybrid framework is the ability to support multiple browsers,platforms and technology stacks. So take proper advantages of capabilities,object clesses and business classes.

To get success via hybrid framework, management support is not only necessary but also critical.but the expectation from automation should be realistic.Even though we are using a hybrid framework, the main objective should be to pick up a small testcases and automate.The small step forward will give automation and the underlying concept a huge boost.

In hybrid framework,automation team can be divided into developers,executors(including report generators and analysis),functionality automator,gatekeeper,reviewer etc.when the responsibilities are given as per knowledge level, the chance to excel with automation will go high.

What is Hybrid Framework? What is Hybrid Framework? Reviewed by Animesh Chatterjee on September 16, 2011 Rating: 5

What is Keyword Driven Framework??

September 07, 2010
With the advancement of automation framework, after record and playback ,Keyword driven framework is the most robust and acceptable framework while automating huge testcases.

It is true that sometimes,if we change the approach , we may get more bugs via automation.Breadth over depth approach to testcase automation gives more bugs. Instead covering all features ,it is better to go for a specific features covering all the objects. There should be a balanced approach for selecting depth and breadth testing.

The more the framework easy to use ,more and more test engineers will jump into automation that will intern increase the coverage and testcase count.Automation will never face the starvation issue as there will be plenty of test cases to automate.

This framework is an application independent way to design testcases. The testcases are supported via data table. Mostly a database supports the vocabulary of keywords.A core library is required to read all these lines from Excel or data source and convert them into equvalent QTP /Selenium/other tool command.

The keywords are simple and clear [like open_bowser,Close_browser,setValue].Each keyword will be mapped to a known action.There can be multiple inputs for the action. Like -Locator,Text. Even an action can be an input to another action.Now this framework is combination of tables[Cycle table,test table]. Below are the sections where I have shown how is the structure of a keyword driven framework...

In keyword driven testing there exists five different parts..

  1. Control file
  2. Testcase file
  3. Startup Script
  4. Driver Script
  5. Utility Script
What is Control File?
Well, this file contains all testcases. Based on some parameters or conditions some of the testcases will be executed.
What is Testcase File?
Detailed design  test cases.Mainly written column like-Keyword ,Object Name , Parameters.It is an excel file in most of the cases.This will also include the start up and tear down methods.
What is Startup Script?
This is the initialization script. It creates the environment to run the testcases.Assembles the testcases those are marked to be runable in control file.
What is Driver Script?
It drives the test flow,gets input from stratup script [Which scripts to run]. It reads the testcase file for the corresponding steps. It takes the input passes that to script engine to generate the code according to the test tool and corresponding script gets executed.
A good framework should allow any combination of features to be tested. Initial days we used Excel to select testcases.As technology progressed,we use web interfaces with simple UI where check box can be checked to select testcases.User friendly,self explanatory web interface ensures good productivity as it reduces the technical barriers.Jenkins,QC are very good example.  This feature will support mini regression to full regression.
What is Script Engine or Utility Script?
Script engine is responsible for tool specific code generator. Kind of tool understandable code generator. The code will be executed on the tool.
Utility script: Utility Scripts are the low level test steps to perform some operation.Some common utility methods are-

  • LaunchApplication
  • CallScript
  • ReportLog- It is also known for debug logger. It gives more clarity on real bug and noice.

Sometimes all supported libraries we put in Utility

  • File Handling,
  • String handling
  • Database access
Execute -Execute and Execute should be main aim for this approach. This is keep the maintenance low and scripts updated. This will give great ROI.
There may or may not be any object repository present. There is further reusable concept hence all functions, keywords etc will be generic in nature.
  • Change management is easier
  • Does not dependent on the UAT
  • Much easier to extend.
  • Similar to functional test script so easy for non technical or non tool experts in terms of understanding.It is easy for them to develop scripts.
  • When developed block by block it becomes highly user friendly.Real power of functional decomposition can be seen. 
  • This will also cut down the test engineers tedious and lengthy testcase creation time. Manual testers will have  bugs.
  • Easy to modify so maintenance cost is low with compared to other framework.So greater ROI could be achieved.
  • Advanced keyword driven framework is having two layers…one is pure code layer and other is pure English or non code elite language.Functional people can prepare the test cases very easily.
  • If the infrastructure is separated from logical layer then any test cases can be automated.
  • After a couple of months when the basic building blocks(unique components) are ready to design  a functional flow, the development cost will down.
  • So this will provide better ROI.
  • If application is getting changed, it can determine the changes on run time .It can change only that component.No need to debug the entire script. Hence it is very easy to maintain.

Due to simple English interface, it is best for manual test engineers,functional analysts and business analysts.

  • Naming convention is tough for large projects.
  • Very tough to remember the keywords.
  • Initial investment for framework design is high.
  • High skilled labor is must for initial days of the project.
  • Not very to understand for new comers.
So to conclude, it is one of the finest flavor of data driven testing but having most time consuming process.But the keyword preparation is one time effort. Once completed,it can handle any size of project.This approach will uncover the code issues in an automation at earliest. Automation code is also a software code that needs to be tested.By this approach,we can test automation code as well.As a result it is more likely to uncover more bugs.
image credit
What is Keyword Driven Framework?? What is Keyword Driven Framework?? Reviewed by Animesh on September 07, 2010 Rating: 5

What is an Automation Framework All About??

April 15, 2009
Image credit:
Before we go ahead and discuss more about automation framework, let us understand few concepts about automation  testing.
Lets us talk about few myth first about automation testing.

  1. Test Automation is very easy and anybody can implement test automation strategy. Anybody can develop a script.Well,this myth is developed by the sales person of the company to make sure their tool gets sold out.While demoing their tool,they always follow the record playback strategy. just record a simple flow , enhance by using test data concept,run the script shows the result.With these sales technique we started feeling that any body can script.But reality is completely different. Automation is not all about record and playback.It needs to be designed,then developed and tested. Definitely test engineer needs to be from some programming language background or they need to learn languages,at least the basics.Automation development is not as complex as application development but these source codes must be saved as application source code. 
  2. Tool design is better than evaluate tools.Mainly  this concept came from the expensive tools available in market. Test engineers or organization try to avoid the license cost of such tool. They prefer to build their own tool. Or try to use free scripting or languages like VBA,macro ,perl,Ruby to design test.These thing can be considered as lean approach .But not Automation.See the below links for lean approach.
What is lean approach,All Lean Tools,Thinking Lean, Muda
But the fact is when you use a proper tool to automate the productivity will be higher which is significantly less than the licence cost. Using other technique can be considered as research and development but not proper automation.
With a proper test tool with licence will give automation test engineer more command over the QA arena. Let us see how?

  • Most advanced tools will have a proper encapsulation of programming. So less advanced coding required here.
  • More the mature tool more great customer care.But opensource tool will have less or no customer support.
  • Advanced tools will get constant update about technologies.
  • Huge support base including different forums.
  • Readily available framework and integration with other technologies.
  • Advance tool will have more functionalities in different technologies covered.

Automation Framework:
In general Framework is an integrated part of automation testing.This is an structured combination of ..
a.Testing best approaches
b.Testing Assumptions
c.Testing Practices
d.Testing Concept

before going for a automation testing...nono....forget about automation testing prior to any testing, client and service provider need to sit together and determine the approaches...

let us understand the tool based testing...
Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may require additional effort to achieve real and lasting benefits. There are potential benefits and opportunities with the use of tools in testing, but there are also risks.
Potential benefits of using tools include:

  • Repetitive work is reduced (e.g. running regression tests, re-entering the same test data, and checking against coding standards).
  • Greater consistency and repeatability (e.g. tests executed by a tool, and tests derived from requirements).
  • Objective assessment (e.g. static measures, coverage).
  • Ease of access to information about tests or testing (e.g. statistics and graphs about test progress, incident rates and performance).Risks of using tools include:
  • Unrealistic expectations for the tool (including functionality and ease of use).
  • Underestimating the time, cost and effort for the initial introduction of a tool (including training and external expertise).
  • Underestimating the time and effort needed to achieve significant and continuing benefits from the tool (including the need for changes in the testing process and continuous improvement of the way the tool is used).
  • Underestimating the effort required to maintain the test assets generated by the tool.
  • Over-reliance on the tool (replacement for test design or where manual testing would be better).

A study report of US NIST says ---
This shows in 2004 the automation coverage was 5% and in 2006 it was 20% but today it is almost 50%
Software product based companies looses $21.2 billion /year due to insufficient testing.
$931 million was invested during 1999 and in 2004 it went up to $2.6 billion.
The solution may be....

1.The scope and need of testing.They must not goahead and do a testing activity for the buzzword testing--QTP etc.We need to think what will be the scope of testing.Will it be Product Oriented? Will it be Company Oriented? or Project Oriented?
Second scoping should be what kind of testing we are going to perform? Functional Testing or Load Testing or Web Service Testing or API Testing?

Tool Evaluation
2.The requirement gathering and identification of proper automation tool is one of the basic criteria before going to testing viz automation testing.We need to understand the nature of requirement , priority of such requirements and priority of them There are several cases where requirement gets changed in every alternate day(Agile Testing).

Now second step of this section is to evaluate the testing tool.The Major search about testing tool in a search engine  as as follows:

for tool selection there are several charts are available..

Over 300 tools are readily available in market.
Load testing-54
Functionality Testing-60
Java test tool-48
other Web Testing-76
So it would be a very tough choice of tools.We need to prepare a checklist for evaluation of such tool.Identify required resources in company.Now do prototype for few sample flows.Check for different actions,Validation and reporting feature.

Tool Evaluation Criteria
3.Again if it passes the above stages then business people and technical persons are required to sit together to identify the scenarios to automate and most critical prior to automation testing.100% requirements can not be covered and is not required for any automation.
Here is a article that gives more insight of tool evaluation-
Some basic Tool evaluation criteria are:

  • Easy coding and develop scripts
  • Easy maintenance
  • Easy and huge support base
  • Less cost
  • Available training doc or training materials
  • Technologies support
  • Execution in distributed mode.
  • ROI
ROI-The real power of Automation
Classic Way of ROI calculation:
ROI=Cost Benefit/Cost
Manual Cost=Development cost+Execution Cost+Maintenance Cost
Automation Cost=Price of Hardware+Price of Software+development cost+Execution Cost+Maintenance cost
ROI=(Manual Cost-Automation Cost)/Automation Cost
Now this calculation is one time calculation. If organization test multiple times ROI will be higher. So if you test your application multiple times for regression or for patch testing,It is wiser to move into automation.

More About Automation Framework
4.Then comes the framework.So it is based on the application and approach.
To my view point application comes first then the approach.

The main advantages are:

  1. Consistency -process repeat ability and resource independent
  2. Structured approach
  3. Maintenance becomes very low.
  4. Maximum Re usability
  5. Proof of adequate testing
  6. Unattended execution or nightly execution
  7. Maximum coverage of requirements.
  8. Common standard across projects 
  9. Non Technical person can start design code
  10. The modification during scripting should be very easy.
  11. Data Orientation for execution over different data set
  12. Huge reduction in testing cycle time
  13. Reduce overall QA cost
  14. Eliminate tedious manual effort and manual error

Automation testing is lengthy process as well as very sophisticated approach.its life cycle can be written as  requirement analysis --->test planning-->test design-->test development--->execution of test cases-->bug logging--->reporting ---> maintenance.

More over automation is used to downsize manual effort not to replace the same.

Types of Framework:

There is no hard and fast rule to design a framework.
a.Record and play back
a.Data Driven Automation Framework
b.Keyword Driven Automation Framework
c.Modular Automation Framework
d.Hybrid Automation Framework
e. BPT Framework.

For more details click here
Test execution toolsTest execution tools replay scripts designed to implement tests that are stored electronically. This type of tool often requires significant effort in order to achieve significant benefits.

Record and play back
This is simple record and then playback.the only enhancement is to parameretize the data that changes.Less reusuability. It is only driver and deliver concept.Capturing tests by recording the actions of a manual tester seems attractive, but this approach does not scale to large numbers of automated tests. A captured script is a linear representation with specific data and actions as part of each script. This type of script may be unstable when unexpected events occur.Most of the time scripts are not modified to check the change in software but a new script is build to test.It is often referred as Linear framework.
When you can go for record playback?
Mostly when your application is like mainframe where we can only tab around the application and send test data.So if any application only accept keystroke , then record and play back could be an option.
Script1+Test Data 1>i.e/any browser-->UAT
Script2+Test Data2 +>i.e/any browser-->UAT
Script3+Test Data3 +>i.e/any browser-->UAT
It is very easy way of developing and delivering script.
find disadvantages here

Data Driven Automation Framework
Repeated use of Test Scripts with different inputs.A data-driven approach separates out the test inputs (the data), usually into a spreadsheet, and uses a more generic script that can read the test data and perform the same test with different data.Generally where the flow remains same and objects does not change, Data driven framework can be very usefull.Testers who are not familiar with the scripting language can enter test data for these predefined scripts.Input data is given a pool of data set.Even the output data comes out from the data pool.It helps us to reduce coading for a large dataset.Ease of Testing of Time consumption.It is best used in IBM-Rational Robo.Data is not hard coded here,instead data are stored and provided from data source like -XML,Excel,flat file,Database or combination of all.As data layer is separated from code , same scripts can be reused multiple time.

find disadvantages here

Keyword Driven Automation Framework
Steps and data are feed to the automation tool.Automation tool judges the steps and input data from a data pool,It gave the output data.In a keyword-driven approach, the spreadsheet contains keywords describing the actions to be taken (also called action words), and test data. Testers (even if they are not familiar with the
scripting language) can then define tests using the keywords, which can be tailored to the application being tested.It is very sensative towords data design.This is based on generic key word based.
The advantage is the setting of keywords can test window/web/oracle etc based application.
More details can be found here
Modular Automation Framework
In this approach,The whole scenario is divided into small business flows-called module.Those modules are written in a library file as a functions.
In the script level the high level business functions are called.They are linked with small module functions.more over this is a very simple approach over record and play back.Objective to make it modular is nothing but to make the script reusable.There are driver in the main script and modular functions are stubs.We create shared object repository for Modular Automation Testing.When Application is huge, Initial development may go high but later the development cost just go down. Maintenance is easy and straight forward.

For more details click here and here.
Problems with this approach can be found here.

Hybrid Automation Framework
This is a combination of all frameworks .They are used whenever necessary.So is a summation of Data driven--keyword driven--Module approach.The main advantage is this is logic driven.Keywords are logic based.
For more Click here

We generally carried by the buzz words in the name of the framework. So migrating to a good framework may not be necessary for your organisation if the  current one is doing just great. So better not to reinvent the wheel or don't change it until it is broken.

few Guidelines while designing the framework:
1.DO version control(use CVS,SVN).
2. Try make the script Environment neutral.
Read More here for speed up automation with file size. Even Object Repository can be a nice part of Automation.How? Click here to learn more
Hoe to improve the framework? See here and here
Image credit:

What is an Automation Framework All About?? What is an Automation Framework All About?? Reviewed by Animesh Chatterjee on April 15, 2009 Rating: 5
Powered by Blogger.