Results for Idea of Framework

How To Create Test Data for Advanced Test Structure or Test Framework

August 07, 2015
In my last couple of posts, Structure of Automation Test Cases and
Best Test Structure::Hermetic Test Pattern , I have shown each testcase
should be independent to each other. But to make them real
independent it requires testdata manipulation. Testdata also need to
be independent ad unique in nature.

Why it is required?
It is required to stop data pollution.Stop using polluted data usage
inflow and outflow. 
I am writing few common methods to make testdata independent:

Ways:
  1. Use testcase id or name at the end of the testdata like Cutomer_Name=”Rabin”+”Test123”;
  2. Use time stamp to make testdata unique like-Customer_Name=”Rabin”+”3103201515082015”;
  3. Use random number generator to create unique testdata.like-“Robin”+rand.nextInt(900) + 100;
I like the Random number generation method and if it is not possible to use, I generally use the testcase id.
More note on Random number generation:
RandomStringUtils.random(10, false,true);

you need to import - org.apache.commons.lang3.RandomStringUtils package
this is very useful when generating 10 digit random number.
AtomicInteger counter = new AtomicInteger(1);
counter.getAndIncrement();

This is also very useful to get unique number.


image credit:http://www.sqledt.com/
How To Create Test Data for Advanced Test Structure or Test Framework How To Create Test Data for Advanced Test Structure or Test Framework Reviewed by Animesh Chatterjee on August 07, 2015 Rating: 5

Best Test Structure::Hermetic Test Pattern

August 07, 2015
In my last post-Structure of Automation Test Cases, I have shown how we can improve the test structure and make a better one.Too many tests will make feedback loop delayed and too less tests may not have enough coverage.Sometimes too many tests are added to a single E2E(End to End) script.This kind of test design will make it spaghetti. The goal is to design a significant approach that will lower down the feedback loop.
Now lets looks at the question below:

  • What is the biggest benefit or goal ? 
  • What is the cost ? 
  • How is the team? 
  • What training team needs? 
  • What is holding automation back? 
We have to prioritise the issues and resolve one by one. Now in this post I am trying to venture the pattern on which the structure is made. The test structure is made on Hermetic Test Pattern. As per the Hermetic test pattern each test should be completely Independent and should not depend on other test. So,It should be atomic in nature and self sufficient.As per the maintenance concern,each test case should be maintained concern,each testcase should be maintained separately and if any dependency outside of the test found(may be a 3rd party library,3rd party service) test engineer should cut the linkage with the service.
The best way to start is to look into smaller objectives but add test slowly to make it mature.
Advantages: Each testcases are atomic in nature so no dependency on anything. Test takes clean start as precondition creates all test data No polluted data inflow our outflow. We can execute the test in different order on need based. We can create more molecularity which increase code re usability. As each testcase is atomic,parallel execution is still possible.The deep advantages are:

  1. Increase confidence in software quality.
  2. Earlier time to produce the software to the market means more revenue.
  3. Reduce the cost in testing.
  4. Consistent repeatable testing
  5. Run tests unattended.
  6. Find more regression bugs
  7. Run tests more often
  8. Better quality software
  9. Coverage is high
  10. Find more bugs
  11. Test on different systems.


Disadvantages: Duplication of code increases if not created modular type. Design testdata for each testcase Execution time increases. High resource utilisation.
Best Test Structure::Hermetic Test Pattern Best Test Structure::Hermetic Test Pattern Reviewed by Animesh Chatterjee on August 07, 2015 Rating: 5

How to give test URL to my framework

January 26, 2012


This section is dedicated to my viewers.Here they can ask question and I will be happy to help or redirect to some page what is best suitable for them. 







Hi Animesh
How to change/give environment name dynamically -to my framework? .Right now we are hardcoding in our scripts.

Regards
Veenetha
Hi Veenetha,
Thank you very much for writting me!! I am trying my level best to solve your query.

Well veenetha,there are several ways how you can insert environment name into your framework:----
1)Ofcourse,I agree,the hardcoding into a script is one of the options.
 String  URL="www.abc.com"
2)The second improvement could be,you can create a dialog box(input to take the environment.This you will place master/global driver and the value you will pass to the child scripts.

Or,you get the URL from user and save it to the notepad and read it dynamically afterwords.
3)Create an environment file for providing URL.I found advantage on creating a separate file for environment details.Updation is very easy and effective.
Say active_URL="google.com"
Tomorrow you can change it to
  active_URL="yahoo.com"
  active_URL="google.com"


My personal suggestion is to go with either 2 or 3.
How to give test URL to my framework How to give test URL to my framework Reviewed by Animesh Chatterjee on January 26, 2012 Rating: 5

Features of today's Automation Testing Framework Part-3

January 08, 2012
Hope you have enjoyed Part one and part two of Framework Building. Now I can say It is bit updated framework. This post will talk more about the fine tuning your framework development.
Portability:When I said Distributed nature in part two What I mean that any test can be operated from any machine.Let us improve the framework little bit more. I can copy the master driver which will automatically create the folder structure add up the necessary dependent files and will be ready to run any testcases.
Ease of Functionality:  This can be achieved when we can give user a platform where user can plug in and plug out any extra feature he/she required/not required respectively.Like Manager , business analysts and functional people can control global testing without knowing the tool.This will truly become Skill Utilization too.
Capture all : Yes!!! that is the key for a successful  framework.Always capture Step information,log,Success criteria,Exit and entry criteria and Error Information.
Scoping and Descoping:   The more you make your framework user friendly , by providing Easy controlled and controllable actions ,More it will be a go to tool for all.This feature will allow business user to scope a set of testcases or scenario for a specific business flow.
Advanced Coverage: The more you fine tune your application the more the test engineers are going to use it.If your Framework supports Negative testing ,their is nothing like it.Manual persons also can jump into your framework to speed up their application testing.It is nothing but to increase your coverage i.e Automation Coverage This simply saving manual effort.
NO Object Repository:  Now a days testing demands very fast execution and report generation.Please refer one of my post on Having Repository.So now a days people like to have a pattern based testing where there will be no Object Repository.Pattern will detect the type of field and will do actions.
Hassle Free Integration: Framework should support a Hassle free way to integrate a newly build module. I suggest a GUI component to do this integration. By this an Automation test engineer or a functional test engineer can make a code and can submit for further use.Framework should not be so complicated that integration of modules will take lot of time and it requires a lot of effective testing time.
Implementation of Source control System : This is one of the basic criteria to make an advanced framework.Yes!!! Track your source code to make everybody understand what kind of change is applied upon the code.History will make sure if the piece of code is untouched or error prone and who has done what kind of change.There is always a chance that the new code is error prone and can create regression. For that case we can always go back to previous version.   
Recordings: New framework should support video recording of each and every testcase execution report. Based on the requirements, framework should support the pass video or fail video along with steps and logs.
 
Features of today's Automation Testing Framework Part-3 Features of today's Automation Testing Framework Part-3 Reviewed by Animesh Chatterjee on January 08, 2012 Rating: 5

How Object repository can affect execution performance?

September 22, 2011
 image
Introduction:
There are several approaches in the market to generate your script and  the framework implementation. But we need to think a manner where script development time ,script maintenance time becomes less.Most importantly we need to find an approach where the execution time is also become less.
What is the importance of downing the execution time?
Well, i am inspired by a philosophy that automation test engineers should not loose time by executing test script. All the test script should be configured once and should be able to run unmanned manner. More over these scripts will run in a computer assisted environment. Computer is not that intelligent as we are.If you insert enough intelligence into script it might slow down the process and on the other side intelligence is less..there is a chance that script will error out. So what is the best approaches to automate your script that will serve long without issue is a question.


Role of Object Repository
Object repository is a place where your testing tool use to store information about objects under test for a flow. This may be .tsr file or a straight xml file.During run time script takes reference of  this file and identify the objects.
image
It is seen if the OR size is less that means test engineers are doing more descriptive programming.Descriptive programming is very powerful but it is a time consuming process.Hence it will slow down the process.
On the other hand if the OR file is big.The seek time will be high. There might be a scenario where object conflict will occur.This also slow down the process.
image
Here this picture shows if OR files grows high in Size how the response time of the scripts increase.
image
image
As per experience, An O.R file should be between 5 to 12 MB..Above that or below that performance is not as good as we expected to be.

Check out here how to speed up an automation framework
Technorati Tags: ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Windows Live Tags: Object,repository,execution,performance,Introduction,script,framework,implementation,manner,development,maintenance,Most,importance,philosophy,automation,computer,environment,intelligence,error,Approach,Role,tool,objects,reference,size,Descriptive,scenario,Here,files,response,Above,Check,scripts,engineers
WordPress Tags: Object,repository,execution,performance,Introduction,script,framework,implementation,manner,development,maintenance,Most,importance,philosophy,automation,computer,environment,intelligence,error,Approach,Role,tool,objects,reference,size,Descriptive,scenario,Here,files,response,Above,Check,scripts,engineers
Blogger Labels: Object,repository,execution,performance,Introduction,script,framework,implementation,manner,development,maintenance,Most,importance,philosophy,automation,computer,environment,intelligence,error,Approach,Role,tool,objects,reference,size,Descriptive,scenario,Here,files,response,Above,Check,scripts,engineers
How Object repository can affect execution performance? How Object repository can affect execution performance? Reviewed by Animesh Chatterjee on September 22, 2011 Rating: 5

How file size can affect execution time?

September 18, 2011

ktip
Introduction:
There are several approaches in the market to generate your script and  the framework implementation. But we need to think a manner where script development time ,script maintenance time becomes less.Most importantly we need to find an approach where the execution time is also become less.
What is the importance of downing the execution time?
Well, i am inspired by a philosophy that automation test engineers should not loose time by executing test script. All the test script should be configured once and should be able to run unmanned manner. More over these scripts will run in a computer assisted environment. Computer is not that intelligent as we are.If you insert enough intelligence into script it might slow down the process and on the other side intelligence is less..there is a chance that script will error out. So what is the best approaches to automate your script that will serve long without issue is a question.
Please read here for the best Approach for automation.
File Structure and size:
Once you finalized your approaches the next important aspect is the component development. Yes the most important component of any framework is File.
Now what ever framework we develop ,the structure always resides inside the Hard disk.Even if we came up with  high speed data bus,high speed data transfer mechanism,during execution it is seen the number of files ,structure and file size slow down the process.
  • if the file size is very less  that means we need to incorporate a lot of files….. the seek and latency time goes high
  • if the file size is high that means we are including every thing inside one library and seek time becomes high.

image
The guideline for file design is written below:
  • try to write same functions into a file but do not club all the function into a single file.
  • Number of files which will be loaded first (like-common functions –common to the entire flow..needs to be called almost every steps, path library) should be less 2/3.
  • library specific to a module will be loaded dynamically and will be released after use.
  • Garbage collector should be called after dropping one file . it depends on the scenario to scenario.
  • More over any point of time the files are handled can not exceed 3/4.
  • Try using lightweight HTML reporting instead of Excel Reporting.(excel-2000 and above takes heavy space in RAM….it is always greater than 2 MB.
  • Using of CSV as a input file is the best approach. One more best approach is to use XML.
How file size can affect execution time? How file size can affect execution time? Reviewed by Animesh Chatterjee on September 18, 2011 Rating: 5

How to Speed up Automation Framework?

September 17, 2011

ktip
Introduction:
What is the catch for defining and developing new framework? somebody might tell the cost for development should low. High productivity.Easy maintenance might be another aspect.But we never consider the time factor for running the scripts. After all this is computer assisted testing.Who cares about the timing? As long as it is a unmanned execution ,nobody cares for execution. This article talks about the a brief idea about how to speed up an automation execution time.
Every scripts are based on some framework.May be a record playback or a serious framework. So if you can configure your framework effectively, run time or execution time will go low.
I have jotted down few improvement areas 
We need to design our automation such a way that we can swap to different tools when needed. We need to preserve the tool independence. Apart from these test engineers should take advantages of some of the downtime or judiciously takes time away from regular tasks that have a due date pretty format, a lot can be accomplished.Object oriented design allowed us to play into test harness in a standard way and reuse existing components to test new features,minimizing the code(power of re usability).
Speeding up automation does not mean a good architecture or a well defined framework. Even though with lot of investment if we create a good architecture for extendable automation framework and script design.it may still fail even though if found high quality bugs.Initial success is no guarantee of long term success.Technical aspects are not the only factors that influence a project.

Before going into details of speeding up automation,we must consider 2 aspects of architectures.

  1. Technical architecture(Software engineering/experience)
  2. Business architecture (Goal of business,bug test case automation)
The business architecture should always exists. It is not only automator's property or a part of his workflow. Manager should push others to learn about automation framework.Manager also should know and understand the power of automation.
The framework creator should be given enough time or encouragement to share his knowledge and expertise.The engineers also need to understand the programming knowledge and enhance the skill.
The automation effort should be tracked in to a good way in a holistic manner.
Speeding up automation also can be found by finding the real objective. The objectives needs to be identified,reduce manual labor costs,improved product quality ,increased testing flexibility,free manual resource for more area to test,reducing strain on network resources as fewer testers are required.Strong support from management continued with the assurance that issues such as funding,test design quality,originally from outside the team and affecting the automation's progress needs to be addressed.Sometime it is better to have a view from outsiders about the process,framework and code.The feedback may a give a lot of internal changes.

Independent testing team concept is a booming concept.Now a days it has become a service line for many companies.Independent testing service optimizes resources availability provides standardize methodology,quality. This process also ensures minimizing defect risks for predefined requirements. For really going good, I recommend, to divide the group into two different ways:
  • Testers
  • Core automation engineers
Testers: Mainly the test engineers who drives testing(Functional testers,QA or acceptance testers).They are more close to the business. They understand the business goal.
 The responsibilities are:   
  • Defines the testing scope,define what is to be tested.
  • Defines the test plan that contains testing quote.
  • Create manual test scripts,entry and exit criteria(outline of the test). 
  • Create report as a proof after executing the testcase manually
  • Supply test data as when it is required
Core Automation Engineers: They are the one who translates the manual scripts into an automation testcases. They may be far from business goals more close to technical aspects nearer to developers. They should have a great programming knowledge.
The responsibilities are:
  • Prepare a road map/test plan for each system to be tested.
  • Prepare automation test script to cover or apply each test quote
  • Write reliable scripts to collect final outcome(It should contain report,bugs,logs etc) suitable for manual test engineers.
  • Become an expert in execution who can understand the script,API.
The separation between testers and automation engineers is necessary as  testing and emulation of user input should not clouded and cross contaminated. Automation engineers needs to have testcases to automate. The testers group are the one who provides them. To bring up speed, testers need to have a constant flow of manual scripts towards automation. Many automation projects either fail or become slow if they don't perform this activity seriously. Converting manual test engineers to automaton test engineer gives a poor result or the strategy may fail.But this way of creating a testing team could be an issue in hiring as testing team grows high. Recent trend is to hire developer vs tester ratio becomes 1:2 from 1:3 to 1:4.So funding could be an issue.

The test automation architect is a key player who is responsible for developing and maintaining the test framework.Architect defines and deploys best practice and coding standards to be used in implementation of the tests and train  test engineers to do minor changes in the abstraction layer. So when your test set is little mature, hiring a test architecture will definitely ed up the scripting and further process.

Abstraction Layer:your abstraction layer should be a balanced one. If the abstraction layer is too thin(too many fields are set in a specific test script), it could lead to a cumbersome maintenance activity. (if any of these filed gets updated,hundreds of field needs to be maintained). If your abstraction layer is too thick(too large or too advanced), the complexity will be too huge for this layer

If we are advancing towards an upgraded framework,test engineers need to learn new technologies,new techniques. It means we need more upgraded class room training or offline training. So hiring experts or creating an expert group who can train and step in when needed,is a good idea.But that may significantly hit the ROI.But we can not overlook the cost of learning and upgraded yourself.

When your testing process are really matured and want to do more speedy testing. Daily Build Engineers come into play. The person in charge of daily builds ensures that the test batch is executed ,makes the first analysis of the result and pass the work needed to analyze the test result to the automation test engineers or QA.A good CI(Continuous Integration) process is must.Hudson or jenkins or cruse control etc will make the automation process more speedy.
  



How to Speed up Automation Framework? How to Speed up Automation Framework? Reviewed by Animesh Chatterjee on September 17, 2011 Rating: 5

12 top features of today's Automation Testing Framework

August 15, 2011
Hope you have enjoyed Part -1 of Features of today's Automation Testing Framework ...Now this is the second part where i am going to discuss few advanced feature of automation framework.

1. Extended environment and multitool Support: Most of designed framework is not environment neutral. They support only one Operating system or one tool specific. Please build a framework that will work for multiple OS or in multiple tool. This can be achieved through designing a  loosely coupled system.

2. Your framework should comply with HIPPA or SOX guideline.

3. Library Architecture: The library file of your framework should have a proper and clear architecture to support the advanced configurations.

4.Advanced features: Any technical change should be caught by framework itself and handle properly.In this way also try to capture the delayed fuse bugs like -memory leak wild pointers etc.

5.Configure Recovery Scenarios: While advancing forward ,framework should have ability to handle the unexpected exceptions. Every tool has its own way of handling exceptions.Use that to configure recovery scenario.Please handle known issues with code itself. As the more intelligence you will put into a framework..it will slowdown the process.



6. Reporting feature:  In my opinion the new age framework should report in HTML format. The header level talks about the summary report..No of pass ,no of failed, Execution start,Execution ends,Duration,Performed by. But the downwards of the report it should give the detailed view...each testcase..expected objective,actual objective,status, pass/fail.
A good option could be configured that if there is a failure case, framework must take a screenshot ,save it as .png format and give the path inside of the report.
Another improvement could be to mail the details of step to the point of contact or the automation engineers.
marketing strategy (presentation,report sharing,demo) can also bring lot of ideas .The main mantra is to engage all.

7.Self documentation: This is one more improvement that can be implemented is the self documentation of steps performed. Gradually this feature will give proper navigation step validation and data validation in future.
8.Data driven Capability:  The framework should support a test that is data driven.Sometimes we see on the same form , if the data entered is different can lead to a creation of different products.So a framework must have a plug in data driven functionality builder.
9.Ease of debugging and logging: A framework should give a easy debugging feature so that the code executed by any test engineers or a customer can debug the code easily. One of the best method could be linear cod generation.
10.Execute without monitoring the scripts: A better ROI can only obtained only if the test is executed without monitoring.Test engineers only responsible for creation of correct scripts. Execution can only happen when it is not monitored.
always listen or give proper value to the feedback of the users.They may pin point lot of relevant process and practise issues.Even they may advice standards.This is very important for long term success.
11. Dynamic Data handle: When a test engineer wants the same test to be executed on different data.Framework should support him with easy excel plugins of data and also give flexibility to repeat the test how many times needed with different data.
12.Distributed Nature: Framework should have flexibility to design and execute the test in different machines without any problem. Design and execution should be in Distributed mode.


Image Credit::insuranceframeworks
12 top features of today's Automation Testing Framework 12 top features of today's Automation Testing Framework Reviewed by Animesh Chatterjee on August 15, 2011 Rating: 5

Top 10 Features of today's Automation Testing Framework

August 15, 2011
Most of the time the test automation engineers build framework or rectify the current one or upgrade the current framework. Importance of framework -

  1. Flexible
  2. Reusable
  3. Easy Maintenance
  4. Better ROI
Examples of automation:
  • sending bank statement
  • credit salary information
  • Any manual process which is repetitive and  tedious.
For automation , we need to pick up a tool and script using underlying programming language. Example of the tools-UFT/QTP,Selenium,Silktest,RFT,Test partner 
The steps we need to follow:
  1. Collect the acceptance testcases/ regression testcases
  2. Record and code the script
  3. Improve the testscript in order to reliability.
  4. Parameterize  the hard coded values
  5. Separate the test data from code by moving variables to a separate files
  6. Create utility functions
So creating a basic framework needs the following things covered:
  1. Prototype components
  2. Test designing
  3. Create test interpreter
  4. Create AUT specific test drivers
  5. Add repository
  6. Add logger,reports capabilities
  7. Create test documentation.
I am trying to cover all the required features that can be included inside a framework--

1. Coverage---Yes, this is the most important feature of any framework. In my view a simple framework which can cover more is much more acceptable than a complex framework which covers less. So decide correctly by which approach you can get more coverage.

2. Business Process: A framework is very much useful when a non technical person can create an automation testcase and execute the same. Basic building block of such kind of framework is to create business process. A non technical person might not understand your piece of code but very well understand the Business  process. He should not bother about the under laying code.



3. Interface to create a Modular approach: This is continuous from the above point.How a non technical person will interact with your automation code? Yes....they need an interface to design the test. Quality center is a classic example of interface. But that is costly.Framework should have an interface where functional people can use that and prepare their test.

4.Re usability:    Try less coding and try to reuse most of the possible combinations. This will reduce ambiguity. Maintenance is much more easy when you have less lines of code. Redundant lines sucks a lot in later phase.

5. Proper Naming Convention:   Please use this whenever possible. Try to make a practice. This will reduce ambiguity inside the code. It is easy to understand when we have a proper Naming convention.





6.Consistent:  By using point-4 , point-5 and using proper commenting you can get consistent test script.Framework should support/Accept only the consistent script.

7.Plug ins:  When you are preparing a data driven framework or testing a script for multiple data set, you require lot more data files to be processed. Framework must support a plug ins for data for driving the script or data pool to supply during testing of same scenario for multiple data.

8.Defined folder structure: A framework structure should have a defined folder structure before hand.

9.Defined Maintenance process: The maintenance process of a framework should be defined previously. The process to update a script /asset/code has to have a prior approval process and version control mechanism.

10.Easy Refactoring and update: Refactoring is an essential part of automation testing in order to restructure of test code or test data for ..

  • increase internal efficiency
  • Stability
  • support upgraded technology
  • Support change in functionality
  • increase maintainability. 



This is the basic building block for an automation framework. Please visit Part-2 for more advanced feature.A better communication between testers and automation engineers is necessary to ensure framework design meets testing objective. Any test framework design effort is very serious factors for project development. A good planing ,scheduling and budgeting is required. It is important to get feedback from testing team. This may lead to greater learning and idea generation. there may be many cycles before perfection.Accept the short fall and work towards it.If the project is complex ,big and we are at a tight schedule.We need to scale back .Communication is the key here.
While designing , we need to make sure-

  1. Find relevant information,code,examples from internet. It is available in different form.
  2. The automation many be developed by a group and maintained by another group. Transparency is the key.
  3. While designing the framework, we need to take care business issues,people and process issues and technical issues.
The objective of the automation are:
  • Test execution can be completed faster
  • consistent and effectiveness
  • Re usability makes better ROI.
  • Reduce the cost of testing
Few basic criteria:
  1. Tool must support UAT
  2. Multiple releases should be carried out
  3. Less manual intervention
  4. Project should be in spiral model.
  5. UAT should be stable
  6. budget should be there to carry out expensive automation

Image Credit::insuranceframeworks

Top 10 Features of today's Automation Testing Framework Top 10 Features of today's Automation Testing Framework Reviewed by Animesh Chatterjee on August 15, 2011 Rating: 5

Test Automation Framework Designs | Test Automation Framework Designs Online | Download Test Automation Framework Designs « Information Technology « Study Aids « Examville

August 04, 2011
Test Automation Framework Designs | Test Automation Framework Designs Online | Download Test Automation Framework Designs « Information Technology « Study Aids « Examville Test Automation Framework Designs | Test Automation Framework Designs Online | Download Test Automation Framework Designs « Information Technology « Study Aids « Examville Reviewed by Animesh Chatterjee on August 04, 2011 Rating: 5

Automation Testing Framework From Management Perspective

August 03, 2011
Framework is the integral part of any automation development. This article talks about the key features of today's high end framework design from a manager's perspective.
Before we start lets understand what management wants from automation:

A good structure that will increase the re usability and reduce the cost:

Abstraction is the key for a good framework.  Three to four folds abstraction is believed to be best framework. In the low level all wrapper functions to be built on top of GUI objects. This layer is tool specific, but more we go upwards, it will be generic.
In second level, the business component or high level components are built using the wrapper or low level components. On top level the business components stays. That is the core business functionality. The business user should not be bothering about the middle layer or lower level layer of testcases. The early bird may catch the worm but the second mouse gets the cheese!!.

If automation is started too early ,the application could vary greatly which is notably increase the cost of automated test maintenance. If stared too late , automated tests would be executed too infrequently to justify the expenditure on automation.If not all our automation regression cycle may delay the release and product may go to the market very late.

Easy Shifting to other tool:-Portability:
So the transformation is something like lower layer is tool specific but more you go towards up, it is all about the core business. So migration from one tool to another is easy. This plan is required as what is best today may not be suitable tomorrow.

No dependency on Core Automation team: Everybody's right to automate:
Abstraction is also required when there is a mixture of users,the tester and automator .This kind of layer architecture hides the technical details from testers.There are many online tools available to have an interface. Like-QC,Flow builder. Even old days excel is a classic example of such interface.All these interfaces are having excel like grid UI to select a testcases, simple drag and drop of components or simple keywords to connect the components. This architecture gives automation a longer life. Easy to use feature.Test engineers can think more scenarios rather the test tool. It gives freedom to the testers. Manual test engineers can benefited by simply designing the flow with this UI. Maximum resource utilization is possible in this approach.
Ease of Resource Movement:
Another way to look at it is to technology neutral test automation architecture gives advantages of resource mobility. Even different segments of users can take part in application testing and can get benefited.  When there is a need like attrition of test developer or executor, this process can eliminate the vacuum. More over this approach is more dependent on process rather people.

It is always better to keep technical knowledge requirement to a minimum for business-level testers,but provide training ans support for any technical tasks for doing such work. Organisation viz management gets benefited if non technical or less technical testers can develop less technical automation scripts,execute,debug and report. In that case core automation team can look into main functionality or core component development.

Second important aspect of framework is the separation between test and execution tool.A good structure is made of
  • Scripts
  • Utilities
  • Data files
  • Programming practices and guideline
  • Well structures
  • Modularity for reusability
Low Maintenance: 
In agile world,where functionalities are getting changed , the maintenance or upgrade cost will be real low for a nicely structured framework. Object map design is another best practice for good framework. Combined these all can lower the maintenance cost.
Below are some of the links which can give more detail view of framework concept:

Idea of Framework
Perspective of Automation testing framework at different phase
Test Automation framework
Features of today's Automation Testing Framework
Features of today's Automation Testing Framework Part-2

Funding:

Management often find themselves into a situation where the budget hits the automation most.Funding for automation is an important factor for survival and support today's automation.Management must understand the importance and benefits of automation.A far sighted champion is necessary to get good growth of automation.

Coverage:
A small scale start will always help to raise the awareness of the benefits of automation.Benefits must be quantitative and qualititive. Both should be clear. The goal of a coverage mode test tend to involve packed targets that appear regularly everywhere in the test model.
There can be different kind of coverage:
  1. A usecase test can often only programmed with a single specific action,making the target extremely sparse.
  2. A bug hunt mode has no specific target at all.The below guidance algorithm must base its decision on other criteria.A simplest of guidance algorithm is the completely random one. The main advantage of such algorithm is speed.,It is best for densely appearing targets.  
Unrealistic imposed objectives(total automation) guarantees failures.Automation is not always successful ,even in some organization identifies the relevant factors and learnt team aspiration whether good or bad. It is not a good idea to try to automate all tests but this can be difficult point for managers and stakeholders accepts.
Automation Testing Framework From Management Perspective Automation Testing Framework From Management Perspective Reviewed by Animesh Chatterjee on August 03, 2011 Rating: 5

Perspective of Automation Testing Framework at Different Time

August 03, 2011
Image Credit:http://textcase.com
A test automation engineer is such a coder who does not have Subject matter expertise .The question is.... does a test automation expert need domain knowledge??

My reply would be 80%-20%....80% not required 20% required. Automation Testing is activated if there are two conditions

1. if you are doing regression testing for single or multiple dataset
2. if your application is stable enough to perform the test repeatedly


Perspective of automation testing can be divided into three parts

1. Development phase
2. Execution phase
3. Maintainance  phase


When we get a big flow to automate , generally seen very less time is given to make it done.Management often misunderstood the term automatic and automated. Allocated time is not sufficient as the development team pushes the patching window or crosses the deadlines.This time an automation test engineers thinks to find out all the reusable components   so that the development time becomes less.We use a lot of loop to do same activity again and again. We try to automate what ever is possible. Our objective is to check an outline to check the overall completeness.Smart enough to complete the task???
Mostly No!!!
standard become important as more people will be developing in a very short span of time. Getting feedback from the framework users will help for latter acceptance. Automation Test development is a dedicated effort, there is nothing called free lunch.(often management asks for free or extra time of a tester to automate the flow.) Yes,we can automate in a shorter span of time. but shall we??
No!!. we must not automate poor tests.Efforts put into automating high value,high quality tests will improve the test execution and provides greater pay back. Similarly, simply automating manual testcases may not give us better ROI or a best automation suite.Select the tests judiciously.

Don't try to automate very complex flows at first.Automation should go in a slow but steady pace.Identify the proper testcase while automating the complex flows. If you automate chaos,all you will get a faster chaos.
Smoke testcases are designed to check health of the builds,releases. This type of tests saves hundreds of hours for testers and developers. Smoke testcases are good place to start the automation development, as they run almost everyday,on every checkin,or on every build.This is a less value add for testers error prone and boring.

Come to execution phase,most of the time after a successful completion of development , our scripts got stuck.This is the time when we have very less no of time to complete the task. 80% people say that development of the scripts are not proper. Even I found a instance where "for" loop sucks if it got stuck in between of the loop.Understanding the loop also a big factor for the execution master who is in charge of execution. sometimes we think if we can write statements instead of loop.Basically we look for consistency of testscripts at this point.
We need to structure our automation regression test for most successful execution.The main objective of regression suite to give confidence and assurance,not detecting bugs.



The second most important issue is data issue.Most of the time either data is not present or wrong data causes the execution to stop.Even using manual testdata sometimes gives wrong output. The third point may be data-hell condition. When we create a lot of modular scripts and try to create data for modular approach we create same data again and again.Say i need to change a set of data there is no mechanism to make the this latent change reflect everywhere. We have to open the child script datasheet one by one and change. there is no consistency at all between the data. These causes problem when running the scripts



The last part is Maintainance ...when the script is old and running for say 3/4 years ..it is extremely difficult without a proper comment or explanations.At this time we try to understand from code itself what it is doing. Remember the more number of lines of code is costly during maintenance .

In agile, we often automate some flows which are unstable. Actually, there is no value add while automating unstable functionalities as maintenance overhead  is huge.
Perspective of Automation Testing Framework at Different Time Perspective of Automation Testing Framework at Different Time Reviewed by Animesh Chatterjee on August 03, 2011 Rating: 5

What is an Automation Framework All About??

April 15, 2009
Image credit: www.cbronline.com
Before we go ahead and discuss more about automation framework, let us understand few concepts about automation  testing.
Lets us talk about few myth first about automation testing.

  1. Test Automation is very easy and anybody can implement test automation strategy. Anybody can develop a script.Well,this myth is developed by the sales person of the company to make sure their tool gets sold out.While demoing their tool,they always follow the record playback strategy. just record a simple flow , enhance by using test data concept,run the script shows the result.With these sales technique we started feeling that any body can script.But reality is completely different. Automation is not all about record and playback.It needs to be designed,then developed and tested. Definitely test engineer needs to be from some programming language background or they need to learn languages,at least the basics.Automation development is not as complex as application development but these source codes must be saved as application source code. 
  2. Tool design is better than evaluate tools.Mainly  this concept came from the expensive tools available in market. Test engineers or organization try to avoid the license cost of such tool. They prefer to build their own tool. Or try to use free scripting or languages like VBA,macro ,perl,Ruby to design test.These thing can be considered as lean approach .But not Automation.See the below links for lean approach.
What is lean approach,All Lean Tools,Thinking Lean, Muda
But the fact is when you use a proper tool to automate the productivity will be higher which is significantly less than the licence cost. Using other technique can be considered as research and development but not proper automation.
With a proper test tool with licence will give automation test engineer more command over the QA arena. Let us see how?

  • Most advanced tools will have a proper encapsulation of programming. So less advanced coding required here.
  • More the mature tool more great customer care.But opensource tool will have less or no customer support.
  • Advanced tools will get constant update about technologies.
  • Huge support base including different forums.
  • Readily available framework and integration with other technologies.
  • Advance tool will have more functionalities in different technologies covered.

Automation Framework:
In general Framework is an integrated part of automation testing.This is an structured combination of ..
a.Testing best approaches
b.Testing Assumptions
c.Testing Practices
d.Testing Concept

before going for a automation testing...nono....forget about automation testing prior to any testing, client and service provider need to sit together and determine the approaches...

let us understand the tool based testing...
Simply purchasing or leasing a tool does not guarantee success with that tool. Each type of tool may require additional effort to achieve real and lasting benefits. There are potential benefits and opportunities with the use of tools in testing, but there are also risks.
Potential benefits of using tools include:

  • Repetitive work is reduced (e.g. running regression tests, re-entering the same test data, and checking against coding standards).
  • Greater consistency and repeatability (e.g. tests executed by a tool, and tests derived from requirements).
  • Objective assessment (e.g. static measures, coverage).
  • Ease of access to information about tests or testing (e.g. statistics and graphs about test progress, incident rates and performance).Risks of using tools include:
  • Unrealistic expectations for the tool (including functionality and ease of use).
  • Underestimating the time, cost and effort for the initial introduction of a tool (including training and external expertise).
  • Underestimating the time and effort needed to achieve significant and continuing benefits from the tool (including the need for changes in the testing process and continuous improvement of the way the tool is used).
  • Underestimating the effort required to maintain the test assets generated by the tool.
  • Over-reliance on the tool (replacement for test design or where manual testing would be better).


A study report of US NIST says ---
This shows in 2004 the automation coverage was 5% and in 2006 it was 20% but today it is almost 50%
Software product based companies looses $21.2 billion /year due to insufficient testing.
$931 million was invested during 1999 and in 2004 it went up to $2.6 billion.
The solution may be....


1.The scope and need of testing.They must not goahead and do a testing activity for the buzzword testing--QTP etc.We need to think what will be the scope of testing.Will it be Product Oriented? Will it be Company Oriented? or Project Oriented?
Second scoping should be what kind of testing we are going to perform? Functional Testing or Load Testing or Web Service Testing or API Testing?


Tool Evaluation
2.The requirement gathering and identification of proper automation tool is one of the basic criteria before going to testing viz automation testing.We need to understand the nature of requirement , priority of such requirements and priority of them There are several cases where requirement gets changed in every alternate day(Agile Testing).

Now second step of this section is to evaluate the testing tool.The Major search about testing tool in a search engine  as as follows:


for tool selection there are several charts are available..


Over 300 tools are readily available in market.
Load testing-54
Functionality Testing-60
Java test tool-48
other Web Testing-76
So it would be a very tough choice of tools.We need to prepare a checklist for evaluation of such tool.Identify required resources in company.Now do prototype for few sample flows.Check for different actions,Validation and reporting feature.


Tool Evaluation Criteria
3.Again if it passes the above stages then business people and technical persons are required to sit together to identify the scenarios to automate and most critical prior to automation testing.100% requirements can not be covered and is not required for any automation.
Here is a article that gives more insight of tool evaluation-http://www.mydigitalvoice.in/2016/10/how-to-pilot-test-automation-project.html
Some basic Tool evaluation criteria are:

  • Easy coding and develop scripts
  • Easy maintenance
  • Easy and huge support base
  • Less cost
  • Available training doc or training materials
  • Technologies support
  • Execution in distributed mode.
  • ROI
ROI-The real power of Automation
Classic Way of ROI calculation:
ROI=Cost Benefit/Cost
Manual Cost=Development cost+Execution Cost+Maintenance Cost
Automation Cost=Price of Hardware+Price of Software+development cost+Execution Cost+Maintenance cost
ROI=(Manual Cost-Automation Cost)/Automation Cost
Now this calculation is one time calculation. If organization test multiple times ROI will be higher. So if you test your application multiple times for regression or for patch testing,It is wiser to move into automation.

More About Automation Framework
4.Then comes the framework.So it is based on the application and approach.
To my view point application comes first then the approach.


The main advantages are:

  1. Consistency -process repeat ability and resource independent
  2. Structured approach
  3. Maintenance becomes very low.
  4. Maximum Re usability
  5. Proof of adequate testing
  6. Unattended execution or nightly execution
  7. Maximum coverage of requirements.
  8. Common standard across projects 
  9. Non Technical person can start design code
  10. The modification during scripting should be very easy.
  11. Data Orientation for execution over different data set
  12. Huge reduction in testing cycle time
  13. Reduce overall QA cost
  14. Eliminate tedious manual effort and manual error


Automation testing is lengthy process as well as very sophisticated approach.its life cycle can be written as  requirement analysis --->test planning-->test design-->test development--->execution of test cases-->bug logging--->reporting ---> maintenance.

More over automation is used to downsize manual effort not to replace the same.


Types of Framework:

There is no hard and fast rule to design a framework.
Basic:
a.Record and play back
Advanced:
a.Data Driven Automation Framework
b.Keyword Driven Automation Framework
c.Modular Automation Framework
d.Hybrid Automation Framework
e. BPT Framework.

For more details click here
Test execution toolsTest execution tools replay scripts designed to implement tests that are stored electronically. This type of tool often requires significant effort in order to achieve significant benefits.

Record and play back
This is simple record and then playback.the only enhancement is to parameretize the data that changes.Less reusuability. It is only driver and deliver concept.Capturing tests by recording the actions of a manual tester seems attractive, but this approach does not scale to large numbers of automated tests. A captured script is a linear representation with specific data and actions as part of each script. This type of script may be unstable when unexpected events occur.Most of the time scripts are not modified to check the change in software but a new script is build to test.It is often referred as Linear framework.
When you can go for record playback?
Mostly when your application is like mainframe where we can only tab around the application and send test data.So if any application only accept keystroke , then record and play back could be an option.
Script1+Test Data 1 +repository1.mtr--->i.e/any browser-->UAT
Script2+Test Data2 + repository2.mtr--->i.e/any browser-->UAT
Script3+Test Data3 + repository3.mtr--->i.e/any browser-->UAT
It is very easy way of developing and delivering script.
find disadvantages here

Data Driven Automation Framework
Repeated use of Test Scripts with different inputs.A data-driven approach separates out the test inputs (the data), usually into a spreadsheet, and uses a more generic script that can read the test data and perform the same test with different data.Generally where the flow remains same and objects does not change, Data driven framework can be very usefull.Testers who are not familiar with the scripting language can enter test data for these predefined scripts.Input data is given a pool of data set.Even the output data comes out from the data pool.It helps us to reduce coading for a large dataset.Ease of Testing of Time consumption.It is best used in IBM-Rational Robo.Data is not hard coded here,instead data are stored and provided from data source like -XML,Excel,flat file,Database or combination of all.As data layer is separated from code , same scripts can be reused multiple time.

find disadvantages here

Keyword Driven Automation Framework
Steps and data are feed to the automation tool.Automation tool judges the steps and input data from a data pool,It gave the output data.In a keyword-driven approach, the spreadsheet contains keywords describing the actions to be taken (also called action words), and test data. Testers (even if they are not familiar with the
scripting language) can then define tests using the keywords, which can be tailored to the application being tested.It is very sensative towords data design.This is based on generic key word based.
The advantage is the setting of keywords can test window/web/oracle etc based application.
More details can be found here
Modular Automation Framework
In this approach,The whole scenario is divided into small business flows-called module.Those modules are written in a library file as a functions.
In the script level the high level business functions are called.They are linked with small module functions.more over this is a very simple approach over record and play back.Objective to make it modular is nothing but to make the script reusable.There are driver in the main script and modular functions are stubs.We create shared object repository for Modular Automation Testing.When Application is huge, Initial development may go high but later the development cost just go down. Maintenance is easy and straight forward.

For more details click here and here.
Problems with this approach can be found here.

Hybrid Automation Framework
This is a combination of all frameworks .They are used whenever necessary.So is a summation of Data driven--keyword driven--Module approach.The main advantage is this is logic driven.Keywords are logic based.
For more Click here

We generally carried by the buzz words in the name of the framework. So migrating to a good framework may not be necessary for your organisation if the  current one is doing just great. So better not to reinvent the wheel or don't change it until it is broken.

few Guidelines while designing the framework:
1.DO version control(use CVS,SVN).
2. Try make the script Environment neutral.
Read More here for speed up automation with file size. Even Object Repository can be a nice part of Automation.How? Click here to learn more
Hoe to improve the framework? See here and here
Image credit:http://www.noesissolutions.com/

What is an Automation Framework All About?? What is an Automation Framework All About?? Reviewed by Animesh Chatterjee on April 15, 2009 Rating: 5
Powered by Blogger.