QTP - Hybrid Automation Frame Work - Bharath

What is Automation Framework?

Framework is a wrapper around complex internal architecture which makes end user to interact with the system easily. It also define guidelines and set standards for all phases of ADLC (Automation Development Life Cycle).

This “Dual Function” (Hybrid) framework is a combination of three frameworks (Functional Decomposition, Data Driven, Keyword Driven) and (re-usability, advanced error logic, custom QTP methods, dual function) techniques.


Key features of Dual function Framework

1. Re-usability, low maintenance design (Dual Function).
2. Support different application environments (Development, Quality Control, production) and custom client settings.
3. Externally Configurable - Execute all the test cases, execute only failed test cases, execute test cases by test case id, execute test cases by using flags...can be configured in different ways.
4. Self Configurable and can run unattended with out results being overwritten.
5. Better ROI (Return Of Investment).
6. Real time status monitoring…display detail status during the test execution and automatically close the pop-up status in x sec.
7. Test status reporting in three ways…detailed, high level and summary log file.
8. System automatically send email notifications with test results once the test is complete.
9. Screen Capture on failed test step, notified in the test case itself, assign unique name and stored in the separate folder for easy accessing.
10. Automation test cases resemble manual test cases (Just replace with keywords and test data)
11. Easy way of creating test cases with test case generators (Select keywords from drop down).
12. Easy to maintain and develop scripts (DP, Custom Methods and Dual Function).
13. Test execution time stamp for each step, so that we can easily trace when the test was executed.
14. Same script can be executed on QTP 9.2, 9.5 and 10.0. Backward, forward compatibility.
15. If the test is running in different location, system automatically identify the place of execution using System IP. If the IP is unknown not in the defined list or changed, it would publish the IP address in the results.
16. If object is missing or object properties got changed on the application, system will notify object is missing and proceed with the next test case. This will not stop the test execution.
17. Calculate each page response time automatically.
18. Framework is designed in such a way that it can be initiated using AOM script or directly run using QTP GUI especially useful during debugging or script creation.
19. It would automatically stop automatically if specific number of test cases got failed in sequence, this will help us know there is serious problem in the application.
20. Multi language support.
21 . Sending test results as SMS.
22. It would automatically add new library files.
23. As the entire code reside in library files, total size of the framework is very small. This will help us to easily copy, backup and upload into version control tools.
24. Automatic results archiving facility, old results will not erased.


Advantages of Dual Function Framework and ROI (Return Of Investment)

1. Reduce testing time.
2. Improve testing productivity.
3. Improve product quality
4. Reduce QA costs.
5. Consistent test results.
6. Calculate page response time of all the pages.
7. Schedule test runs.



Frame Work Folder Structure

Framework contain different files associated with different functionality, all the files to be placed in the corresponding folders; so that it is easy to maintain.

Note: Framework can't identify the folders if folder names get renamed, except sub folders created under "AutomationTestDocuments".

AutomationTestDocuments folder contain sub folders relating to each test. If we have two tests, there will be two sub folders. Create new folders as per the test requirement. There is no restriction on the folder name.

Each test (ProductionLoginTesting from the above screen shot) contain 2 Files (Startup, Control) and 3 sub-folders (AutomationTestCases - contain test cases; ExecuationLog - system automatically create two new files once test is complete, based on the configuration, Test summary.txt and Response time.xls; ScreenShots - contain application screen shots, if there are any errors. It would automatically assign unique name with time stamp for each screen shot and notified in the test step itself.

Config folder contain xls files containing custom client settings.

ExecutedScripts folder contain copy of the test folder after execution. If we run the same test again and again (schedule on daily basis) results are over written. In order to preserver the test results, framework will copy the test folder into newly created  folder with time stamp, once the test got completed.

Frame_WorkGep folder contain QTP code of one action "KickOff" with single line of code "Call StartUP". Entire project code code reside in the VBS file.
Following are the reasons for choosing this kind of design.
1. VBS file are very light in terms of size.
2. Framework is entirely based on DP (Descriptive programming).
3. We can easily implement any configuration management tool.

Library folder contain application libraries which contain code relating to keyword functions and contain sub folder called "CoreLibraries" that contain files corresponding to actual framework only.

Dual Function Framework Architecture 

Note: High-level design intended to show in a simple manner. Many other scripts are called between intermediate stages.

QTP AOM Script

1. vbs file created using AOM.

2. Attached to windows scheduler.

3. Identify the place of execution using system IP. If script is executed on new location, it would consider IP address as place of execution.  

4. As there are many tests under "AutomationTestdocuments", script need to be updated with the test folder name and path that we are planning to execute.

5. Script will open the QTP and associate the test specified in the step 4.

6. Two parameters are passed as environment variables (location and test path).

7. Copy the results in "ExecutedScripts" folder to protect results from overwriting, if initiated through AOM script only.

8. Test can also be executed in standalone mode without AOM script, especially useful during test creation and debugging. 

KickOff Action

This is the only QTP action used in the framework, it contain "Call funKickOff" to initiate KickOff script.

KickOff Script

Contain global object declaration, initialization and controlling test execution mode. This script will initiate StartUp script.

StartUp script

This script will configure the framework as per the settings provided in the startup spreadsheet.

Control Script

Start up script will initiate control script. Based on the startup spreadsheet settings, script will read each test case from the control spread sheet and call the driver script with test case id, test case name. This script will also generate real time status of the test in the form of pop-up message and update the test status.

Driver script

Based on the test case id and test case name, it will map the test case lower bound and upper bond, then it will read each line of the test case and call corresponding key word functions. 

How lower bound is computed ?

Based on the test case id received from the control script, it will start verifying the line having the keyword "TestCaseID", if ID under P1 (Column F) is matching, then it is treated as lower bound.

How upper bound is computed ?

Once script identify the lower bound, it will start searching the immediate "STOP" key word, this will be considered as upper bound.

Then it will start reading all the keywords between lower and upper bound by calling corresponding key word functions and updating each step.

Application library

These functions will actually manipulate(enter data or perform certain action on the control) the application as per the design and send the status back to the driver script.

To understand different variables that I have created in the framework, look into the function available under this link.

Note: Above scripts contain complex validation and error logic techniques, just gave you high level overview. Following top down approach and each script is independent entity, this will help us to maintain framework with ease and change the functionality without effecting other scripts.

Start-up spreadsheet

Row 1 - Default URL - If you are using the same test in different environments (Development, Quality control, production), just change the URL, scripts would run without any issues. This will help us to run the tests in different environments with very little change.

Row 2 - DB Name - If you are accessing the database through QTP, you can specify the name after creating DSN.

Row 3 - Test Execute Flag - Drop Down values All,Yes,No,Empty - Execute Flag,Check,Test Case Ids

These settings are applied by the control script on the control spreadsheet.

All - Execute all the test cases

Yes(Y) - Execute test case with Y flag

No(N) -   Execute test case with N flag

Empty - Execute Flag - Execute test case with empty flag

Check - When selected, it would just generate test report with consolidate pass/fail status (if test is run multiple time, initially all test cases, next failed test cases) without actual execution the test. 

Test Case Ids - Execute only specific test case ids.

Row 4 - Test Execution Status Flag -  Drop Down values All,Pass,Fail,Bad Inputs,Empty - Execution Flag,Check,Test Case Ids

These settings are applied by the control script on the control spreadsheet.

If we want to control the test based on Pass/Fail status use this control.

All - Execute all the test cases

Pass - Execute test case with Pass status

Fail -   Execute test case with Fail status 

Bad Inputs -   Execute test case with Bad Inputs status 

Empty - Execute Flag - Execute test case with empty flag

Check - When selected, it would just generate test report with consolidate pass/fail status (if test is run multiple time, initially all test cases, next failed test cases) without actual execution the test.

Test Case Ids - Execute specific test case ids.

Row 3 and 4 can be used in different combination while executing the test, so that desired test cases are executed. Most of the combination's are considered.  

Row 5 - Execute Specific Test Case Ids - Mention test case ids that you want to execute, "Test Case Ids" flag need to be selected in Row 3 or 4. This will override all the settings.

Row 6 - Tester Name - Name of the tester who is executing the script, same name appear in control and log files after test execution.

Row 7 - Release no/ Module name.

Row 8 - Test Cycle - Mention test cycle.

Row 9 - Exceptions exist counter - Test would automatically stop once the failed test case count reach the counter value. It doesn't make sense to keep on executing the test with 50 or more failed test cases, some thing gone wrong badly.

Row 10 - Test status pop-up display - Drop down values On,Off,Valid Test Case

On - Always display real time test status pop-up.

Off - Don't display real time test status pop-up.

Valid Test Case - Display while executing valid test case, don't display for invalid test case, determined from the combination of Row 3 and 4 flag settings.

Row 11 - Test status pop-up display (sec) - How may seconds real time test pop-up need to be displayed, automatically close after specified seconds.

Row 17 - Object Highlighting - On/Off

On - It will highlight all the objects during test execution, reduce test execution speed.

Off - No highlighting

Row 18 - Silent Mode - On/Off

On -Framework generate many messages during execution, when test is getting executed unattended, we can suppress these message by selection On.

Off - User need to click OK for each framework message, used during debugging.

Row 19 - Calculate average page response time - On/Off

On - It will generate an excel file with each page response time and store it in the "ExecuationLog" folder. Extra coding required, need to modify excel COM model function as per the requirements.

Off - It will not generate any page response file.

Row 26 - Beep on any click - On/Off

On - Generate beep sound when system select any control. 

Off - No beep.

Reason for implementing this feature - One of the developer told me that you are not calculating page response time correctly there is flaw in page synchronization code. In order to prove that I am correct, I have implemented this feature. When system select any action on the control it will generate a beep, once page sync is complete, it will generate different beep. After showing the same to the developer, he agreed that my page response time code is correct.

Control Spreadsheet

Headers highlighted in YELLOW - User need to fill the columns.

Headers highlighted in WHITE - System automatically fill the columns during execution.

One control spread sheet for entire test.

Column 1 - Test Case Id - Each test case is assigned unique id in ascending order only.

Column 2 - Test Case Description - Optional, small description about the test case.

Column 3 - Execute Flag(Y/N/Empty) - Control script will retrieve the settings from the start up spread sheet Row 3 and check the flag. If both flags match, system will execute the test case else it will ignore it. This system will help us to quickly configure which test cases to be executed.

Column 4 - Test case file name - Need to specify the corresponding test case file name.  Each test can have multiple test cases if required, so that functionality can be distributed and easy to maintain.

Test case - Group of test steps(procedure) with specific test data to validate a requirement.

Test case file - Group of test cases placed in a file.

Column 5 - Response Time - Consolidated response time of all the test steps under specific test case.

Column 6 - Execution Status - Once test case id is executed, system will update the status with Pass/Fail/Bad Inputs (test data provided to the test steps are incorrect).

We can also use this flag to control test execution. Row 4 start up spreadsheet.

Column 7 - Execution Place or System IP / Status Message - Display execution place/ Error and check point messages, if there are any inconsistencies.

Column 8 - Time stamp - Test execution time stamp in Local time.

Column 9 - Test tester name - As specified in the start up spreadsheet.

Column 10 - Release no/ module name - As specified in the start up spreadsheet.

Test case Spreadsheet

Before Execution

 

Headers highlighted in YELLOW - User need to fill the columns.

Headers highlighted in WHITE - System automatically fill the columns during execution.

Test case spreadsheet consist of test steps similar to manual test cases.

Column A - Keyword - Perform specific action on the screen like enter data, retrieve data, check point. These keywords are selected from the drop down, not required to type the keywords. Once test case creator select the keyword, based on the design, column F to Q are highlighted and corresponding cells are inserted with comments, these are input parameters P1, P2, P3....P20. "LogInSignon" (Row 118) contain 4 parameters, P1 - User name, P2 - Password, P3 - Credentials are correct, P4 - Language.

After Execution

System start filling columns B, C, D and F during test execution.

Mapping of manual and automation test cases to check the coverage

 

We can create new column "Automation test case ID" in the manual test cases spreadsheet, so that there will be one to one mapping without missing any functionality.

Test summary log file

Generated and placed in the 'ExecutionLog" folder one test is completed.

It contain following information.
1. Place of execution.
2. Test start and end time with time zone.
3. Total executes test cases (Pass/fail/bad inputs) and ignored test cases (based on start up spreadsheet settings).
4. Star up spreadsheet details.
5. Information relating to sending email notifications.
6. Average page response time (depend on the start up spreadsheet settings)
7. Available PC physical memory before and after execution with time stamp.
8. Test termination message By system completed/ By user/ By exception exist counter.
9. Whether email sent successfully.
10. PC information.

It is a summary of Test results + Framework settings + PC info.




Real-Time status POP-UP message

 

It provide live health of your test, this can be configured in the start up spread sheet.

Above image is self explanatory.

If your test is running for 2 hours and don't have this feature, you will understand nothing if you look at your running test after one hour. It would be better to implement this feature as per your requirements, not so exhaustively as mentioned above. 

Note: Implementing these features will not have any performance bottleneck in terms of execution speed or memory usage.

 Test case generator (Keyword sheet to create test cases)

Simple Sheet

 

Test case generator spreadsheet contain two sheets

1. KeyList - Where you design keywords, same as above screen shot

2. Keyword - Where you create automation test cases by selecting the keywords from the drop down.

Macros are written to connect both sheets

How to create KeyWords?

Lets create keyword for the login screen, where you have User name, Password and Login button.

Flow - Enter user name, Password and select login button.

Keyword name - LoginSignOn 

Input Parameters - User name, Password

Assign two parameters P1, P2 under column E and F.

Create cell comments for P1 and P2 as user name and password.

O- Optional parameter

M - Mandate parameter (System will generate "BAD INPUTS" if users don't enter values and execute the script.

Points to be considered while creating Keywords

1. Standard CamelCase format.

2. Keyword granularity (never have same keyword doing same or overlapping functionality).

3. When creating complex keywords choose following format (product name + page Name + functionality)

4. Don't worry about the length of keyword, you are not typing it, just selecting from the drop down. It should be self explanatory.

Advantages of this design.

1. It is not required to enter test object properties in the test case, already implemented in the code.

2. We can created a document which contain screen shot of the page and corresponding keywords. By looking at it any one can create test cases.

3. Generally in keyword driven framework, we can enter one value or fire one method per line; in this design you can enter maximum of 20 values in a single line. 

4. It is not required to memorize all the inputs associated with a keyword, Just select the Keywork, it would automatically highlight the parameters and assign cell comments to each parameter, so that you will understand easily what P1, P2...parameters for.

5. Each key word is self explanatory designed using camel case, any person can understand it.

Complex Sheet with lengthy keywords

After creating the core framework, next job would be creating keywords as per the functionality and executing the test.

Continued... QTP - Hybrid FrameWork - Part II

Source: Bharath Marrivada