Adding AI capabilities to a Test Automation Framework

The following blog documents the experiences of Celsior in using auto-healing and AI in order to automatically maintain test automation scripts.  Although the process of integrating AI has not been smooth, the learnings from the journey and the results have been excellent.

Test Automation is used as a standard practice in Quality Engineering to keep pace with fast and frequent application changes. Due to frequent application changes, however, test automation scripts tend to fail. This slows down the development process because of the maintenance time required for triaging and fixing the issues. A major cause of the failures in the existing scripts is the modification by the developers of object properties (identifiers). When object properties change, automation testers need to manually update the object locators in the object repository to make sure that the scripts run without any issues. To minimize manual efforts, the auto-healing concept is used.  This identifies the object changes and fixes them automatically without human intervention. 

Auto-healing in test automation addresses the leading cause of test script failure i.e., object changes. It detects changes in the object properties and automatically searches and updates them with new attributes to ensure the tests do not break. It offers a host of benefits for tester and team by taking less time and effort to ensure all tests run smoothly and avoid interruption during execution. Auto-healing is applied to test automation to dynamically adapt their testing scope to the changes. With such features, test scripts adapt and make testing a lot easier and more efficient. This concept is logic based, however, and requiring QE to implement the logic in the framework is a bit complex. Another way of implementing auto-healing in the test automation framework is to leverage Artificial Intelligence. AI can be used in different ways to make the framework more robust in case the properties of the objects change.  

At Celsior, we started by implementing the code-based auto-healing framework.  This worked well and gave us the desired results. The only challenge with this approach was that it was not expandable–so we started exploring other options to make the auto-healing framework robust and reliable. To make auto-healing scalable and accurate, we investigated use of AI with the test automation framework. Based on research, we came across a couple of ways we could leverage AI within the test automation framework for making automation scripts auto-heal. One way to implement AI is to use object attributes (DOM properties); another way is to use object images; and a final way is a combination of both object attributes and images for the auto-healing mechanism.  

It soon became apparent that the journey from code-based auto-healing to an AI-based auto-healing approach was not going to be easy. So, we decided to take small steps as we were not sure whether the approach we identified was going to work or not. We started with developing a proof of concept (POC) using the object attributes (DOM properties). First step in the POC was to develop a means to integrate the AI model easily with the test automation framework. The second step was to collect the data which was needed to train the AI model. After brainstorming and research, we decided to collect data from applications in different domains to train our model. So, using various domains, we began to collect the object properties of elements from applications and saved it in Excel format. Since data is critical for AI/ML models, it is very important to process the data and use it to drive the AI engine learning.  

The data we collected was huge and required processing different type of elements with different attributes. We thought to divide the data based on the element type. After segregating the data, we labeled the data and started to generate the algorithms. We shortlisted a couple of algorithms which we could use. We then segregated the data for training and testing purposes and started training the model. After training, we tested the model and to our surprise, the accuracy of the model was very low. So, we changed the data variation from segregating the data based on element types to segregating based on the attributes.  We also removed the attributes that had no values for most elements, so that we wouldn’t lose important data for tags. Then we trained the model again and after testing the model, confidence scoring, and precision improved.  It was still not at the level which we wanted it to be. 

To further improve accuracy, we tuned the model using various hyperparameter tuning techniques like grid search and random search. We also used various methods to take only the important features into consideration and to remove features which were not helpful. Doing so we could achieve a 50 to 60 percent accuracy. To complete the POC and verify whether we could integrate the AI model with our test automation framework, we opted to go with this model after gradually improving it. To integrate the model with our test automation framework we deployed it on the server and implemented API integration to communicate with the model. We added API calls in the framework to call the AI engine whenever during test execution an object was not identified in the traditional way. The AI engine then returned the closed match of the object and subsequently performed the operation on the newly found object. Since performing the operation solves just one part of the issue, we implemented logic in the AI engine to return the changed or latest property of the object so that it could then be updated in the object repository. This helped in saving significant maintenance effort for future test executions. 

We achieved around 50 to 60 percent accuracy with the DOM approach which is good in reducing maintenance efforts, however we still wanted to take this accuracy to around 90 percent. So, we started work on the alternative, image-based approach. With this approach, we use images of the object to find them on the web page. Also, we add variations like zoom in, zoom out, contrast change, color filters, shearing, translation, horizontal flip, etc. The model was trained with these variations and then integrated with our test automation framework. We are now able to achieve better accuracy with this approach compared to the DOM properties-based approach. For efficient object management, we have created an Object Manager utility to capture object images and their properties from the application under test. This utility crops the object image and passes it on to the AI engine for training, retraining, and feedback. 

To make the AI-powered automation framework even more robust, we are planning to integrate the DOM-based and image-based AI engines. This will give us more flexibility and accuracy in auto-healing test scripts and will also help us in providing technology support to a wide variety of applications and frameworks. 

ABOUT THE AUTHORS
Vikas Shukla
Director – Quality Engineering
Celsior  

Himanshu Gosain
Senior QE Architect
Celsior

Read other blogs of the series to get more insights on Quality Engineering, QE services, and QE Automation:

Moving from Quality Assurance to Quality Engineering
Test Automation – Getting the most out of open-source
Mobile Test Automation – How to get the most bang for your buck
ETL Data Validation – Better decision making through improved data quality
Role of Quality Engineering within DevOps and CI/CD