Implementation of Machine Learning techniques to overcome the challenges of performance testing



Acknowledgement:
I thank Christy Jacob and Nikhil Guddati for their technical assistance with ML algorithms and performance test data used in this white paper.

Abstract
While Machine Learning has been popular in various domains like social media, product recommendations, etc., this white paper will focus on the implementation of Machine Learning algorithms in the performance testing domain.

Overview: Supervised Learning is a machine learning task in which a function is learned, or a relationship is established between a given set of input features and an output feature using the training dataset. And then this learned function is further used to predict outputs for a new set of inputs from the testing dataset.

Regression is a type of Supervised learning task used to predict the target value based on the relationship between variables obtained from the dataset.


The test environment is one of the critical challenges of performance testing.

Ideally, the performance testing environment should be sized with the same capacity as that of production, to avoid any risks associated with interpreting system performance characteristics for the production environment. However, dedicated environments are not typically available due to budget issues, and certain industries forgo executing tests in a production environment for security reasons. As a result, testers end up with scaled-down environments for performance testing. These tests come with added risks.

We have used Machine Learning to address the above-stated challenges. The outcome of these experiments has been documented in this whitepaper.


To implement Machine Learning for performance testing, we will discuss the stages involved to create and evaluate an ML model:

• Defining the parameters
• Generating training data
• Choosing an ML model
• Evaluating the model

Defining the Parameters
To begin, we must identify the inputs and define the expected outputs. Response Time is a critical parameter that needs to be closely monitored during performance testing. So, in this example, we will predict the Response Time of a page/request by feeding Total Samples, Sent Bytes, Received Bytes, and Request Name as inputs.

Output (Label) a Response Time

Input (Feature) a Threads, Sent Bytes, Received Bytes, Request/Page

Generating Training Data
The first and most essential step is to collect relevant data corresponding to our problem statement. We generated Input data by executing performance tests using JMeter in a controlled test environment (to eliminate noise due to server performance or additional network traffic).

Performance reports generated from these tests contain the following data: Sent Bytes, Bytes, Threads, Request Name, and Response Time.

The performance test documented below generated 490 records for each page request.



Choosing an ML Model
The process of training an ML model involves assigning an ML algorithm (the learning algorithm) to learn from the training data. ML model is the outcome created by the training process.

We installed the Machine Learning library and uploaded our performance test data in it. The library evaluated various algorithms with varying computations and settings to identify the best - performing model.

For our test data, the following algorithms were evaluated. The FastTreeRegression algorithm returned the highest accuracy at 82.3%.



Evaluating the Model
The model evaluation aims to estimate the generalization accuracy of a model based on future (unseen/out-of-sample) data. Now, that we had an ML Model, we used it to predict the response time for a new set of data (which was not used for training the model). The results of ML Model were tested against actual data, and the deviation was within acceptable limits.




The deviation from actual values is within milliseconds for the predictions made by the algorithm. Therefore, it successfully evaluated the performance of the application without having to create a replica of the production environment.

Each application’s behavior varies under load, so, generating an optimal model depends on data collected with performance tests. Based on the behavior of the application, we need to estimate which part of the data will provide the most accurate results. That is why it is essential to generate a series of test data for better prediction.

References

https://www.qualitestgroup.com/white-papers/lets-perform-right-performance-testing-planning-overcoming-challenges/
https://www.agileconnection.com/presentation/machine-learning-and-data-science-quality-and-performance-engineering
https://www.datasciencecentral.com/profiles/blogs/performance-testing-training
https://data-flair.training/blogs/machine-learning-applications/