Criteria catalogue for the best possible choice of test automation tools

In the first part of this blog series, we outlined the challenges that arise when selecting the right tool for test automation. The second part was about the relevance and possible classification of tool selection criteria, some of which can be standardised, but some of which need to be variable. In this third article we will look at the list of criteria, the criteria catalogue, how it was validated and the results of this validation.

Symbolbild: aufgeschlagenes Buch
Designed by Freepik

List of criteria

The following figure shows the final list of selection criteria. The variable criteria have been marked as “SuT criteria” (software under test). The criteria identified can be modified, supplemented and updated.

table showing list of criteria
Figure 1: List of criteria

The standard criteria are divided into fourteen (14) main criteria, including compatibility, operability, performance and documentation. Sub-criteria are assigned to the main criteria. For example, the main criterion “documentation” has three (3) sub-criteria.

Criteria catalogue

Once the criteria had been established, the next step was to create the actual criteria catalogue. Since there are different, multi-dimensional goals involved in making decisions, it is advisable to take a systematic approach. This requires multi-criteria decision analysis (MCDA). Utility value analysis is one method of multi-criteria decision analysis (Gansen 2020, p. 5). Utility value analysis can be applied whenever an evaluation is being carried out or an assessment is being made, e.g. in project management or controlling.

table showing sample for utility value analysis
Figure 2: Sample table showing a utility value analysis

Requirements for the criteria catalogue

Before actually compiling the criteria catalogue, we started by defining its requirements. These are summarised in the table below in the form of user stories and outline all the work steps required.

Nr.User Stories
1As a user, I would like to be able to enter project master data in order to better understand by whom, when and for which project the catalogue was created.
2As a user, I would like to compile a new catalogue in order to perform a utility value analysis.
3As a user, I would like to perform a utility value analysis in order to make an objective decision.
4As a user, I would like to apply fair and moderate weighting to the criteria in order to better control their relevance to the project.
5As a user, I would like an overview of the basis for evaluation in order to better assess alternative solutions.
6As a user, I would like a clear overview of the utility value analysis performed in order to quickly identify the most useful tool.
7As a user, I would like to access to the most recently edited utility value analysis in order to make further edits.
8As a user, I would like to be able to export the catalogue in order to share it.
Table 1: Requirements for the criteria catalogue

Structure of the criteria catalogue

The criteria catalogue was developed using Excel as well as Visual Basic for Applications (VBA). The developed tool was divided into different workbooks, each reflecting a specific task.

The starting screen

When you open the file, a dialogue window appears (see figure 3). You first have to choose whether to open the most recently edited catalogue or to create a new catalogue. If you want to create a new catalogue, a form is displayed which has to be filled in. SuT criteria entries are added to the catalogue as variable criteria (see figure 4).

starting screen
Figure 3: Starting screen of the criteria catalogue
screenshot SuT criteria

Figure 4: SuT criteria entries in the catalogue

Utility value analysis

A utility value analysis is performed in four steps. Once the criteria have been identified, they are weighted. Then fulfilment of the criteria is measured and finally the utility value is calculated for each alternative (Gansen 2020, p. 6). Once the evaluation criteria have been properly assessed, an objective, rational decision can be made with the help of the utility value analysis (Kühnapfel 2014, p. 4).

Weighting the criteria

Once the criteria, especially the variable criteria, have been established, these criteria need to be weighted. It is crucial to weight the criteria according to their importance for the specific test automation project in order for them to make the best possible contribution towards the achievement of the project’s goals. The sum of the weightings for standard criteria and variable criteria should always add up to 100%. For the standard criteria, the main criteria are weighted first using the pairwise comparison method in the form of a matrix and are compared with each other in pairs (Wartzack 2021, p. 315).

comparison of main criteria for a sample project

Figure 5: Pairwise comparison of main criteria for a sample project

Main criteria: security, installation effort, language support

The importance of each sub-criterion is then determined using an ordinal scale from zero to two:

0 = “unwichtig”; 1 = “teilweise wichtig”; 2 = “wichtig”

The points awarded are added up and multiplied by the corresponding weighting of the main criterion. This produces a proportional weighting for all standard criteria. The same procedure is applied to the other main criteria and sub-criteria. In the end, the sum of all weightings for the standard criteria equals 100%.

Measuring the fulfilment of criteria

The starting point is to answer the following question: “To what extent is the criterion fulfilled by the test automation tools being assessed?”. A 5-step model is used to determine the degree of criteria fulfilment, as shown in figure 7 (Gansen 2020, p. 17).

0Not fulfilled.
1Insufficiently fulfilled.
2Partially fulfilled.
3Fulfilled to a satisfactory extent.
4Well / completely fulfilled.
Table 2: Evaluation scheme for criteria fulfilment

If a score of 4 is given, the tool completely fulfils a criterion. If it does not fulfil the criterion, a score of 0 is given. This method converts the relevant properties of a test automation tool into a measurable quantity.

Evaluation consolidation

Once the degree of fulfilment has been established for each criterion, the utility value can be calculated. The following formula is used to determine the utility value:

N i = utility values of the alternative i

Gj = weighting of the criterion j

nij = partial value of the alternative i with respect to the criterion j

The partial values are added up. The result represents the actual value of a tool. The alternatives being considered can then be ranked according to the calculated utility values and this can subsequently be used for decision-making (Gansen 2020, p. 6). The tool solution that best meets all requirements and has the highest utility value can be selected.

Certain criteria can be identified as essential. These are called knock-out criteria (K.O. criteria). If a knock-out criterion (K.O. criterion) is not fulfilled (0), this is deemed a “breach”, which leads to the immediate exclusion of the alternative solution.

screenshot evaluation selection
Figure 6: Evaluation and selection

Navigation bar and export

The navigation bar provides the user with an overview of the catalogue’s structure and allows them to navigate through it effortlessly.

screenshot navigation bar
Figure 7: Navigation bar of the criteria catalogue

The catalogue can be exported as a PDF or Excel file and saved in any location.

screenshot template criteria catalogue
Figure 8: Excerpt of the blank template for the criteria catalogue

Results of the criteria catalogue validation

The following insights were gained from the validation:

  • The target model for identifying variable criteria was useful for collecting together ideas for the SuT criteria.
  • The use of pairwise comparison helped to identify the important factors and main criteria. The matrix made it possible to make comparisons. This resulted in a significant reduction in “gut decisions”.
  • The criteria catalogue offered a systematic, understandable and transparent method for recommending a suitable tool. It was also possible to determine the strengths of the recommended test automation framework. This verified the framework’s coverage of selection criteria with higher relevance. This reduces the need for discussions within the team that would delay the final decision.
  • The 5-step model used for evaluation required very in-depth knowledge of test automation tools, which is usually lacking. This would result in staff evaluating some of the criteria based on their own feelings and perceptions. Consequently, the evaluations would be partially subjective and the selected alternative would ultimately not be the optimal one. In order to obtain a more reliable result in this scenario, at least one other person with testing expertise should be involved in the evaluation.

Conclusion & outlook

In this work, we have examined a structured and systematic approach to the selection process. This can be used to select a suitable GUI test automation tool that meets the requirements of both the project and the company. A corresponding criteria catalogue was developed, which primarily provides transparency and comprehensibility in the decision-making process.

We intend to use the criteria catalogue in other projects. The experience gained during this process will be incorporated into the next version of the criteria catalogue (version 2.0).

This article was technically supported by:

Kay Grebenstein

Kay Grebenstein works as a tester and agile QA coach for ZEISS Digital Innovation, Dresden. Over the last few years, he has ensured quality and tested software in projects in various specialist domains (telecommunications, industry, mail order, energy, …). He shares his experiences at conferences, meetups and in publications of various kinds.

Alle Beiträge des Autors anzeigen

This post was written by: