{"id":1906,"date":"2022-07-28T08:43:18","date_gmt":"2022-07-28T08:43:18","guid":{"rendered":"https:\/\/blogs.zeiss.com\/digital-innovation\/en\/?p=1906"},"modified":"2022-08-05T14:14:09","modified_gmt":"2022-08-05T14:14:09","slug":"criteria-catalogue-for-the-best-possible-choice-of-test-automation-tools","status":"publish","type":"post","link":"https:\/\/blogs.zeiss.com\/digital-innovation\/en\/criteria-catalogue-for-the-best-possible-choice-of-test-automation-tools\/","title":{"rendered":"Criteria catalogue for the best possible choice of test automation tools"},"content":{"rendered":"\n

In the first part of this blog series<\/a>, we outlined the challenges that arise when selecting the right tool for test automation. The second part<\/a> was about the relevance and possible classification of tool selection criteria, some of which can be standardised, but some of which need to be variable. In this third article we will look at the list of criteria, the criteria catalogue, how it was validated and the results of this validation.<\/p>\n\n\n\n

\"Symbolbild:
Designed by Freepik<\/a><\/sup><\/em><\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

List of criteria<\/strong><\/h2>\n\n\n\n

The following figure shows the final list of selection criteria. The variable criteria<\/em> have been marked as \u201cSuT criteria\u201d (software under test). The criteria identified can be modified, supplemented and updated.<\/p>\n\n\n\n

\"table
Figure 1: <\/em>List of criteria<\/figcaption><\/figure>\n\n\n\n

The standard criteria<\/em> are divided into fourteen (14) main criteria, including compatibility, operability, performance and documentation. Sub-criteria are assigned to the main criteria. For example, the main criterion \u201cdocumentation\u201d has three (3) sub-criteria.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Criteria catalogue<\/strong><\/h2>\n\n\n\n

Once the criteria had been established, the next step was to create the actual criteria catalogue. Since there are different, multi-dimensional goals involved in making decisions, it is advisable to take a systematic approach. This requires multi-criteria decision analysis (MCDA). Utility value analysis is one method of multi-criteria decision analysis (Gansen 2020, p. 5). Utility value analysis can be applied whenever an evaluation is being carried out or an assessment is being made, e.g. in project management or controlling.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"table
Figure 2: <\/em>Sample table showing a utility value analysis<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Requirements for the criteria catalogue<\/strong><\/h2>\n\n\n\n

Before actually compiling the criteria catalogue, we started by defining its requirements. These are summarised in the table below in the form of user stories and outline all the work steps required.<\/p>\n\n\n\n

Nr.<\/th>User Stories<\/th><\/tr><\/thead>
1<\/td>As a user, I would like to be able to enter project master data in order to better understand by whom, when and for which project the catalogue was created.<\/td><\/tr>
2<\/td>As a user, I would like to compile a new catalogue in order to perform a utility value analysis.<\/td><\/tr>
3<\/td>As a user, I would like to perform a utility value analysis in order to make an objective decision.<\/td><\/tr>
4<\/td>As a user, I would like to apply fair and moderate weighting to the criteria in order to better control their relevance to the project.<\/td><\/tr>
5<\/td>As a user, I would like an overview of the basis for evaluation in order to better assess alternative solutions.<\/td><\/tr>
6<\/td>As a user, I would like a clear overview of the utility value analysis performed in order to quickly identify the most useful tool.<\/td><\/tr>
7<\/td>As a user, I would like to access to the most recently edited utility value analysis in order to make further edits.<\/td><\/tr>
8<\/td>As a user, I would like to be able to export the catalogue in order to share it.<\/td><\/tr><\/tbody><\/table>
Table 1: <\/em>Requirements for the criteria catalogue<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Structure of the criteria catalogue<\/strong><\/h2>\n\n\n\n

The criteria catalogue was developed using Excel as well as Visual Basic for Applications (VBA). The developed tool was divided into different workbooks, each reflecting a specific task.<\/p>\n\n\n\n

The starting screen<\/h3>\n\n\n\n

When you open the file, a dialogue window appears (see figure 3). You first have to choose whether to open the most recently edited catalogue or to create a new catalogue. If you want to create a new catalogue, a form is displayed which has to be filled in. SuT criteria entries are added to the catalogue as variable<\/em> criteria<\/em> (see figure 4).<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"starting
Figure 3: <\/em>Starting screen of the criteria catalogue<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n
\"screenshot<\/figure>\n\n\n\n

Figure 4: <\/em>SuT criteria entries in the catalogue<\/p>\n\n\n\n

<\/div>\n\n\n\n

Utility value analysis<\/strong><\/h3>\n\n\n\n

A utility value analysis is performed in four steps. Once the criteria have been identified, they are weighted. Then fulfilment of the criteria is measured and finally the utility value is calculated for each alternative (Gansen 2020, p. 6). Once the evaluation criteria have been properly assessed, an objective, rational decision can be made with the help of the utility value analysis (K\u00fchnapfel 2014, p. 4).<\/p>\n\n\n\n

<\/div>\n\n\n\n

Weighting the criteria<\/strong><\/strong><\/h4>\n\n\n\n

Once the criteria, especially the variable criteria<\/em>, have been established, these criteria need to be weighted. It is crucial to weight the criteria according to their importance for the specific test automation project in order for them to make the best possible contribution towards the achievement of the project\u2019s goals. The sum of the weightings for standard criteria<\/em> and variable criteria<\/em> should always add up to 100%. For the standard criteria<\/em>, the main criteria are weighted first using the pairwise comparison method in the form of a matrix and are compared with each other in pairs (Wartzack 2021, p. 315).<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"comparison<\/figure>\n\n\n\n

Figure 5: Pairwise comparison of main criteria for a sample project<\/em><\/em><\/p>\n\n\n\n

<\/div>\n\n\n\n

Main criteria: security, installation effort, language support<\/p>\n\n\n\n

The importance of each sub-criterion is then determined using an ordinal scale from zero to two:<\/p>\n\n\n\n

0 = “unwichtig”; 1 = “teilweise wichtig”; 2 = “wichtig”<\/em><\/p>\n\n\n\n

The points awarded are added up and multiplied by the corresponding weighting of the main criterion. This produces a proportional weighting for all standard criteria<\/em>. The same procedure is applied to the other main criteria and sub-criteria. In the end, the sum of all weightings for the standard criteria<\/em> equals 100%.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Measuring the fulfilment of criteria<\/strong><\/h4>\n\n\n\n

The starting point is to answer the following question: \u201cTo what extent is the criterion fulfilled by the test automation tools being assessed?\u201d. A 5-step model is used to determine the degree of criteria fulfilment, as shown in figure 7 (Gansen 2020, p. 17).<\/p>\n\n\n\n

0<\/td>Not fulfilled.<\/td><\/tr>
1<\/td>Insufficiently fulfilled.<\/td><\/tr>
2<\/td>Partially fulfilled.<\/td><\/tr>
3<\/td>Fulfilled to a satisfactory extent.<\/td><\/tr>
4<\/td>Well \/ completely fulfilled.<\/td><\/tr><\/tbody><\/table>
Table 2: Evaluation scheme for criteria fulfilment<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

If a score of 4 is given, the tool completely fulfils a criterion. If it does not fulfil the criterion, a score of 0 is given. This method converts the relevant properties of a test automation tool into a measurable quantity.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Evaluation consolidation<\/strong><\/strong><\/h4>\n\n\n\n

Once the degree of fulfilment has been established for each criterion, the utility value can be calculated. The following formula is used to determine the utility value:<\/p>\n\n\n\n

\n
\n
\"\"<\/figure>
\n

N i<\/sub> = utility values of the alternative i<\/p>\n\n\n\n

Gj = weighting of the criterion j<\/p>\n\n\n\n

nij = partial value of the alternative i with respect to the criterion j<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n\n\n

The partial values are added up. The result represents the actual value of a tool. The alternatives being considered can then be ranked according to the calculated utility values and this can subsequently be used for decision-making (Gansen 2020, p. 6). The tool solution that best meets all requirements and has the highest utility value can be selected.<\/p>\n\n\n\n

Certain criteria can be identified as essential. These are called knock-out criteria (K.O. criteria). If a knock-out criterion (K.O. criterion) is not fulfilled (0)<\/em>, this is deemed a \u201cbreach\u201d, which leads to the immediate exclusion of the alternative solution.<\/p>\n\n\n\n

\"screenshot
Figure 6: <\/em>Evaluation and selection<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Navigation bar and export<\/strong><\/h4>\n\n\n\n

The navigation bar provides the user with an overview of the catalogue\u2019s structure and allows them to navigate through it effortlessly.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"screenshot
Figure 7: Navigation bar of the criteria catalogue<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

The catalogue can be exported as a PDF or Excel file and saved in any location.<\/p>\n\n\n\n

<\/div>\n\n\n\n
\"screenshot
Figure 8: <\/em>Excerpt of the blank template for the criteria catalogue<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Results of the criteria catalogue validation<\/strong><\/h2>\n\n\n\n

The following insights were gained from the validation:<\/p>\n\n\n\n