Selecting test automation tools – criteria and methods

The first article presented the general challenges and influencing factors that arise when selecting test automation tools, based on the results of various interviews. It showed that no standard method exists for selecting test automation tools, although there are some initial approaches involving checklists. The purpose of the thesis was therefore to find a simple, productive approach that would support the search for appropriate test automation tools on the basis of a criteria catalogue, taking account of the project application.

The basic requirement in the development of a criteria catalogue is to determine which criteria are actually relevant in the selection of a tool. I will look at this in the second part of the blog series.

Relevance of the selection criteria

There is extensive discussion in the literature on suitable criteria for determining the quality of software products and, by extension, of test automation tools. The criteria identified for the criteria catalogue were largely developed on the basis of literature research. The sources used are presented briefly below:

The ISO 25010 list of software quality characteristics provides a useful checklist when deciding whether or not to test each criterion. Similar lists of quality characteristics can be found elsewhere (Spillner/Linz 2019, p. 27). In each case, the authors provide guidance that can be used to determine whether the individual quality characteristics have been fulfilled for a project. In the field of test automation, there are lists of criteria in Baumgartner et al. 2021, Lemke/Röttger 2021, Knott 2016 and others. These criteria relate to such factors as the supported technologies, options for test case description and modularisation, target group, integration into the tool landscape and costs. However, the objective here is simply to identify relevant criteria, not to evaluate them. There are additional criteria from the expert interviews and the analysis of the ZDI projects.

The selection criteria identified in this work therefore take findings and experience from existing papers on the subjects of quality, test automation and the creation and review of requirements for test automation tools, and combine them into one approach with practical relevance.

Classification of criteria

In the selection of test automation tools, influencing factors such as experience, costs or interest groups were mentioned. For many, costs were the key factor. During the course of this work, it was found that criteria such as integration or compliance are defined differently depending on the role, but are included more or less as standard in the list of criteria. And they are unchangeable, regardless of the application undergoing testing. But there is a small proportion of criteria that vary depending on the application being tested. Here is a scenario to illustrate the problem: In the medical technology industry, a new, web-based application is being developed – the frontend with Angular 8 and NodeJS and the backend with Java Microservices. The test automation tool to be selected must primarily be appropriate for the framework conditions specified by the web application undergoing testing. Before an automation tool can be used, the interface technologies of the application must be examined. In practice, test automation tools have specific characteristics and are therefore used for different technologies. Some tend to specialise in web testing while others are more at home in desktop testing. Whether it’s a web application or mobile application, there are always certain expectations that apply to the test automation tool. This means the most critical factor is the test object or the system under test (SuT). This forms the basis for selecting the tool (Baumgartner et al. 2021, p. 45). In summary, the criteria can be classified in two types: the standard proportion and the variable proportion.

The standard criteria are not dependent on the test object. These criteria have been grouped into four categories based on the quality criteria: features, system, usability and provider-related criteria. By contrast, the variable criteria are dependent on the SuT. The variable criteria may include the supported application types, operating systems, interface technologies, browser and devices.

Variable selection strategy

Variable selection means selecting a number of variables to be included in the list of criteria. To support the selection of variable criteria, a target model with AND/OR decomposition based on GORE (Goal Oriented Requirements Engineering) and the work of Lapouchnian (2005) and Liaskos et al. (2010) was introduced during the course of my work. It had proven to be effective at recording exact alternatives or variabilities (Mylopoulos/Yu 1994, p. 159-168) so that alternative designs can be evaluated during the analysis process (Mylopoulos et al. 2001, p. 92-96). The targets and requirements are linked via AND/OR decomposition. The AND decomposition expresses that fulfilment of the relevant targets or requirements is required. An OR-link means that the fulfilment of one of these targets or requirements is sufficient. In this process, the initial expectations for the tool are formulated in explicit terms and irrelevant requirements are avoided.

Approach to tool selection

Based on the types of criteria identified (Spillner et al. 2011), this work designs a structured approach to selecting the appropriate test automation tool. The new approach can be divided into five stages:

1. Develop the project requirements
2. Identify the variable criteria using the AND/OR target model
3. Weight the categories and criteria
4. Evaluate the different tool solutions
5. Evaluation and selection

The next blog article looks at how the criteria catalogue has been structured on the basis of the approach outlined above. We will also show how the variable criteria have been included in the catalogue and present the results of the validation of the criteria catalogue.

This article was technically supported by:

This post was written by: