Software systems are becoming increasingly complex due to the constantly growing number of applications on different platforms. A decisive factor for the success of a software product is its quality. For this reason, more and more companies are carrying out systematic checks and tests, if possible, on the various platforms, in order to be able to ensure a given quality standard. In order to be able to keep short release cycles despite the higher testing effort, it becomes necessary to automate the tests. This in turn leads to the need to define a test automation strategy. One of the first steps in introducing a test automation strategy is to evaluate suitable test automation tools. Since each project is unique, both the requirements and the choice of tools vary. This blog series is intended to provide guidance in selecting the appropriate solution.
Introduction
As part of my final thesis, I took on the task of providing the Scrum teams at ZEISS Digital Innovation (ZDI) with an aid to help them find the right test automation tool quickly and flexibly. The challenge here was that the projects have specific scenarios and the requirements may have to be weighted separately. I would like to present the status of the work and the results to you in this and the next blog articles.
Software development has long been an area of rapid change. But while in the past these developments have mainly taken place at the technological level, we are currently observing major changes in the area of processes, organisation and tools in software development. However, these new trends come with challenges such as changes in requirements management, shortened timelines and especially the increased demands on quality. Today, the development of needs-based solutions and the optimisation of quality increase both efficiency and effectiveness within software development.
In addition, software systems are becoming more and more complex due to the constantly growing number of applications on different platforms. Since quality is a decisive factor for the success of a software product, more and more companies are carrying out systematic checks and tests, if possible, on the various platforms, in order to ensure a given quality standard. A SmartBear survey conducted in early 2021 found that many companies, regardless of the sector, already perform different types of testing, with web application testing leading the way at 69% and API/web services testing second at 61%. Desktop testing is done by 42% of the respondents. A total of 62 % of the respondents state that they perform mobile testing, 29 % of which for native applications (apps) (SmartBear, 2021, p. 12). In order to be able to maintain short release cycles despite the higher testing effort, it is becoming necessary to automate the tests.
As ZDI, we support our customers inside and outside the ZEISS Group in their digitisation projects. We also offer individual test services. That is why we have a large number of projects with different technologies and different solutions. As a small selection, keywords such as Java, .Net, JavaFX, WPF, Qt, Cloud, Angular, Embedded etc. should be mentioned here. In such complex projects, quality assurance is always in the foreground and the projects are dependent on the use of modern test tools that support the project participants in the manual, but especially in the automated tests. It would be desirable to have a tool that supports this automation effectively and efficiently. However, testers face a variety of challenges when selecting a test automation tool.
Challenges
In the research and interviews conducted as part of my work, the greatest challenges in test automation were named as the variety of test tools available. Due to this fragmentation a variety of tools is available for the same purpose.
The choice becomes even more difficult because the tools differ not only in the technology they can test, but also in their work approach. When automation is mentioned, it is always associated with scripting. In recent years, however, a new approach to GUI testing has been developed called NoCode/LowCode. This approach basically requires no programming knowledge and is therefore popular with automation beginners. Nevertheless, scripting remains the dominant method, although sometimes both approaches are used to involve as many of the quality assurance team as possible (SmartBear, 2021, p. 33).
The type of test automation approach, and therefore the use of a test automation tool, depends on the requirements in the project. This means that the requirements must always be re-examined for each new project.
Inventory
In the course of the interviews, the analysis of the current approach and the evaluation of an internal survey, I was able to identify the following procedures for the selection of tools in the projects, which have become established as a “quasi-standard”:
- Das The tool is Open Source and costs nothing.
- The tool has been used before and is considered trustworthy.
One aim of the survey was to find out to what extent the project situation has an influence on the tool selection. Therefore, the next step was to examine the project situations and the process models used in the projects and their influence. It turned out that it is not the process models that have a direct influence on the tool selection, but the people or groups who use the tool as future users or who approve its use.
When examining the influence of these operationally-involved participants, it became apparent that there are other interest groups that have a direct or indirect influence on the selection of a tool. These are, for example:
- Software architects, who define the technology or tool chains within the project,
- The management, which sets guidelines for the purchase or use of tools (OpenSource, GreenIT, strategic partnerships, etc.) or
- The company’s IT, which prevents the use of certain tools through the infrastructure used (firewall, cloud, etc.).
Furthermore, the first approaches of checklists for the selection of test tools were already found during the research. They were mostly based on a few functional criteria and were determined in workshops by the project members. The evaluation showed that there was no uniform method and that the tools selected in this way were very often later replaced by other tools. This became necessary because essential requirements for the tool were overlooked during the selection.
In the majority of cases, the forgotten requirements were of a non-functional nature, such as usability criteria or performance requirements. Therefore, a next step in checking relevant criteria was to refer to ISO/IEC 25000 Software Engineering (Quality criteria and evaluation of software products.
The next blog article in this series will show how the criteria catalogue is structured and how the final list of criteria is composed.
Literature
SmartBear (2021): State of Software Quality | Testing.
This article was technically supported by:
Kay Grebenstein
After graduating in computer sciences, Kay Grebenstein was drifting over to “The Dark Side” of software development and he is now working as a software tester and agile QA coach at ZEISS Digital Innovation, Dresden. Over the last few years, he was responsible for quality assurance and software testing in many projects in different business domains (telecommunications, industry, commerce, energy sector, …). He has experience with both the classical as well as the agile process models.
This post was written by:
