The Test Analyst – Just a Reviewer for Automated Test Cases? A Field Report from an Agile Environment

Introduction

In an agile environment, regression testing helps to maintain a high level of quality. With each user story, newly developed functions are added, while the old functions still have to work. By the tenth sprint, the effort involved in the regression is so high that it is impossible to test everything manually. The only solution is test automation.

If a project is built from scratch, it is possible to properly integrate the test automation from the start. At the same time, the tester often feels like a lone warrior facing several developers. Now then, how can we realize the time-consuming automation of the functions in the daily work of a test analyst?

Project environment

In our project, we are creating new software in a JavaScript environment. It is implemented by means of the Electron framework. Consequently, Spectron is the preferred tool for the automation of test cases. Jira is used for the project platform, and the project is implemented based on the scrum model. The project team (based on FTE) consists of:

• the test automation is done by the developers
• the review for the test cases is done by the testers
• the creation and approval of the Spectron test cases are codified in the Definition of Done

Advantages

• Time saved in testing: The real reason for this procedure is the scarcity of resources on the testers’ part. If they had had to take on the automation as well, the entire project would not have been possible.
• Change of perspective in testing: The testers can learn quite a lot in the discussion and review. For example, the implementation becomes clearer when questions are asked about why a test has been written this way. This can result in test cases that would otherwise not have been thought of.
• Development of know-how: Since writing tests while the development is underway is routine work for programmers, the basic understanding regarding the creation of automated tests is generally very good. In our project, this has already proven useful for several reasons:
  • Parts of the applications could be covered using technical tricks that a tester would not have been readily able to provide, e.g. automated testing of the correct representation of a point cloud in a chart and the display of the details of a selected point.
  • Technical refinements enabled us to significantly improve the performance and stability of the Spectron tests.
  • After changing the tools used, the duration of a complete Spectron run was reduced by half an hour (time savings of 25%)

• Change of perspective in development: Due to the fact that the developers concerned themselves with the user’s perspective regarding the functions and the interface of the software, a large number of errors could be avoided, and the basic understanding increased due to the intense interaction with the testers.

Disadvantages

• More time required for the developers: The time saved in one place is needed elsewhere. However, the work can be shouldered by several people in this case.
• Structure: Developers classify test cases into logical areas from a technical point of view. As they are not always identical to the functional logic, testers may have trouble finding and reviewing specific test cases.

Challenges and solutions

• Traceability DEV QA: The review is done in Git (diff tool) in the project. In the project, the test team reviewed modified and newly created Spectron test cases, but not deleted ones, based on the assumption that those had been replaced in the course of the modification. Consequently, some requirements were no longer covered.
  Solution: To solve the problems with the review, training everybody who has to work in and review using Git on how to use Git is particularly helpful. Doing a walkthrough with the test team and the development team in the case of major modifications is useful as well to allow the testers to better understand the implementation by the developers.

• Traceability of Spectron requirements: This is a challenge that was specific to our project environment. The agile team uses Jira for the requirements and test management, but in the client’s environment, the requirements and test cases have to be specified by means of different requirements management software for legal reasons. Since the systems do not know each other, automatic traceability cannot be ensured.
  Solution: To overcome this obstacle, we established a direct allocation of the Req ID to the Spectron clusters.

Conclusion

In conclusion, we can say that our concept of only having the test analyst review automated test cases instead of writing them proved to be effective in our project. The division of tasks between the testers and the developers fits very well with the agile method (scrum). The advantages far outweigh the disadvantages.

This approach is perfect for an agile project that is being built from scratch with a small staff and a high standard of quality. However, you should use this method from the start. Integrating it at a later time is almost impossible because gradually expanding the test cases after each user story is much more manageable and easier than creating the test cases en bloc. Furthermore, decisions regarding the implementation, structure, architecture, and most importantly, the processes (Definition of Done, …) are made at the beginning.

This post was written by: