Simple Case History of the QA Strategy for Agile Teams

We would like to present our agile visualization tool, the QA battle plan, a tool that allows agile development teams to recognize and eliminate typical QA issues and their effects.

Like a wrong architectural approach or using the wrong programming language, the wrong testing and quality assurance strategy can result in adverse effects during the course of a project. In the best case, it only causes delays or additional expenses. In the worst case, the tests done prove to be insufficient, and severe deviations occur repeatedly when the application is used.

Introduction

Agile development teams notice issues and document their effects in the Retrospectives, but they are often unable to identify the root cause, and therefore cannot solve the problem because they lack QA expertise. In such cases, the teams need the support of an agile QA coach. This coach is characterized on the one hand by his knowledge of the agile working method, and on the other hand by his experience in agile quality assurance.

The first step in the agile QA coach’s work is recording the status quo of the testing methods of the agile development team. For this purpose, he will use the QA battle plan, e.g. within the framework of a workshop. The QA battle plan provides a visual aid to the development teams which they can use to assess the planning aspects of quality assurance. Throughout the project, the QA battle plan can also be used as a reference for the current procedure and as a basis for potential improvements.

Anti-patterns

In addition, the QA battle plan makes it possible to study the case history of the current testing method. By means of this visualization, the agile QA coach can reveal certain anti-pattern symptoms in the quality assurance and testing process, and discuss them directly with the team. In software development, an anti-pattern is an approach that is detrimental or harmful to a project’s or an organization’s success.

I will describe several anti-patterns below. In addition to the defining characteristics, I will present their respective effects. As a contrast to the anti-pattern, the pattern—good and proven problem-solving approaches—will also be presented.

The “It’ll be just fine” Anti-pattern

This anti-pattern is characterized by the complete lack of testing or other quality assurance measures whatsoever. This can cause severe consequences for the project and the product. The team cannot make any statement regarding the quality of their deliverables, and consequently, they do not, strictly speaking, have a product that is ready for delivery. Errors occur upon use by the end user, repeatedly distracting the team from the development process because they have to analyze and rectify these so-called incidents, which is time-consuming and costly.

No testingEffectSolution
• There are no tests• No quality statement• “Leave quickly”
• Testing is done in the user’s environment• Introduce QA

The solution is simple: Test! The sooner deviations are discovered, the easier it is to remove them. In addition, quality assurance measures such as code reviews and static code analysis are constructive measures for consistent improvement.

The Dysfunctional Test Anti-pattern

ISO 25010 specifies eight quality characteristics for software: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. When new software is implemented, the focus is most often on functional suitability, but other characteristics such as security and usability play an important role today as well. The higher the priority of the other quality characteristics, the more likely a non-functional test should be scheduled for them.

Therefore, the first question to be asked at the start of a software project is: Which quality characteristics does the development, and therefore the quality assurance, focus on? To facilitate the introduction of this topic to the teams, we use the QA octant. The QA octant contains the quality characteristics for software systems according to ISO 25010. These characteristics also point to the necessary types of tests that result from the set weighting of the different functional and non-functional characteristics.

Functional tests onlyEffectSolution
• There are only functional tests• No quality statement about non-functional characteristics• Discuss important quality characteristics with the client
• It works as required, but…• Non-functional test types
• Start with the QA octant

The Attack of the Development Warriors Anti-pattern

Many agile development teams—especially teams that consist only of developers—only rely on development-related tests for their QA measures. Usually, they only use unit and component tests for this purpose. Such tests can easily be written with the same development tool and promptly be integrated in the development process. The possibility to obtain information about the coverage of the tests with respect to the code by means of code coverage tools is particularly convenient in this context. Certainty is quickly achieved if the code coverage tool reports 100% test coverage. But the devil is in the detail, or in this case, in the complexity. This method would be sufficient for simple applications, but with more complex applications, problems arise.

With highly complex applications, errors may occur despite a convenient unit test coverage, and such errors can only be discovered with extensive system and end-to-end tests. And for such extensive tests, the team needs advanced QA know-how. Testers or trained developers have to address the complexity of the application at higher levels of testing in order to be able to make an appropriate quality statement.

The attack of the development warriorsEffectSolution
• Development-related tests only• No end-to-end test• Include a tester in the team
• No tester on the team• Bugs occur with complex features• Test at higher levels of testing
• 100% code coverage• Quick

The “Spanish Option” Anti-pattern

The time in which a function has been coded and is integrated into the target system is becoming ever shorter. Consequently, the time for comprehensive testing is becoming shorter as well. For agile projects with fixed iterations, i.e. sprints, another problem arises: The number of functions to be tested increases with every sprint.

Plain standard manual tests cannot handle this. Therefore, testers and developers should work together to develop a test automation strategy.

Manual WorkEffectSolution
• There are only manual tests• Delayed feedback in case of errors• Everybody shoulders part of the QA work
• Testers are overburdened• Test at all levels of testing
• Introduce automation

The Automated Regression Gap Anti-pattern

A project without any manual tests would be the other extreme. Even though this means a high degree of integration in the CI/CD processes and quick feedback in case of errors, it also causes avoidable problems. A high degree of test automation requires great effort—both in developing the tests and in maintenance. The more complex the specific applications, and the more sophisticated the technologies used, the higher the probability of test runs being stopped due to problems occurring during the test run, or extensive reviews of test deviations being required. Furthermore, most automated tests only test the regression. Consequently, automated tests would never find new errors, but only test the functioning of the old features.

Therefore, automation should always be used with common sense, and parallel manual and, if necessary, explorative tests should be used to discover new deviations.

100% test automationEffectSolution
• There are only automated tests• Very high effort• Automate with common sense
• Everybody is overexerted• Manual testing makes sense
• Build stops due to problems

The Test Singularity Anti-pattern

Tests of different types and at different levels each have a different focus. Consequently, they each have different requirements regarding the test environment, such as stability, test data, resources, etc. Development-related test environments are frequently updated to new versions to test the development progress. Higher levels of testing or other types of tests require a steadier version status over a longer period of time.

To avoid possibly compromising the tests due to changes in the software status or a modified version, a separate test environment should be provided for each type of test.

One Test EnvironmentEffectSolution
• There is only one test environment• No test focus possible• Several test environments
• Compromised tests• By level of testing or test focus
• No production-related tests• “One test environment per type of test”

The “Manual” Building Anti-pattern

State-of-the-art development depends on fast delivery, and modern quality assurance depends on the integration of the automated tests into the build process and the automated distribution of the current software status to the various (test) environments. These functions cannot be provided without a build or CI/CD tool.

If there are still tasks to be done relating to the provision of a CI/CD process in a project, they can be marked as “to do” on the board.

No Build ToolEffectSolution
• There is no build tool• No CI/CD• Introduce CI/CD
• Slow delivery• Highlight gaps on the board
• Delayed test results
• Dependent on resources

The Early Adopter Anti-pattern

New technologies usually involve new tools, and new versions involve new features. But introducing new tools and updating to new versions also entail a certain risk. It is advisable to proceed with care, and not to change the parts/tools of the projects all at once.

Early AdopterEffectSolution
• Always the latest of everything…• Challenging training• No big bang
• Deficiencies in skills• Old tools are familiar
• New problems• Highlight deficiencies in skills on the board

Recipes for Test Automation (Part 2) – Data Salad

Testomatoes on data salad with stressing

The test data always pose a particular challenge in manual testing, and even more so with respect to test automation. In most manual tests, the test cases frequently only include general information regarding the test data to be used. This method does not work for test automation.

In my previous post, „Ingredients and appliances for test automation, and who is the chef“, I described the prerequisites for test automation that have to be met in order to successfully implement automated processes. I also mentioned another challenge in this context: The test data. I would like to take a closer look at this issue in this blog post.

What happens if we fail to focus on the test data in the test automation, and we rely on test data that do not take the test automation into account?

Testing can be compared with cooking. The test case is the recipe, and the test data are the ingredients. With manual testing, we follow the recipe/test case and find the ingredients/test data as required. This does not work with automated testing. Here, the test data, or ingredients, have to be provided spot-on, in the exact quantities required. This means that it is not enough to indicate the type and form of the test data: the exact instance of the test date has to be specified in the test script as well.

Furthermore, the test data are used up, or the test data age over the course of the testing. Just like in a restaurant, where you eventually run out of tomatoes or the green salad wilts. So then, where do we get the ingredients we need, and in sufficient quantities.

In projects, I often hear: “We’ll just create a—hopefully anonymized—clone of the production data.” However, such a clone only provides a part of the data required for the test automation. This means that such a clone is quite useful for unchanging master data. But the cook in the kitchen will not always get the same order for the exact same salad. One guest wants olives on their salad, another does not; one wants yogurt dressing, the other wants oil and vinegar. Depending on the ingredients and changes to the order, the price changes as well. This means that we also need dynamic data for our test data, so-called transactional data, to comprehensively represent the business processes. There are two approaches to providing the necessary dynamic data for the test automation, each of which has its advantages and disadvantages. With the first approach, the required data are picked from the anonymized clone of the production data. However, the effort required to determine the respective filter criteria and then formulate a corresponding query can quickly become very high. The disadvantage of this approach is the fact that the quantity of the filtered data is limited, and the data can be used up during testing.

With the second approach, the required test data are newly created in the database, ensuring that all the test data required for the test case are provided. Although these test data are used up as well, they can be recreated time and again by the automated scripts. Creating the so-called synthetic test data can be arduous as well, e.g. when there are large amounts of dependent data. Therefore, the decision of which approach is to be used for which test case has to be evaluated for each individual case.

Furthermore, the test data often have to be dynamized in the test automation. What does that mean? Let us look at another example from the kitchen. Everyone knows that a good Peking duck should be ordered 24 hours before it is to be eaten. As the date of consumption is a variable date, dynamization can offer a solution for the automation. The result then looks like this: Date of consumption = order date + 24 hours. This and similar kinds of dynamic test data are also used in test automation.

Conclusion: In most cases, the solution is like a good salad—it’s all in the mix. Let us once more summarize the recipe: Take an anonymized clone of the production data for the basic, unchanging master data (you can also create smaller amounts of data yourself), and add some well-selected dynamic data from the clone of the production data by means of a query in the test case. Now add a well-selected dose of synthetic test data. All of this should be well balanced to match the test case template. Lastly, top the whole thing off with a splash of dynamized test data in the template, and you have the perfect data salad for your test automation. After serving the test data to the database, it is important to thoroughly clean up the data kitchen. That means returning all the static, synthetic and dynamic test data to their original state if they were modified by the test case. Simply put, every test case necessarily entails a clean-up of the test data because tidiness is the top priority in a good data kitchen.

As we all know, various appliances are also used in the kitchen to automate the cooking process. My next blog post in this series on recipes for test automation addresses the question how much test automation is good for a project. Until then: Happy testing, and keep your data kitchen neat and tidy.

Recipes for Test Automation (Part 1) – Soup

Ingredients, appliances, and who is the chef?

A colleague recently approached me and asked me whether I know a recipe for good test automation. I told him that, just like with a good soup, it takes more than a recipe: The kitchen appliances, ingredients, and the cook are important as well. Accordingly, the project framework, the selection of the test tools, and the testers involved in the test automation are decisive in test automation.

How do we find out if the framework is right for test automation? Generally speaking, we have to differentiate between software projects that are still in the pre-planning stage where test automation is planned from the start, and projects that are already underway, where test automation is intended to support the project.

First of all: The earlier well-planned test automation is started, the more likely it will be successful. This is often due to the fact that the initial effort involved in test automation is underestimated, and the benefits, in terms of figures at least, are not reaped until after the project has been completed.

To counter this, a project that is about to start requires good planning, and a project in progress requires a preliminary study of the ongoing processes, test cases, and framework. Such a preliminary study is used to analyze and evaluate the existing test case portfolio, the test object and the testers’ knowledge, among other aspects. This way, we can identify the project’s true weaknesses in the field of testing.

Test automation is often used to try and solve problems that have nothing to do with the test automation, and that can sometimes even be exacerbated by it. Just like with cooking: When the soup tastes bad, replacing the cook with an automaton will not necessarily improve it. The result will, in fact, often be even worse—just think of the coffee maker out in the hallway. It is important to ensure that the prerequisites for the test automation have reached a minimum quality before you can start selecting the tools or begin the actual automation. Test automation is the dab of cream that tops off the soup, i.e. test automation can only be successful with a well-functioning test process. The test cases also need a sufficient level of detail for automation The level of detail required depends on the level of technical know-how the tester has in regard to the test object. Some people might suggest that the technical department, i.e. those who have the technical know-how, could carry out the test automation. However, that is usually not a good idea because these colleagues cannot spare the time. On the one hand, their everyday work always comes before the test automation in terms of priorities, and on the other hand, they would need to gain proficiency in the use of the tools and testing experience first. Therefore, the technical department should preferably provide the manual test cases with a sufficient level of detail that the tester understands even without knowledge of the domain. The tester then uses their testing experience and knowledge of the tools to carry out the automation.

It is illusory to believe that you can automate everything, which is why the test cases are prioritized in the preliminary study. In this context, it must be checked whether the test object or the software that is to be tested can be automated. Dynamic object IDs that change every time they are requested, for example, pose a particular challenge in test automation.

And then, like the cook in the kitchen, the tester who carries out the test automation also plays an important role. Firstly, they need the necessary knowledge of the tools, and secondly, they have to be able to decide where, specifically, test automation is worthwhile. For example, it makes no sense to automate a feature of the software that is rarely needed, yet takes great effort to automate.

This is why you should always adhere to the recipe:

  • Use common sense and an instinct for the effort involved to create a harmonious composition: Keep an eye on the cost and benefits.
  • Do not rely on technology alone: Your craftsmanship is important as well.
  • Test automation only works if the basic test tasks and processes work first: First, you learn to slice and dice onions, then you get to work on the bouillabaisse.

The test data are another challenge in test automation. I will address this issue in my next blog post. Until then: Happy Holidays—and don’t rely on some machine to prepare your Christmas dinner.