The QA Navigation Board enables teams to make targeted and efficient decisions for every software project with regard to aspects of quality assurance, their order of priority and their implementation. We have created a short video tutorial to explain how it works.
Figure 1: QA Navigation Board for practical application
How the QA Navigation Board works
Every software project is characterised by its own individual focus. The resulting quality requirements are equally as specific. For this reason, suitable measures – according to each topic – must be established in order to test these quality requirements. With this in mind, we developed the QA Navigation Board to accurately evaluate the methods and plan the sequences in a meaningful way: The Board is an aid that enables teams to organise testing for software projects in the best possible way. The following questions are taken into account: What needs to be tested, how, to what extent, where and when?
Video Tutorial (German only)
An introduction to the tool is helpful, as it ensures that the Board can be used correctly and without delay. We have therefore explained how it works in a supporting video tutorial. How is it structured? In an ideal world, which test planning steps should be carried out in sequence and why? This enables the user to make the best possible start, step by step.
Experiences using the QA Navigation Board
The QA Navigation Board is already being used consistently and with great success across numerous projects. Teams that have used the Board say that: It is inclusive of every single member of the team and their respective skills. It ensures that no aspect of the software is overlooked and that quality assurance takes place in line with their objectives. Working with the QA Navigation Board therefore ensures clarity, gives a clear focus and allows for collaborative creation across the whole team in a workshop.
Introductory workshop
For teams that want to incorporate the Board into their processes, we are still offering a detailed schedule for running an introductory workshop for the QA Navigation Board. A detailed description of how to complete the Board as well as the meaning behind each individual point can be found and referred back to at any time here.
When it comes to recognising and eliminating typical QA problems and their effects, the Board can be used as a tool for the “case history of the QA strategy for agile teams” – a description of the procedure is available here.
We would like to present our agile visualization tool, the QA battle plan, a tool that allows agile development teams to recognize and eliminate typical QA issues and their effects.
Like a wrong architectural approach or using the wrong programming language, the wrong testing and quality assurance strategy can result in adverse effects during the course of a project. In the best case, it only causes delays or additional expenses. In the worst case, the tests done prove to be insufficient, and severe deviations occur repeatedly when the application is used.
Introduction
Agile development teams notice issues and document their effects in the Retrospectives, but they are often unable to identify the root cause, and therefore cannot solve the problem because they lack QA expertise. In such cases, the teams need the support of an agile QA coach. This coach is characterized on the one hand by his knowledge of the agile working method, and on the other hand by his experience in agile quality assurance.
The first step in the agile QA coach’s work is recording the status quo of the testing methods of the agile development team. For this purpose, he will use the QA battle plan, e.g. within the framework of a workshop. The QA battle plan provides a visual aid to the development teams which they can use to assess the planning aspects of quality assurance. Throughout the project, the QA battle plan can also be used as a reference for the current procedure and as a basis for potential improvements.
Anti-patterns
In addition, the QA battle plan makes it possible to study the case history of the current testing method. By means of this visualization, the agile QA coach can reveal certain anti-pattern symptoms in the quality assurance and testing process, and discuss them directly with the team. In software development, an anti-pattern is an approach that is detrimental or harmful to a project’s or an organization’s success.
I will describe several anti-patterns below. In addition to the defining characteristics, I will present their respective effects. As a contrast to the anti-pattern, the pattern—good and proven problem-solving approaches—will also be presented.
The “It’ll be just fine” Anti-pattern
This anti-pattern is characterized by the complete lack of testing or other quality assurance measures whatsoever. This can cause severe consequences for the project and the product. The team cannot make any statement regarding the quality of their deliverables, and consequently, they do not, strictly speaking, have a product that is ready for delivery. Errors occur upon use by the end user, repeatedly distracting the team from the development process because they have to analyze and rectify these so-called incidents, which is time-consuming and costly.
No testing
Effect
Solution
• There are no tests
• No quality statement
• “Leave quickly”
• Testing is done in the user’s environment
• Introduce QA
The solution is simple: Test! The sooner deviations are discovered, the easier it is to remove them. In addition, quality assurance measures such as code reviews and static code analysis are constructive measures for consistent improvement.
The Dysfunctional Test Anti-pattern
ISO 25010 specifies eight quality characteristics for software: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. When new software is implemented, the focus is most often on functional suitability, but other characteristics such as security and usability play an important role today as well. The higher the priority of the other quality characteristics, the more likely a non-functional test should be scheduled for them.
Therefore, the first question to be asked at the start of a software project is: Which quality characteristics does the development, and therefore the quality assurance, focus on? To facilitate the introduction of this topic to the teams, we use the QA octant. The QA octant contains the quality characteristics for software systems according to ISO 25010. These characteristics also point to the necessary types of tests that result from the set weighting of the different functional and non-functional characteristics.
Functional tests only
Effect
Solution
• There are only functional tests
• No quality statement about non-functional characteristics
• Discuss important quality characteristics with the client
• It works as required, but…
• Non-functional test types
• Start with the QA octant
The Attack of the Development Warriors Anti-pattern
Many agile development teams—especially teams that consist only of developers—only rely on development-related tests for their QA measures. Usually, they only use unit and component tests for this purpose. Such tests can easily be written with the same development tool and promptly be integrated in the development process. The possibility to obtain information about the coverage of the tests with respect to the code by means of code coverage tools is particularly convenient in this context. Certainty is quickly achieved if the code coverage tool reports 100% test coverage. But the devil is in the detail, or in this case, in the complexity. This method would be sufficient for simple applications, but with more complex applications, problems arise.
With highly complex applications, errors may occur despite a convenient unit test coverage, and such errors can only be discovered with extensive system and end-to-end tests. And for such extensive tests, the team needs advanced QA know-how. Testers or trained developers have to address the complexity of the application at higher levels of testing in order to be able to make an appropriate quality statement.
The attack of the development warriors
Effect
Solution
• Development-related tests only
• No end-to-end test
• Include a tester in the team
• No tester on the team
• Bugs occur with complex features
• Test at higher levels of testing
• 100% code coverage
• Quick
The “Spanish Option” Anti-pattern
The time in which a function has been coded and is integrated into the target system is becoming ever shorter. Consequently, the time for comprehensive testing is becoming shorter as well. For agile projects with fixed iterations, i.e. sprints, another problem arises: The number of functions to be tested increases with every sprint.
Plain standard manual tests cannot handle this. Therefore, testers and developers should work together to develop a test automation strategy.
Manual Work
Effect
Solution
• There are only manual tests
• Delayed feedback in case of errors
• Everybody shoulders part of the QA work
• Testers are overburdened
• Test at all levels of testing
• Introduce automation
The Automated Regression Gap Anti-pattern
A project without any manual tests would be the other extreme. Even though this means a high degree of integration in the CI/CD processes and quick feedback in case of errors, it also causes avoidable problems. A high degree of test automation requires great effort—both in developing the tests and in maintenance. The more complex the specific applications, and the more sophisticated the technologies used, the higher the probability of test runs being stopped due to problems occurring during the test run, or extensive reviews of test deviations being required. Furthermore, most automated tests only test the regression. Consequently, automated tests would never find new errors, but only test the functioning of the old features.
Therefore, automation should always be used with common sense, and parallel manual and, if necessary, explorative tests should be used to discover new deviations.
100% test automation
Effect
Solution
• There are only automated tests
• Very high effort
• Automate with common sense
• Everybody is overexerted
• Manual testing makes sense
• Build stops due to problems
The Test Singularity Anti-pattern
Tests of different types and at different levels each have a different focus. Consequently, they each have different requirements regarding the test environment, such as stability, test data, resources, etc. Development-related test environments are frequently updated to new versions to test the development progress. Higher levels of testing or other types of tests require a steadier version status over a longer period of time.
To avoid possibly compromising the tests due to changes in the software status or a modified version, a separate test environment should be provided for each type of test.
One Test Environment
Effect
Solution
• There is only one test environment
• No test focus possible
• Several test environments
• Compromised tests
• By level of testing or test focus
• No production-related tests
• “One test environment per type of test”
The “Manual” Building Anti-pattern
State-of-the-art development depends on fast delivery, and modern quality assurance depends on the integration of the automated tests into the build process and the automated distribution of the current software status to the various (test) environments. These functions cannot be provided without a build or CI/CD tool.
If there are still tasks to be done relating to the provision of a CI/CD process in a project, they can be marked as “to do” on the board.
No Build Tool
Effect
Solution
• There is no build tool
• No CI/CD
• Introduce CI/CD
• Slow delivery
• Highlight gaps on the board
• Delayed test results
• Dependent on resources
The Early Adopter Anti-pattern
New technologies usually involve new tools, and new versions involve new features. But introducing new tools and updating to new versions also entail a certain risk. It is advisable to proceed with care, and not to change the parts/tools of the projects all at once.
The QA Navigation Board provides a visual aid to agile development teams which they can use to assess the planning aspects of quality assurance at an early stage. During the project duration, the QA Navigation Board can also be used as a reference for the current procedure and as a basis for potential improvements. » QA Navigation Board
The QA Navigation Board is developed within the framework of a workshop run by an agile QA coach. The duration of the workshop should not exceed 1.5 hours.
Preparation
All the parties involved in the agile project should be invited:
Development team (developers, testers)
Scrum master
Product owner
Other stakeholders and shareholders
The QA Navigation Board is affixed to a bulletin board or wall. In addition, each participant receives a printout of the QA Octant as a worksheet.
Step 1:
Presentation of the QA Navigation Board and the objectives of the workshop by the host (agile QA coach), and introduction of the participants.
Step 2:
Brief presentation of the QA Octant and the quality characteristics. The goal is for all the participants to be able to complete the worksheet and to understand the quality characteristics so that they do not talk at cross purposes later.
Furthermore, the participants agree on the dimensions of the QA Octant: Which label is to be given to the intervals of the diagram (1, 2, 3 or S, M, L, XL, etc.)? Then, the worksheets are handed out and completed within 5 to 10 minutes by each participant, the name of whom is indicated on the worksheet (cf. blog post: How to use the QA Octant).
Step 3:
At the end of this time, the host collects the worksheets and puts them up on a bulletin board or wall.
The host then goes through each of the quality characteristics. For this purpose, he identifies the common denominator (average) of each characteristic and discusses the greatest deviations with the respective persons (cf. planning poker). Once the team reaches a consensus regarding the value of a characteristic, the host documents this value.
Step 4:
Based on the valuation of the quality characteristics, the participants then deduce the necessary types of tests. The higher the value of a quality characteristic, the more likely it requires testing by means of an appropriate test procedure. The team then places the types of tests determined in the test pyramid of the QA Navigation Board.
Step 5:
Once all types of tests have been determined and placed, the necessary test resources and other test artifacts can be placed on the QA Navigation Board. A checklist can help in this respect (cf. blog post: The QA Map or “How to complete the QA Navigation Board”).
Step 6:
When the team has mostly completed the QA Navigation Board, it is put up in an appropriate place in the team room. The host concludes the workshop and points out that the QA Navigation Board can be updated and further developed by the team, and also used in retrospectives.
The QA Navigation Board provides a visual aid to the development teams which they can use to assess the planning aspects of quality assurance at an early stage. During the project duration, the QA Navigation Board can also be used as a reference for the current procedure and as a basis for potential improvements. But how should the types of tests and other test artifacts be placed on the QA Navigation Board?
To answer the question, “How and where do we want to test?”, the team would have to comb through the entire development process to find and document test and QA aspects. The development process can be different for every project, which could quickly make this issue highly complex (Fig. 1).
Figure 1: Development and QA process
Again, to facilitate the introduction of this topic to the teams, we have developed the QA Map. The QA Map gives the team a practical tool to plan and document the measures required for optimal testability of the projects. The objective is to determine all QA-relevant issues for the teams and development projects, using a playful approach and at an early stage.
Figure 2: The QA Map
After defining all the key test areas by means of the QA Octant and determining the necessary types of tests, all aspects of the test strategy, such as types of tests, resources and tools, can be visualized, discussed, and prioritized.
A good practice resulting from the workshops done in the past is using two tools to control the completion of the QA Map. The first is a competent host who leads the workshop in the right direction, and the second is using a check list. The check list comprises appropriate questions that are intended to provide suggestions in the workshop in order to complete the various parts of the QA Map. These questions are listed below and allocated to the respective field to be completed.
Requirements
What are the requirements?
Do the requirements support the preparation of the test case?
Can requirements and tests be linked?
Test / Code
Where do we place the tests?
Do we have the necessary skills?
Repository
Where do we store the test artifacts?
Are there different artifacts?
Test Management
How do we plan our tests?
How do we document our tests?
How do we report? And to whom?
Automation
How much test automation is required?
Do we need additional tools?
Do we need test data?
Build
How often do we want to build and test?
How do we want to integrate QA?
Do we want to test maintainability?
Test Environments
Do we have an adequate environment for every test?
Will we get in each other’s way?
Figure 3: Example 1 of a completed QA Navigation BoardFigure 4: Example 2 of a completed QA Navigation Board
Once all types of tests have been selected and the team has started to place the other test artifacts (e.g. tools, environments), the host can withdraw. The team should put up the final picture in the team room as an eye-catcher. This way, the QA Navigation Board plan can be used as a reference for the current procedure and as a basis for potential improvements.
One of the questions asked at the start of a software project is: Which quality characteristics does the development, and therefore the quality assurance, focus on? To facilitate the introduction of this topic to the teams, we use the QA Octant. The QA Octant contains the quality characteristics for software systems according to ISO 25010. These characteristics also point to the necessary types of tests that result from the set weighting of the different functional and non-functional characteristics.
Depending on how much the implemented requirements affect the quality characteristics, it is necessary to check these characteristics by means of a corresponding type of test. Apps with a high data throughput for example require efficiency tests, whereas web shops should be tested for compatibility in various browsers. Thanks to the simple visualization and weighting of the different quality characteristics, the QA Octant can be used for planning.
The team asks the product owner or the department: “How important is each of the quality characteristics?” The goal of this round of questions is to visualize a ranking in the weighting of the different characteristics. Most of the respondents will not really differentiate between the quality characteristics, or rather they will answer: “Everything is important!”
It is now up to the team and the host of the meeting to clarify the question to the point that such a differentiation is possible. Different questioning techniques can be used for this purpose.
Differentiation is for example possible by delimiting the area of application. If an HTML-based technical application is used in a company network, and the IT compliance regulations specify one browser and one operating system version, the aspect of compatibility and the associated tests can be ranked lower. If, by contrast, a large number of different combinations of platforms are used, extensive testing has to be planned.
For further differentiation, you can for example use a negative questioning technique: “What happens if, for example, usability is reduced?” Using the example of an application for monthly invoicing, we assume that a negative effect on the usability increases the time it takes to issue an invoice from two to four hours. Since the application is only used once every month, this “delay” would be acceptable, and usability can be ranked lower in the QA Octant.
This questioning technique can be expanded by prioritizing by means of risk assessment. “What happens, or which consequences arise if, for example, the security characteristic is lowered?” The answers result from the following aspects:
What financial impact would a failure of the application have if the focus on this characteristic was reduced?
How many users would be affected by a failure of the application if the focus on this characteristic was reduced?
Would a failure of the application cause danger to life and limb if the focus on this characteristic was reduced?
Would a failure of the application affect the company’s reputation if the focus on this characteristic was reduced?
If results and findings are available with respect to one or several of the quality characteristics, you can compare them to the open quality characteristics and proceed similarly to the complexity comparison for the planning or estimation.
Asking the right questions produces an overview of the quality characteristics. Thanks to the simple visualization and weighting of the different quality characteristics, the QA Octant can be used for planning the types of tests.
The result is not always the most important part of the QA Octant: “the journey is the destination” as well. Due to the weighting in the team and together with the PO and/or the department, different opinions are more discernible, and all the parties involved develop a better understanding.
