In the first and second part, we successfully completed the agile transition of the test manager’s responsibilities, and we proved that small scrum teams no longer require a test manager. Several scrum teams working together on the same product pose another challenge. Can the previous observations made with respect to a small scrum project with a single team also be adopted for large projects?
Do we need a test manager for large scrum projects?
In many companies that use scrum, several teams work together on the development of the same product. There have already been some thoughts regarding scaled scrum on the company level and how it is possible to comply with agile principles even as the level of complexity increases. In this context, Boris Glober identifies two problematic areas: “compliance with the scaled scrum framework” and “scaling of the requirements process”. To solve this, he introduces additional roles: the company scrum master and the company product owner, who assist all the scrum masters and product owners of the project. Both roles coordinate with their counterparts in the individual scrum teams, and they manage the scrum tools on the company level, such as the company product backlog and the company scrum board.
Can this concept be adopted for testing as well, i.e. do we need something like a company test owner or a company QA master? The purpose of this new tool would be the coordination of and responsibility for the overall, integrative testing process, establishing the necessary common test structures, and filling the backlog with stories, tasks and incidents relating to QA.
Figure 1: Coordination meeting on QA issues
For projects with a manageable number (2 to 8) of scrum teams, this kind of coordination can be done within the framework of the general coordination meeting, the scrum of scrums. If the coordination becomes more test-specific and time-consuming, it is advisable to establish a separate test meeting as required. Each team sends a team member with the necessary test know-how to this meeting. If the number of teams is higher, or if coordination between several projects is required with respect to QA issues, the guilds can again be used for this purpose. The guilds collect examples of best practices that they provide to all the projects, or they appoint coaches who bring agile testing methods to new projects. The guild masters agree on important decisions and moderate the scrum teams wherever it is necessary to define common rules and solutions.
We can conclude that agile companies no longer need a test manager, even for large projects. However, this is possible only by way of a complete agile transition of the test manager’s responsibilities because each of the test manager’s responsibilities—in particular with respect to communication and coordination—has to be matched with an agile tool.
In the first part, we successfully completed the agile transition of the test manager’s operative responsibilities, and we proved that small scrum teams no longer require a test manager. But will this work for the test manager’s strategic responsibilities, too? And who assumes the test manager’s strategic responsibilities?
Who assumes the test manager’s strategic responsibilities in agile companies?
A test manager’s strategic area of responsibility includes quality management (QM), which, according to ISO 8402, comprises “all activities of the overall management function that determine the quality policy, objectives and responsibilities and implement them by means such as quality planning, quality control, quality assurance and quality improvement within the quality system”.
Figure 1: Responsibilities of the test manager (according to the ISTQB)
Irrespective of whether a company uses traditional or agile methods in development, the company’s management is responsible for the quality management. The scrum team cannot, should not and must not assume this responsibility. At first glance, nothing in the organization and the division of responsibilities changes. At second glance, however, a problem arises for agile companies. In the traditional world, the test manager used to act as the interface between the operative and the strategic level. They transmitted information between the management and the operative level, passed strategic specifications and methods on to the test teams, or forwarded developments and improvements to the management.
But how does information get transmitted from the strategic to the operative level and vice versa in an agile environment? The answer: The agile transition does not stop with the allocation of the test manager’s operative responsibilities to the scrum team. The solution for the agile transition of the “interface” between the two levels is similar to the agile transition of the test manager’s operative responsibilities. New concepts and tools are available for the current corporate communication tasks.
One of the new concepts is the so-called guild. Guilds (called competence teams in our company) are a tool that puts and keeps knowledge and information management on track. They are organized in a matrix structure apart from the normal company structure. The objective of the guilds is to pool the company’s know-how carriers, offer the staff a platform to exchange knowledge or carry out training programs, or agree general project decisions such as the development of test environments or rules for code quality. Depending on the company’s goals, the structure and composition of the guilds can differ: For example, guilds can be divided according to competence areas such as Java development, .NET development, test or process analysis. Or entire scrum teams can be aggregated in guilds, working together on a specific subject in the project, such as QA, database connection, GUI, or interfaces (see Figure 1).
Figure 2: Example of classification and structure of the guilds by competence area
The guilds work according to the following pattern: The primary role that each employee assumes in their daily work (e.g. developer, tester [QAlas], product owner and scrum master) is determined. With this role, they are allocated to a corresponding guild. Within the guild, the members exchange information relating to their areas of responsibility, carry out in-house training seminars, collect the available knowledge in portals, or discuss best practices in individual projects. A guild master is responsible for the moderation and coordination within the guild. In accordance with the motto “primus inter pares”, they do not have more rights than their colleagues in the guild, and they are elected from among the guild members. The guild master is the principal contact for the members of their own guild, the other guild masters and the management. This is necessary because the guilds engage in active interdisciplinary exchange, and they are also supposed to forward feedback from the operative level—such as new developments or technologies—to sales, or suggestions regarding training seminars to HR.
In the last part, we will look at what happens when several scrum teams work together on the same project and the effort required to coordinate them increases for the field of QA as well.
Some time ago, a colleague of mine asked me whether we still needed a test manager in an agile development process like scrum. My first response was no because the Agile Manifesto and the scrum framework only know three roles: product owner, development team, and scrum master. Accordingly, the scrum team—i.e. the three scrum roles mentioned above—does not provide for a test manager. But on second thought, I wondered who among the scrum team was supposed to assume the test manager’s responsibilities in and around the sprint?
Studies such as the 2014 ASQF Industry Report[1] and the 2011 Standish Chaos Report show that agile methods have already become a permanent fixture in companies. Furthermore, the Standish Chaos Report shows that projects using agile processes are more likely to be successful than “conventional projects”. The Agile Manifesto of Ken Schwaber and Jeff Sutherland was the basis for this development. It defines basic principles and specifications that uncover “better ways of developing software by [the parties involved in the process] doing it and helping others do it” [²].
Figure 1: Scrum process and parties involved
The key principles from the Agile Manifesto are:
Individuals and interactions are more important than processes and tools.
Working software is more important than comprehensive documentation.
Client collaboration is more important than the original specifications.
Responding to change is more important than following a plan.
Companies that switch to the agile method of development have a competitive advantage compared to companies using traditional methods. However, transitioning the processes to development methods such as scrum—also called agile transition—presents a great challenge. Agility is not achieved by divvying the development milestones up into sprints and appointing a product owner (see Figure 1). Comprehensive changes in the organization are required to achieve an agile way of working and living.
The challenge of the agile transition becomes very clear when you look at the example of the test manager. The question: If we do not need a test manager in scrum, who assumes their responsibilities? The product owner? The scrum master? The team?
According to the Scrum Alliance, the product owner is the person responsible for developing the product accurately and on time. The product owner fills and refines the product backlog, and ensures that everybody knows what it contains and what is given which priority. Consequently, they are usually closest to the “business side” of the project. Scrum requires the development team to be a group with mixed functions—pooling all the necessary skills—that are required for the development of the product. The team organizes itself, i.e. it independently chooses the content to be implemented in the sprint and takes care of the planning, management, and implementation. The scrum master is the “pilot” guiding them through the depths of the scrum framework, helping the rest of the scrum team to comply with scrum principles. Another task of the scrum master is removing obstacles that hamper the team’s progress
Even after studying the scrum roles, we do not know who assumes the test manager’s responsibilities, or how they are allocated. To answer these questions, we must first determine the test manager’s responsibilities in the traditional testing and quality assurance process. According to the International Software Testing Qualifications Board (ISTQB), the certification board for testers, the test manager’s role and responsibilities include more than just supervising the test project. They manage the testing department or the test team, and thus, the resources for the tests. They prepare reports; escalate to development, technical department and project management; assess test projects; ensure compliance with the company’s quality processes; procure the testing tools for the organization; and review the test plans and the test cases.
The responsibilities can be allocated to two fields: strategic and operative (see Figure 2). The operative level includes the planning and conceptual design of the test cases and tests, monitoring the execution of the tests, and the communication within the project. The strategic level includes the quality management tasks.
Figure 2: Responsibilities of the test manager (according to the ISTQB)
The operative responsibilities cannot be assumed by the scrum master or the product owner. The product owner does not get involved in the implementation, nor the scrum master in the development. In agile development, the testing tasks are assumed by the team. Following the definition and allocation of the test manager’s responsibilities, we can examine how they can be integrated into the agile process. However, there is a problem: It is not possible to allocate them to a specific person because within the scrum framework, all tasks are distributed across the agile team.
The solution to the problem lies within the scrum framework itself. It provides a comprehensive package of tools and artifacts that can be matched with the test manager’s responsibilities. We did a complete agile transitionof the responsibilities in our scrum teams, and we found that scrum provides a tool or artifact for every one of the test manager’s responsibilities.
Figure 3: Agile transition of the test manager – test coordination
For example, the test strategy and the paramount quality characteristics are considered during the planning of the sprint and the backlog grooming. The pass-fail criteria, i.e. the criteria that determine whether a sprint has been successful or the test has been completed, are defined in the definition of done (see Figure 3).
Figure 4: Agile transition of the test manager – test implementation
In the sprint review, the implementation of the specifications is verified and validated (see Figure 4).
Figure 5: Agile transition of the test manager – test coordination
In addition, the stories and their representation on the scrum board and in the backlogs facilitate documentation of the progress and the assessment of the quality level (see Figure 3).
Accordingly, there is an agile tool or artifact for each of the test manager’s responsibilities. And thus, complete agile transition of the test manager’s responsibilities is achieved. Provided that the scrum team has the testing know-how required for the implementation of the upcoming development and testing tasks, small projects do not require a test manager. If the know-how is not yet given, the team has to be coached accordingly.
In the next part, we will examine who can assume the responsibilities on the strategic level, and what happens in projects where several scrum teams work on the same product together.
Using agile methods is not a 100% guarantee of better software. This is due to the fact that the agile context constitutes a major challenge for many traditional methods of quality assurance. In a system that frequently changes, how can we permanently ensure that the components that are deemed functional now do not fall back into an undefined state due to future improvements?
While the waterfall model provides for testing only at the end of the project, and the frequently used V-model (see Figure 1) also has a clearly defined chronological sequence, in an agile environment, it is crucial that tests are, if possible, always done under the exact same conditions and with the least possible effort because of the frequency at which they have to be done. Furthermore, they should be ready as soon as possible after the implementation of the function to be tested. This is the only way to ensure that they can keep up with the continuous changes.
Figure 1: The V-model and test-driven development
Test-driven Development
The biggest problem with traditional testing in this context is the fact that it is usually done downstream. It is only done when there is something to test. This seems to be trivial at first glance because you would think that if you want to test something, you need a corresponding test object.
Even though the test cases and the relevant data can be derived from the specifications, without an actual test object, the execution seemingly does not produce an analyzable result. However, this dogma does not apply for automated tests. Whereas continual manual testing would be far too expensive in the long run, automated tests can safely be written and executed shortly before the actual implementation, so that they temporarily test only a shell of the future test object. A shell like the one provided by an interface specification, for example.
In this case, the test is a kind of automated specification, the requirements of which are not fulfilled at first (red area in Figure 2). During the implementation stage, the developer can check their code against them, getting immediate feedback whether their work meets the requirements. Once this status is achieved (green area), the state of the source code is assured, and the developer can then restructure it (yellow area). In the case of new requirements or fundamental changes, the tests will fail again (red area again) and must first be passed correctly again (green area again).
More finely granulated structures then result from this general sequence of actions. They result in a very high test coverage that allows for errors to be found early on and simplifies the correction of such errors. Furthermore, interfaces and their implementation are defined more thoroughly than with traditional methods. This is what enables the high level of adaptability of the source code, which is essential for agile software projects.
Figure 2: Sequence of the test-driven method
In addition, one of the most important characteristics of automated tests is their legibility. Basically, they can be understood to be not only a test of the functionality, but also a kind of documentation of the latter. Because of how close they are to the functionality they test, and because of their clear structure, they describe the behavior of the code to be tested in a way that a normal written text would not be able to. And as their result is furthermore completely independent from the test object, they automatically tell the observer if they are no longer up to date.
Behavior-driven development
Component tests are clearly legible only for the group of people who understand the respective programming language and who know the test framework used. Outsiders, such as test managers, project managers or even end users, gain very little from the granularity and the level of detail.
Another weakness becomes evident when you consider the development process as a whole. Tests created with test-driven development are excellent for verifying parts of the software based on their specifications, but not for validating them against the acceptance criteria. But those are the criteria that need to be fulfilled, which is why they should come before the implementation in testing.
This is why the so-called behavior-driven or acceptance test-driven development evolved from the well-known test-driven development. With this type of development, the tests and specifications merge thanks to the use of certain tools and keywords, allowing the test results to be analyzed in plain language. For this purpose, we turn away from describing individual functions towards detailing the expected behavior. This way, the definition of the individual test cases no longer requires background knowledge of the program’s architecture, allowing even non-developers to understand it, and often, it even goes so far as to no longer speak of test cases, but of specifications.
Behavior-driven development and test-driven development are not to be understood to be rivals, however. The former primarily serves to validate the software and test the system integration, while the latter serves to verify and thus ensure the functional correctness. Consequently, it makes sense to combine the two.
Effort
Due to the fact that developers are expected to write the tests in addition to the actual product development, test-driven development involves greater effort than traditional methods at first. An adequate infrastructure needs to be established, the use of additional tools needs to be learned, and additional code that does not immediately result in usable features needs to be written. This may be discouraging at first, but in fact it only illustrates the effort that will otherwise be required much later over the course of the project. Instead of the effort involved in long testing and bug-fixing phases near the end of the project, a large part of the quality assurance is prepared at a time when the project team still has quite a lot of room to maneuver. The work required decreases as the project progresses because the infrastructure usually has to be established only once, and then only needs to be adjusted here and there.
Figure 3: When is TDD profitable?
Conclusion
Using the methods presented in this post constitutes a significant break with the traditional methods of software development in some ways, but it facilitates a kind of quality assurance and documentation that is difficult to achieve otherwise. Basically, the software itself reveals its current status without the need to maintain external resources that can quickly become obsolete. By combining the different approaches, you can ensure a high level of quality and a comprehensive understanding of the project across all levels of abstraction.
