Exploratory Testing (Part 1) – The Test for Lazy People or the Crown of Creation in Terms of Testing?

We are testers and love test methods, where the focus is on human strengths like creativity and collaboration. Nevertheless, we do not want to act without rules, or rather without framework conditions, or plead for an abstinence of documentation.

We have already encountered some prejudices about exploratory testing, therefore we have asked ourselves the question how these come about. In this article we will talk about those prejudices in more detail in order to get rid of them. But first we will take a closer look at exploratory testing.

What is exploratory testing?

Exploratory testing can do much more than just working off test cases in a monotonous way: It promotes creative thinking during a test session, blasts limitations and satisfies curiosity. At the same time, it requires discipline regarding reaching the set goal during a test implementation and with regard to documentation.

ISTQB (independent panel of experts for standardized training for software testers) describes exploratory testing in its glossary as a „an approach to testing whereby the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests“ (archived version 3.4 of the ISTQB® GTB Glossary). Unlike the “classic“ testing with prepared test cases, it is therefore an intuitive test approach where each previous idea leads to the next one. There are various tools to achieve a new starting point, to develop ideas and create more structure. These include for example the test charter, the heuristics, the test oracle or also the test tours.

In summary, the tester creates the tests through experience, intuition, curiosity and structured approach. Nevertheless, this type of testing is not only well-suited for experienced testers but also for project newcomers who are just getting to know the test object and/or the procedure. To take the person by the hand a goal should be defined or an overview of the object should be created. This can be done either in direct exchange via pair testing or in groups via the mob-testing approach.

Prejudices against exploratory testing

But what prejudices are there now against which exploratory testing must stand its ground?

Exploratory testing is often notorious for having no structure; for not being comprehensible; and being unquantifiable – plain and simple “I-will-click-around-and-see-what-happens”. The statement of obsolete documentation further complicates it to charge for exploratory testing in client projects. This is an understandable reason, if it is assumed that exploratory testing is documentation-free and thus the test performance is not traceable. The following statement and the killer argument are therefore also made quickly that there is no time available for this test.

In the following, we will take a closer look at the allegations just presented as well as other negative statements and comment on them.

Prejudice Number 1: “Exploratory testing is not comprehensible.” / “I cannot test exploratively-tested content again.” / “How am I supposed to do a bug retest?” These statements made us curious. When asked what is meant by exploratory testing, we often received the answer that it is used for spontaneous clicking around in the test object. It should promote getting to know the test object or test further variants that were not considered in the previous test-case creation. And thus, it is carried out in an unplanned as well as unstructured way. No documentation was mentioned. This explains how the prejudice comes about that this test method is not comprehensible, and that tests and bug fixes are difficult to retest. It is absolutely true that exploratory tests support spontaneous test taking. However, care must be taken to always document the tests steps, expected results, or even deviations, and thus make the tests traceable. The depth of documentation must be defined individually for each project according to its framework conditions. As a conclusion we maintain that: Exploratory testing does not mean not to document.

To counteract the first prejudice and to bring structure into exploratory testing, session-based test management can be used. During a session, the test object is checked with maximum flexibility and freedom. The framework conditions are set by means of a defined standpoint and a charter (purpose and procedure for a session). In addition to some other components, this also includes documentation by means of session reports in which the test steps carried out, deviations, the test object, and other information can be recorded. These can be adjusted depending on the person or project being tested. Various examples of session reports can be found on the Internet.

At this point, a special application should be mentioned: Exploratory testing in a regulated environment. Documentation plays a central role here. The prejudice here: Exploratory testing is impossible or very difficult to implement in a regulated environment due to regulatory requirements. In a further article, we will show how to implement it in spite of the requirements.

Prejudice Number 2: “We have no time for exploratory testing” / “First of all, we have to work through our prepared test cases before we can test exploratively.“

In order to implement the exploratory test method actively in the project, one must be prepared to adapt the previous test procedure and recognize the advantage of saving time. Because if you see the exploratory test only as a supplement, without an alignment of the test structure used up to now, this procedure is seen only as a burden or as an annoying addition, which results in the no-time prejudice. However, if we proceed as envisaged in exploratory testing, and if the test case creation takes place at the same time as the test case execution, the now superfluous creation of test cases that are not executed at the end, and thus represent a waste of time, is eliminated. The time gained allows the tester to develop new test ideas and thus to test the test object more intensively. For example, debriefings can be held to share lessons learned with other team members. This time saving can also be used to increase the test automation rate.

Prejudice Number 3: “Documentation is all well and good, but exploratory testing is not sufficient.”:
This statement is correct, if it is meant that exploratory testing should be considered as a complete test approach in its own right. In addition to the exploratory tests, at least automated tests should be defined and executed (for example unit tests, integration tests and GUI tests). If the project has a high degree of automation, it is even possible to perform the manual testing purely exploratively. Alternatively, the exploratory method can be used as a complement to the classical testing approach to make manual testing easier, more flexible and freer. How much exploratory testing is applied in a project must always be carefully examined. It is not always possible to make all manual tests exploratory. This can be due to various factors, e. g. an insufficient degree of automation, or a lack of skills on the part of the testers. In our view, they should have skills such as creativity, conscientiousness, initiative, personal responsibility, an understanding of quality, and social aspects such as communication and teamwork skills.

Conclusion

In this article, we have shown the advantages of exploratory testing and put together arguments to refute the existing prejudices against it. From our point of view, it is worthwhile to rethink the previous test procedure and to engage in exploratory testing in order to exploit its positive effects. Of course, each project should be considered on its own merits and the possibilities for exploratory testing should be determined – without, however, ruling this out completely from the outset. Negative statements or experiences should be questioned where appropriate. When choosing the method and its execution, the skills of the testers must always be considered. At this point we would like to point out that documentation is also essential for exploratory testing.

This post was written by:

This post was written by: