When an external testing service provider and its client agree on the scope of the next testing project, they use an estimation of testing cost. In most cases, however, preparing an estimation of testing cost that is as accurate as possible and accepted by all parties involved is a challenge for two reasons: the timing of the preparation and the parameters defined for the specific assessment of the test scenarios or test cases. Most readers are probably wondering what the above headline has to do with this problem. It is the result of a realization based on my experience preparing estimations of testing cost in the past. Time and time again, I experienced the same fundamental discussions and amendments of the estimation of testing cost I prepared. I will return to this later.
First of all, I will describe the basis of an estimation of testing cost.
Basis of an estimation of testing cost
There are several ways for the testing service provider to bill the client for testing services. So far, I have used two of them in projects where I prepared estimations of testing cost: Firstly, on a T&M (time and material) basis, and secondly, based on the complexity of the test scenarios to be executed.
With the first option, the testing service is billed based on the time spent on the tests (worker days). For this purpose, we estimate the number of days or hours and the number of testers required for the execution of the tests.
Since this type of estimate of the test effort usually refers to a specified period of time, there is no allocation to specific scenarios or test cases. Consequently, the client usually accepts this estimate in its entirety and awards the contract accordingly.
The second option referred to in this blog post is billing on the basis of the test scenarios and the effort required by means of a complexity model. In a first step, we identify the necessary test scenarios based on the test activities to be performed. In a second step, we determine the corresponding test cases and allocate them to the respective scenario. Ideally, these scenarios represent a business process and therefore comprise several systems within the company. Such a process could, for example, cover the following systems:
- Account management software (in-house use by the client)
- Archiving (in-house use by the client)
- Accounting software (in-house use by the client)
- Device management (in-house use by the client)
- Client portal (external use by clients)
- Device usage (external use by clients)
- Additional tests in databases
Consequently, different business processes usually entail different levels of effort required for the execution of the tests. In addition to the trivial consideration of the effort of testing itself, we have to take into account that the overall effort not only comprises the execution of the tests as such, but the prerequisites to be fulfilled (e.g. certain data constellations) as well.
Furthermore, the test activities can increase if the client requires additional support during testing. Due to the coordination and the different availabilities of the company’s employees, the necessary amount of time can increase. All of these aspects directly affect the effort required for testing.
Determining the complexity
In order to determine the effort and calculate the price of a scenario for testing, we have to determine the level of complexity of the test tasks to be performed.
For this purpose, I used different methods in past projects.
With the first, “simple” option, I classified the complexity into three classes (small scenario, medium scenario or large scenario). But even there, there were exceptions for certain scenarios agreed with the client where the effort was rated higher because additional external service providers were involved in the tests.
In order to determine the complexity more accurately, I developed and introduced a more detailed version with a broader range of complexity categories. This enhanced the classification ranges from “1” for a very small scenario to “9” for a highly sophisticated scenario involving external service providers.
The following three segments within an individual scenario were defined to determine the category of the scenario:
- Effort for the test preparations
- Effort for testing
- Effort for the test specification
Each of these segments is then given points depending on the time required. The sum of the points in all three segments is converted by means of a conversion key, resulting in the complexity of the scenario.
Problems with the estimation of testing cost
We will now describe problems that may occur with the estimation of testing cost, or circumstances that often cause the client to start negotiating the categories of the scenarios after the estimate has been prepared.
From the client’s point of view, this is usually about the cost of the tests, which the client wants to keep as low as possible without reducing the scope of the tests.
Since the estimation of testing cost usually has to be prepared long before the tests are to be performed, precise specifications are not yet available in most cases. In the best case, first drafts of the technical specifications have already been prepared. However, they often do not yet include the entire system environment that is affected by the software adjustment. The missing documents have a negative effect on the following aspects.
With estimations of testing cost, the test effort of a scenario is determined based on previous test runs or, in the case of new topics, it is estimated and categorized accordingly. Since the effort in the individual segments (e.g. effort for testing) is estimated based on the time required, discussions about the estimated time spans and the corresponding category of a scenario are common. After reviewing the estimate, the client frequently believes that it is possible to perform the tests of a scenario in less time. Often, they will get their way, and the estimation of testing cost is amended – first successful haggling.
Such minor deviations can be compensated by experienced testers who know the systems involved like the back of their hand and can therefore perform the tests more quickly. Inexperienced testers who do not have the necessary expertise for one or several of the systems involved in the scenario will need more time, causing the first discrepancy between the estimated and the actual effort.
Another discrepancy arises if software is modified across several systems, causing the work required to meet the testing prerequisites to increase. This can, for example, happen if certain data constellations can no longer be readily generated in the legacy system, e.g. if write permissions on a database are eliminated. Since software adjustments are done at regular intervals (semi-annually) in this project, scenarios representing the same business process repeatedly emerge after some time. These previous estimates are used for comparison. Consequently, small additional efforts such as the additional work required to fulfill the testing prerequisites are allocated to a lower category as well. The estimation of testing cost is amended – second successful haggling.
Sometimes, the category a scenario has been allocated to in an estimation of testing cost may also be too low. This can happen, for example, when the software includes entirely new functions for which empirical values do not yet exist. Consequently, the scenario has to be allocated to a higher category in the next estimate. The client checks this change in category and, having little experience in this respect, uses the previous estimate as a basis. In this case, the estimation of testing cost is not amended – third successful haggling.
This kind of negotiation does not happen with every scenario in every cost estimate, but it happens every so often with individual scenarios in every estimate.
Conclusion
In my opinion, an estimation of testing cost based on time expenditure alone should only be done for internal use by the test service provider in order to determine the complexity. Estimating the effort required for the preparation, the testing, etc. based on the time can only work if the necessary parameters are quantifiable and defined by both parties. Discrepancies already arise when test cases are executed and recorded by different testers. The first tester records his results more accurately and with additional screenshots, while the second tester sufficiently records the results by copying-pasting them from the system. This alone can cause a discrepancy of up to +/- 30 minutes.
An estimation of testing cost requires the use of evaluation criteria. These criteria should be comprehensible and assessable for both the client and the test service provider. Furthermore, a certain basis of mutual trust is necessary in order to be able to prepare an estimate that is convincing and acceptable for both parties. While the estimate of some scenarios may be lower, others may be allocated to a higher category. This way, the estimation of testing cost as a whole will be well balanced.
Once you have found a common basis, working on an estimation of testing cost becomes much easier. When both parties are in agreement about the estimated effort, subsequent amendments – “haggling like at the bazaar” – are for the most part minimized.
This post was written by:
