The Custom Usability Index (CUI) in Action – Usability Tests in Agile Software Development

Starting point – The story so far…

In two previous blog posts, I presented the concept of the Custom Usability Index and described the preparation phase, where the usability categories are customized and weighted, in more detail. In this post, I would like to elaborate on the application itself, and explain the steps necessary to obtain a valid CUI evaluation in the end.

This is the starting point for our application example: We are developing a new web application for a business client. At the start of the project, the product owner in charge had the usability categories explained to them by a UX expert, and then defined the target states. Then, they prioritized the latter according to their requirements, laying the foundation for the regular rating of the future application using the CUI. After a certain period of development, the first release candidate has now been deployed to the QA system, ready for testing.

The Custom Usability Index (CUI) in action – Now your software is up to bat!

What do we do now? There are several different test methods for testing the 13 usability categories. The choice of the test method depends not only on its suitability for the respective usability category, but on numerous other factors as well. The availability of the end users, the know-how of the usability tester, and time and financial resources play an important role. In practice, the preferred scenario in which numerous end users are comprehensively tested by several usability testers with minute-keepers over an extended period of time, is rare.

What do we forfeit by restricting the resources? Basically, accuracy and independence of the evaluation. But this is not the primary objective of the CUI. The tests, irrespective of their scope, are primarily supposed to identify problems and potential for improvement. That means that, even if the result of test A deviates from test B, the problematic issues are clear, enabling concepts for developing software with better usability. A resource-friendly method of testing is the heuristic evaluation. Heuristic evaluation is an inspection method where a group of usability experts check an application for compliance with specific guidelines and heuristics (in this case, the usability categories of the CUI adapted to the concept of use). According to Nielsen, approx. seven testers will find 80% of all usability errors; follow-up tests are less productive because few additional errors are identified.

Nielsen (1995): Number of heuristic evaluations vs. usability problems found https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

In real-life agile software development projects, however, just getting several usability experts for the test is a rarity, which is why the evaluation is usually done by 1-2 experts. Depending on the complexity of the application and the scope of the documentation, a usability expert who is familiar with the application should be able to obtain a viable evaluation result within one to two days. This means that even with several “small-scale” usability tests, you can enhance the quality of your software and realize some improvements.

An example: Evaluation of 13 categories, 31 criteria “in a nutshell”.
Let us start with the category of effectiveness. The product owner defined the following target state for this category: “The user has access to the functions required for the completion of the tasks at all times, and reaches them in no more than two steps. In doing so, they are given no more than the information they need at the time (focus is to be on the data).”

Furthermore, effectiveness was weighted with the Fibonacci number 3.
The category of effectiveness comprises the criteria scope of functions and adequacy. For the “scope of functions” criterion, the different use cases of the web application have to be tested, and in addition, the tester has to check whether the required functions can be reached in no more than two steps.

The current release candidate fully meets the target state for this criterion, and consequently, it is awarded the maximum rating of five points.
Now for the second criterion of adequacy, which requires the user to be given no more than the information they need at the time, and the focus to be on the data.

Several minor anomalies cause a 2-point deduction, resulting in a rating of 3 points. Together with the rating of the scope of functions criterion, the average rating for the category of effectiveness is 4 points.
The other 12 categories and 29 criteria should be rated the same way as shown in the above example. As the complete documentation of an entire CUI test would go beyond the scope of this blog post, we are going to skip to the next step and take a closer look at the final calculation.

Example of a complete CUI calculation

The table above shows a completed CUI calculation matrix after the completion of the test. The final CUI value is calculated as follows:

  • Calculation of the average value of a category (sum of the ratings / number of criteria)
  • Calculation of the relative weighting (weighting value of the category / sum of all weighting values)
  • Calculation of the final category value (average value of the category * relative weighting)
  • CUI value = sum of the final category values
  • CUI degree of achievement = ((CUI value – 1) / 4)

Let’s run this calculation for our example:

  • Rating of the effectiveness category (scope of functions 5 points + adequacy 3 points) / 2 criteria = 4 points
  • Sum of the weightings of all categories = 24
  • Relative weighting of the effectiveness category = 3/24 = 0.125
  • Single value of the effectiveness category = rating of 4 * relative weighting 0.125 = 0.50

Hooray, it’s done! – Now what?

First of all: Congratulations for taking care of your users’ needs. Now, the following steps need to be taken:

  • Develop possible solutions for the identified problems in the form of wireframes and prototypes
  • Illustrate the effect of future user stories with respect to the different usability categories
  • Question the specifications of your product owner… You are the expert!
  • Actively present alternatives instead of unquestioningly implementing the specifications you are given
  • Carry out further CUI tests at regular intervals, thus tracking the effect of the improvements and/or new specifications on the usability of your software

The Custom Usability Index allows you to obtain useful results with few resources. The lightweight method is smoothly integrated into the agile software development process and identifies flaws in your software. Regular tests enable long-term tracking and make usability truly comprehensible. Take the leap—your users will appreciate it!

Brief insight into usability tests in agile software development!

Protractor – Automated Testing with Angular

Critical errors that become evident only in live operation constitute a major financial risk, and they also mean negative publicity both for the product and the companies involved. This is why testing is a fundamental, integral part of modern software development. High test coverage and prompt feedback of the test results allow for the quality and maturity of the product to be appropriately documented and confirmed.

Using test automation tools constitutes a solution that enables quick execution of such tests and meets the requirements of modern development projects. These tools work according to the principle of tool-based collection of information via the graphic interface of the product to be tested, which enables the automated execution of scripted interactions, and, as a result, the assessment of the respective application.

Test automation tools ensure quick and continuous feedback regarding the quality of the software to be tested. But there are some points that have to be observed when using them. There are various tools available on the market which use different approaches as to how they are integrated into the development and testing process, or which technologies they support. The efficient use of a test automation solution depends primarily on the engine used to control the graphic interface. This engine has to optimally support the technology to be tested. Development projects using “new” technologies such as Angular2 in particular face the problem that available and familiar tools are not always the same state of the art as the technology on which they are used.

CLINTR project and testing with Protractor

We use Angular2 as the development framework for our current software development project, Clintr, and we wanted a high frequency of automated test cases from the start. Clintr is a web application that alerts service providers to prospective customers in their contact network. For this purpose, it uses and analyzes data of the provided XING API to derive a demand for services in companies according to defined criteria in a fully automated manner. If a demand for services in a company has been identified, Clintr searches the service provider’s network of contacts (e.g. XING or CRM systems) for contact paths to the prospective customer. Spring Boot-based micro-services with Kubernetes are used as container cluster manager in the back-end, while Angular (>2) is used in the front-end. In order to be able to release new versions of the application at a high frequency, a continuous delivery pipeline into the Google cloud was established, both for the test and the production environment.

Because we used Angular2, we chose the automation test tool Protractor. Protractor is based on Selenium and the WebDriver framework. As usual, the interface tests are done in the browser, simulating the behavior of a user using the application. Since Protractor was written directly for Angular, it can access all the Angular elements without limitation. Furthermore, additional functions for waiting for components such as “sleeps” or “waits” are not necessary because Protractor recognizes the state the components are in and whether they are available for the intended interaction.

How to

For the execution, you need AngularCLI and NodeJS. Then the interface tests (end-to-end or e2e) can be created in the project. To prepare for the local test run, you use the console to switch to the project directory and enter “ng serve”. After entering “ng e2e”, the test cases are then run on the localhost.

The end-to-end tests consist of type script files with the extension .e2e-spec.ts, .po.ts, or just .ts. The test cases are described in the .e2e-spec.ts files. Only tests contained in these files are executed. The following example shows the header of a .e2e-spec.ts file:

    import { browser, by, ElementFinder } from 'protractor';
    import { ResultPage } from './result-list.po';
    import { CommonTabActions } from './common-tab-actions';
    import { SearchPage } from './search.po';
    import { AppPage } from './app.po';
    import { CardPageObject } from './card.po';
    import * as webdriver from 'selenium-webdriver';
    import ModulePromise = webdriver.promise;
    import Promise = webdriver.promise.Promise;
     
    describe('Result list', function () {
     
     let app: AppPage;
     let result: ResultPage;
     let common: CommonTabActions;
     let search: SearchPage;
     
     beforeEach(() => {
     app = new AppPage();
     result = new ResultPage();
     common = new CommonTabActions();
     search = new SearchPage();
     result.navigateTo();
     });

Like the other file types, it starts with the imports. Then the test cases start with describe. The string in brackets specifies which area is to be tested. Below that, the individual .po.ts files required for the subsequent tests are created and instantiated. The beforeEach function allows for preconditions for the test to be defined. For the purpose of reusability, the tests can also be exported to modules (see the following code example):

    it('should display the correct background-image when accessing the page', require('./background'));
    it('should send me to the impressum page', require('./impressum'));
    it('should send me to the privacy-policy page', require('./privacy-policy'));
     
    it('should open the search page after clicking clintr logo', require('./logo'));

The following code lists common e2e tests. First, it specifies what is expected, and then the test is executed. You should keep in mind that the e2e tests in the .e2e-spec.ts only calls the methods of the .po.ts and then waits for the result to be returned. The executing methods belong in the .po.ts.

    it('should still show the elements of the searchbar', () => {
     expect(result.isSearchFieldDisplayed()).toBe(true);
     expect(result.isSearchButtonDisplayed()).toBe(true);
    });
     
    it('should show the correct Search Term', () => {
     expect(result.getSearchTerm()).toBe(result.searchTerm);
    });

The following code example shows the .po.ts relating to the .e2e-spec.ts above. Each .e2e-spec.ts does not necessarily have its own .po.ts or vice versa. A .po.ts can, for example, contain tab-related actions such as switch or close tabs. As long as a .e2e-spec.ts only uses methods from other .po.ts, it does not necessarily need its own .po.ts. As mentioned above, the .po.ts starts with the imports, and then the class (ResultPage in the example) is created.

When called up, the navigateTo method causes the test to navigate to the specified page. Since the test is not supposed to do this directly in the present case, it navigates to the Search page first. There, a search term is entered and the search is started. Thus, the test arrives at the result_list page where the tests are subsequently run.

    import { element, by, ElementFinder, browser } from 'protractor';
    import { SearchPage } from './search.po';
    import * as webdriver from 'selenium-webdriver';
    import { CardPageObject } from './card.po';
    import ModulePromise = webdriver.promise;
    import Promise = webdriver.promise.Promise;
     
    export class ResultPage {
     
     public searchTerm: string = 'test';
     
     search: SearchPage;
     
     navigateTo(): Promise<void> {
     this.search = new SearchPage();
     return this.search.navigateTo()
     .then(() => this.search.setTextInSearchField(this.searchTerm))
     .then(() => this.search.clickSearchButton());
     }

Each of the following three methods queries an element of the page. The first two tests have a Union Type return value. This means they can either return a boolean or a Promise<boolean>, i.e. either a Boolean or a promise of a Boolean. When using Promise as the return value, it should always be followed by a then; otherwise, asynchronous errors may occur.

    isSearchButtonDisplayed(): Promise<boolean> | boolean {
     return element(by.name('searchInputField')).isDisplayed();
    }
     
    isSearchFieldDisplayed(): Promise<boolean> | boolean {
     return element(by.name('searchButton')).isDisplayed();
    }
     
    getSearchTerm(): Promise<string> {
     return element(by.name('searchInputField')).getAttribute('value');
    }

Example

An implementation example of a test case in ClintR is the test of the link to the legal notice. First, it is supposed to click on the link. Then, the test is supposed to switch to the newly opened tab and confirm that the URL contains /legal-notice. Lastly, it is supposed to close the tab. This test was initially created only for the home page.

    it('should send me to the impressum page',() => {
     impressum.clickImpressumLink();
     common.switchToAnotherTab(1);
     expect(browser.getCurrentUrl()).toContain('/legal-notice');
     common.closeSelectedTab(1);
    })

As, according to the acceptance criteria, the legal notice has to be accessible from every subpage, the test was later adopted into all the other specs. To keep the code clear, it was decided to export this test into a module (impressum.ts).

    import { browser } from 'protractor';
    import { AppPage } from './app.po';
    import { CommonTabActions } from './common-tab-actions';
     
    module.exports = () => {
     let common: CommonTabActions = new CommonTabActions();
     new AppPage().clickImpressumLink().then(() => {
     common.switchToAnotherTab(1);
     expect(browser.getCurrentUrl()).toContain('/legal-notice');
     common.closeSelectedTab(1);
     });
    };

It is used in the e2e-spec.ts this way:

    it('should send me to the impressum page', require('./impressum'));

Particularities, notes & problems

Certain given functions can be written in each e2e-spec.ts, e.g. beforeEach, beforeAll or afterEach and afterAll. As the names suggest, the code contained in one of these functions is executed before or after each test or all the tests. In our example, each test should include its own page view. Accordingly, the navigateTo method can, for example, be written in the beforeEach function. afterEach can, for example, be used to close tabs that were opened during the tests.

Each test starts with the word it. If you add an x before this word, i.e. xit, this test will be skipped in the test run. But, contrary to a commented-out test, notice will be given that one or more tests have been skipped in the test run. If you write a test case with f, i.e. fit, only those tests starting with fit will be taken into account in the test run. This is expedient when you have a large number of test cases and only want to run some of them.

When working with Promise, which you get from some methods, you should keep in mind that incorrect handling can cause asynchronous errors. Many events, such as pressing a button or querying whether an element is displayed, return such a Promise. Even opening a page returns a Promise<void>. To avoid errors, each Promise that entails further actions such as pressing a button or outputting a resulting value should explicitly be followed by a then. For example:

    pressButton().then( () => {
     giveMeTheCreatedValue();
    });
    //If this value is again a Promise, which should trigger something, then the whole thing would look like this:
    pressButton().then( () => {
     giveMeTheCreatedValue().then( () => {
     doSomethingElse();
     });
    });
    // or slightly shorter
    pressButton()
     .then(giveMeTheCreatedValue)
     .then(doSomethingElse);

Further information on Promises is available here.

Conclusion

Protractor is highly suitable for the automation of interface tests in a software development project using Angular2. The documentation on the project side is very detailed and comprehensive. Using Selenium allows the tests to be easily integrated into the build process.

The Agile Test Manager – An Oxymoron? (Part 3)

In the first and second part, we successfully completed the agile transition of the test manager’s responsibilities, and we proved that small scrum teams no longer require a test manager. Several scrum teams working together on the same product pose another challenge. Can the previous observations made with respect to a small scrum project with a single team also be adopted for large projects?

Do we need a test manager for large scrum projects?

In many companies that use scrum, several teams work together on the development of the same product. There have already been some thoughts regarding scaled scrum on the company level and how it is possible to comply with agile principles even as the level of complexity increases. In this context, Boris Glober identifies two problematic areas: “compliance with the scaled scrum framework” and “scaling of the requirements process”. To solve this, he introduces additional roles: the company scrum master and the company product owner, who assist all the scrum masters and product owners of the project. Both roles coordinate with their counterparts in the individual scrum teams, and they manage the scrum tools on the company level, such as the company product backlog and the company scrum board.

Can this concept be adopted for testing as well, i.e. do we need something like a company test owner or a company QA master? The purpose of this new tool would be the coordination of and responsibility for the overall, integrative testing process, establishing the necessary common test structures, and filling the backlog with stories, tasks and incidents relating to QA.

Coordination meeting on QA issues
Figure 1: Coordination meeting on QA issues

For projects with a manageable number (2 to 8) of scrum teams, this kind of coordination can be done within the framework of the general coordination meeting, the scrum of scrums. If the coordination becomes more test-specific and time-consuming, it is advisable to establish a separate test meeting as required. Each team sends a team member with the necessary test know-how to this meeting. If the number of teams is higher, or if coordination between several projects is required with respect to QA issues, the guilds can again be used for this purpose. The guilds collect examples of best practices that they provide to all the projects, or they appoint coaches who bring agile testing methods to new projects. The guild masters agree on important decisions and moderate the scrum teams wherever it is necessary to define common rules and solutions.

We can conclude that agile companies no longer need a test manager, even for large projects. However, this is possible only by way of a complete agile transition of the test manager’s responsibilities because each of the test manager’s responsibilities—in particular with respect to communication and coordination—has to be matched with an agile tool.

The Agile Test Manager – An Oxymoron? (Part 2)

In the first part, we successfully completed the agile transition of the test manager’s operative responsibilities, and we proved that small scrum teams no longer require a test manager. But will this work for the test manager’s strategic responsibilities, too? And who assumes the test manager’s strategic responsibilities?

Who assumes the test manager’s strategic responsibilities in agile companies?

A test manager’s strategic area of responsibility includes quality management (QM), which, according to ISO 8402, comprises “all activities of the overall management function that determine the quality policy, objectives and responsibilities and implement them by means such as quality planning, quality control, quality assurance and quality improvement within the quality system”.

Figure 1: Responsibilities of the test manager (according to the ISTQB)

Irrespective of whether a company uses traditional or agile methods in development, the company’s management is responsible for the quality management. The scrum team cannot, should not and must not assume this responsibility. At first glance, nothing in the organization and the division of responsibilities changes. At second glance, however, a problem arises for agile companies. In the traditional world, the test manager used to act as the interface between the operative and the strategic level. They transmitted information between the management and the operative level, passed strategic specifications and methods on to the test teams, or forwarded developments and improvements to the management.

But how does information get transmitted from the strategic to the operative level and vice versa in an agile environment? The answer: The agile transition does not stop with the allocation of the test manager’s operative responsibilities to the scrum team. The solution for the agile transition of the “interface” between the two levels is similar to the agile transition of the test manager’s operative responsibilities. New concepts and tools are available for the current corporate communication tasks.

One of the new concepts is the so-called guild. Guilds (called competence teams in our company) are a tool that puts and keeps knowledge and information management on track. They are organized in a matrix structure apart from the normal company structure. The objective of the guilds is to pool the company’s know-how carriers, offer the staff a platform to exchange knowledge or carry out training programs, or agree general project decisions such as the development of test environments or rules for code quality. Depending on the company’s goals, the structure and composition of the guilds can differ: For example, guilds can be divided according to competence areas such as Java development, .NET development, test or process analysis. Or entire scrum teams can be aggregated in guilds, working together on a specific subject in the project, such as QA, database connection, GUI, or interfaces (see Figure 1).

Figure 2: Example of classification and structure of the guilds by competence area

The guilds work according to the following pattern: The primary role that each employee assumes in their daily work (e.g. developer, tester [QAlas], product owner and scrum master) is determined. With this role, they are allocated to a corresponding guild. Within the guild, the members exchange information relating to their areas of responsibility, carry out in-house training seminars, collect the available knowledge in portals, or discuss best practices in individual projects. A guild master is responsible for the moderation and coordination within the guild. In accordance with the motto “primus inter pares”, they do not have more rights than their colleagues in the guild, and they are elected from among the guild members. The guild master is the principal contact for the members of their own guild, the other guild masters and the management. This is necessary because the guilds engage in active interdisciplinary exchange, and they are also supposed to forward feedback from the operative level—such as new developments or technologies—to sales, or suggestions regarding training seminars to HR.

In the last part, we will look at what happens when several scrum teams work together on the same project and the effort required to coordinate them increases for the field of QA as well.

The Agile Test Manager – An Oxymoron? (Part 1)

Some time ago, a colleague of mine asked me whether we still needed a test manager in an agile development process like scrum. My first response was no because the Agile Manifesto and the scrum framework only know three roles: product owner, development team, and scrum master. Accordingly, the scrum team—i.e. the three scrum roles mentioned above—does not provide for a test manager. But on second thought, I wondered who among the scrum team was supposed to assume the test manager’s responsibilities in and around the sprint?

Studies such as the 2014 ASQF Industry Report[1] and the 2011 Standish Chaos Report show that agile methods have already become a permanent fixture in companies. Furthermore, the Standish Chaos Report shows that projects using agile processes are more likely to be successful than “conventional projects”.  The Agile Manifesto of Ken Schwaber and Jeff Sutherland was the basis for this development. It defines basic principles and specifications that uncover “better ways of developing software by [the parties involved in the process] doing it and helping others do it” [²].

Scrum process and parties involved
Figure 1: Scrum process and parties involved

The key principles from the Agile Manifesto are:

  • Individuals and interactions are more important than processes and tools.
  • Working software is more important than comprehensive documentation.
  • Client collaboration is more important than the original specifications.
  • Responding to change is more important than following a plan.

Companies that switch to the agile method of development have a competitive advantage compared to companies using traditional methods. However, transitioning the processes to development methods such as scrum—also called agile transition—presents a great challenge. Agility is not achieved by divvying the development milestones up into sprints and appointing a product owner (see Figure 1). Comprehensive changes in the organization are required to achieve an agile way of working and living.

The challenge of the agile transition becomes very clear when you look at the example of the test manager. The question:  If we do not need a test manager in scrum, who assumes their responsibilities? The product owner? The scrum master? The team?

According to the Scrum Alliance, the product owner is the person responsible for developing the product accurately and on time. The product owner fills and refines the product backlog, and ensures that everybody knows what it contains and what is given which priority. Consequently, they are usually closest to the “business side” of the project. Scrum requires the development team to be a group with mixed functions—pooling all the necessary skills—that are required for the development of the product. The team organizes itself, i.e. it independently chooses the content to be implemented in the sprint and takes care of the planning, management, and implementation. The scrum master is the “pilot” guiding them through the depths of the scrum framework, helping the rest of the scrum team to comply with scrum principles. Another task of the scrum master is removing obstacles that hamper the team’s progress

Even after studying the scrum roles, we do not know who assumes the test manager’s responsibilities, or how they are allocated. To answer these questions, we must first determine the test manager’s responsibilities in the traditional testing and quality assurance process. According to the International Software Testing Qualifications Board (ISTQB), the certification board for testers, the test manager’s role and responsibilities include more than just supervising the test project. They manage the testing department or the test team, and thus, the resources for the tests. They prepare reports; escalate to development, technical department and project management; assess test projects; ensure compliance with the company’s quality processes; procure the testing tools for the organization; and review the test plans and the test cases.

The responsibilities can be allocated to two fields: strategic and operative (see Figure 2). The operative level includes the planning and conceptual design of the test cases and tests, monitoring the execution of the tests, and the communication within the project. The strategic level includes the quality management tasks.

Responsibilities of the test manager (according to the ISTQB)
Figure 2: Responsibilities of the test manager (according to the ISTQB)

The operative responsibilities cannot be assumed by the scrum master or the product owner. The product owner does not get involved in the implementation, nor the scrum master in the development. In agile development, the testing tasks are assumed by the team. Following the definition and allocation of the test manager’s responsibilities, we can examine how they can be integrated into the agile process. However, there is a problem: It is not possible to allocate them to a specific person because within the scrum framework, all tasks are distributed across the agile team.

The solution to the problem lies within the scrum framework itself. It provides a comprehensive package of tools and artifacts that can be matched with the test manager’s responsibilities. We did a complete agile transition of the responsibilities in our scrum teams, and we found that scrum provides a tool or artifact for every one of the test manager’s responsibilities.

Agile transition of the test manager – test coordination
Figure 3: Agile transition of the test manager – test coordination

For example, the test strategy and the paramount quality characteristics are considered during the planning of the sprint and the backlog grooming. The pass-fail criteria, i.e. the criteria that determine whether a sprint has been successful or the test has been completed, are defined in the definition of done (see Figure 3).

Agile transition of the test manager – test implementation
Figure 4: Agile transition of the test manager – test implementation

In the sprint review, the implementation of the specifications is verified and validated (see Figure 4).

Agile transition of the test manager – test coordination
Figure 5: Agile transition of the test manager – test coordination

In addition, the stories and their representation on the scrum board and in the backlogs facilitate documentation of the progress and the assessment of the quality level (see Figure 3).

Accordingly, there is an agile tool or artifact for each of the test manager’s responsibilities. And thus, complete agile transition of the test manager’s responsibilities is achieved. Provided that the scrum team has the testing know-how required for the implementation of the upcoming development and testing tasks, small projects do not require a test manager. If the know-how is not yet given, the team has to be coached accordingly.

In the next part, we will examine who can assume the responsibilities on the strategic level, and what happens in projects where several scrum teams work on the same product together.