Tester Teatime (Part 2): Spelling in applications – anything but trivial

“Tester Teatime” is a blog post format, which addresses topics that testers deal with on a daily basis. As certain issues or topics tend to recur again and again, the aim here is to create a basis for explaining such phenomena and finding solutions for them. To that end, the blog focusses on stimulating discussions and new ways of thinking. In testing, we can learn a lot from each other by observing our behaviour in our daily lives!

Moderator: Welcome to Tester Teatime! In this interview with testers from ZEISS Digital Innovation (ZDI), we will once again discuss exciting topics.

Today, we are talking to Sandra Wolf (SW), a tester at ZDI, about “spelling”. Sandra, why are we discussing this topic and what’s the connection with software development?

SW: A tester’s normal workday is full of challenges. During the testing process in particular, full concentration is required as every detail has to be checked for quality. One of these details is spelling and grammar. The importance of correct spelling is often underestimated.
In our daily work, it is not unusual to find spelling mistakes in the software. But the tester is often ridiculed when they report these to Development for correction. According to the prevailing view, these are only minor and insignificant errors. Today’s talk aims to dispel this opinion. Spelling and punctuation are not exactly the most popular topics and are often perceived as being very dry. Yet these very rules, which we have been learning since our school days, act as a guide for us and our brains. Spelled correctly, a word is easier to read, easier to combine in a sentence to form a statement and thus easier to process for the brain. Attentive readers – or in the case of software development – users will inevitably stumble across incorrect spelling in the software. It has even been demonstrated that certain personality types react differently to incorrect spelling in terms of their emotional response (cf. Weiß, 2016). Thus, contrary to their reputation as being dry, errors in this area can trigger emotions, which in turn affect use of the software.

two people having an interview
Image: Stanislaw Traktovenko and Sandra Wolf during the interview at the Tester Teatime.

Moderator: What kind of influence are we talking about here?

SW: For instance, correct spelling radiates respectability. In job applications and official requests, error-free spelling is an absolute must. For example, studies have even shown that a loan application is less likely to be approved if there are linguistic errors in it (cf. Weiß, 2016). If we now apply this to the software we develop, only one possible conclusion can be drawn: Spelling is essential for the user experience and external appearance of the software. And so this topic should clearly be taken more seriously in the development process and receive more attention than it has previously.
If we look at the common workday of testers and developers, we know that their focus is on the functionality of the software. Of course, it is understandable that a seemingly cosmetic issue like spelling takes a back seat to elaborately programmed application parts. However, this should not fool all those involved in the process as to its importance. For it is quite clear that the success of a product, and thus also of an application, can be affected by the linguistic quality. First impressions count: When reading a text or using a software, we automatically judge the level of education of the creators based on these (cf. Frost, 2020). Incorrect spelling can therefore cast a bad light on good software.

Moderator: Can you give me a detailed example of this?

SW: Poor spelling can lead to less confidence in the quality of the software and a resulting decline in acceptance of the application. The user might presume that little value is placed on quality in general if even the spelling is handled carelessly. After all, correct spelling expresses not only professionalism but also a certain respect for the reader/user. It has even been found that the quality of a text can affect whether a prospective buyer chooses to make the purchase. Placing this in the context of software development, it can definitely save money if attention is paid to spelling from the onset and reports of such errors are taken seriously (cf. Frost, 2020).
Ultimately, we also present our company in our projects, which is why the issue of spelling can have more far-reaching effects than we may initially think. In the best case, good spelling can improve or maintain the reputation of our software development. This in turn can lead to more customers and higher sales because the consistent quality of our software products can be an argument in favour of working with ZDI.

Moderator: I would like to build on this and let Stanislaw Traktovenko (ST) from our usability team have his say. What is the importance of spelling from your point of view? Do you also see the impact to the extent described by Sandra?

ST: The way I see it, spelling has an impact on the readability and therefore on the perception of the information in the application. We assign this to the usability principles of consistency and language. Spelling errors therefore potentially have a direct impact on the usability of an application. For instance, incorrect spelling disturbs the flow of reading and thus the way the user perceives the software. It creates a negative sentiment and the user no longer engages in the task they were actually pursuing with the software. The user is distracted by the incorrect spelling and this affects the user’s effectiveness and efficiency. Even though spelling is only a small part of usability, it can have a bigger impact than we think, as Sandra explained earlier.

Moderator: Thank you Sandra and Stanislaw for these interesting insights. The impact is indeed more far-reaching than expected, which is quite amazing. We can thus summarise that the seemingly dry topic of spelling must be taken seriously in all software projects in order to deliver the highest possible quality and to adequately present both the products and our company. Though the topic of spelling may seem trivial at first, ultimately it has a major effect and is therefore important for all of us. The topic should therefore definitely receive the attention it deserves.

In the following articles, we will address other issues from the daily lives of testers and discuss possible solutions for these.

GraalVM – A look at performance

GraalVM has now been on the market for over two years. It promises two main benefits: better runtime properties and the integration of multiple programming languages.

This blog post will focus on performance; it will not look primarily at whether and to what extent a particular program is executed faster on the Graal JDK than on a conventional JDK. After all, the specific measured values and the relative comparisons are not solely dependent on the program being tested, and they have little general applicability. Moreover, they are merely snapshots: Both GraalVM and, to take one example, OpenJDK, are undergoing continuous development, meaning that the measured values will be continuously changing too. This blog post will instead look at mainly the following questions: Why should GraalVM have a much better performance? What makes it different from conventional JDKs? This will allow us to evaluate whether all programs are executed with a better performance or if no appreciable improvement can be expected, or whether the performance increase is only to be expected in certain application scenarios. And ultimately, whether this means “conventional” Java is too slow.

Laptop connected with LED Display, which shows "Hello World".

The development of compilers

The performance of a Java program is fundamentally determined by the compiler, and in this case too, our key focus is to establish what it is that makes GraalVM different. So first, let’s get an understanding of compilers.

In the early days of programming, there were no compilers at all – the machine code was programmed directly. This was confusing and difficult to understand, and it soon led to the development of an assembler code. However, in principle, this was a direct mapping of the machine code, the only difference being that alphabetic abbreviations were now used instead of binary or hexadecimal opcodes. We cannot speak of a programming language and compiler here, at least not within the scope of this blog post.

Over time, it became necessary to develop more and more complicated programs, and the assembly code became increasingly impracticable. For this reason, the first higher programming languages were developed in the 1950s. These needed a compiler to convert the source text into machine code.

The first of these was the classic AOT (ahead-of-time) compiler. The source text is analysed (syntax analysis) and transferred to an internal tree structure (syntax tree), which is used to generate machine code (code generation). This creates a binary file that can then be directly executed.

As an alternative to AOT compilation, a program can also be executed by an interpreter. In this case, the source text is read in and implemented by the interpreter line-by-line. The actual operations (e.g. addition, comparison, program output) are then executed by the interpreter.

Compared to the interpreter, the AOT compiler has the benefit that the programs are executed much faster. However, the generated binary files are system-dependent. What’s more, the interpreter has better error analysis capabilities, since it has access to runtime information, for example.

Java, interpreters and JIT compilers

When the Java programming language was being designed, one of the goals was to ensure that it was architecture-neutral and portable. For this reason, the Java source code was translated into platform-independent bytecode right from the outset. This could then be interpreted by a runtime environment or JRE (Java Runtime Environment). This meant the translated bytecode was platform-independent. For example, applets could be executed on a Windows PC, Mac or Unix workstation without any adaptations, as long as the JRE – regardless of the applet – was already installed on the workstations.

It is worth noting that this idea of a combined solution – AOT up to the bytecode, then interpretation at runtime – does not come from the world of Java. Pascal, for example, was using p-Code as long ago as the 1970s. [1]

When Java technology was released in 1995 [2], this platform independence initially went hand-in-hand with major drawbacks when it came to performance. Many of the programming languages that were relevant at the time, such as “C”, compile their source code directly in machine code (AOT). This can be run natively on the appropriate system, meaning performance is much better than with interpretation of the bytecode. At that time, many IT specialists had come to the conclusion that “Java is slow” – and they were right.

However, “high performance” became an additional goal in the process of designing the Java programming language, which is why the JIT (just-in-time) compiler was introduced in 1998 [2]. This significantly reduced the losses in performance due to pure interpretation.

In JIT compilation, the bytecode is initially interpreted at the program start, but the system continuously analyses the parts of the program to determine which are executed and how often. The frequently executed parts of the program are then translated into machine code at runtime. In future, these parts of the program will no longer be interpreted, and the native machine code will be executed instead. So in this case, execution time is initially “invested” for compilation purposes so that execution time can then be saved at each subsequent call.

JIT compilation therefore represents a middle ground between AOT compilation and interpretation. Platform independence is retained since the machine code is only generated at runtime. And as the frequently used parts of the program are executed as native machine code after a certain warm-up time, the performance is approximately as good as with AOT compilation. As a general rule, the more frequently individual parts of the program are executed, the more the other, interpreted parts of the program can be disregarded in the performance analysis. And this applies above all to frequently run loops or long-running server applications with methods that are called continuously.

Runtime optimisations

With the mechanisms we have looked at so far, the JIT compiler was platform-independent but could not achieve the execution time of the AOT compiler, let alone exceed it. At the time the JIT compiler was integrated into the JDK, it was by no means certain that this would be enough to see it triumph.

However, the JIT compiler does have one significant advantage over the AOT compiler: It is not solely reliant on the static source code analysis, and it can monitor the program directly at runtime. As the vast majority of programs behave differently depending on inputs and/or environment states, the JIT compiler can make optimisations with much greater precision at runtime.

One major benefit in this context is speculative optimisations, where assumptions are made that are true in most cases. To ensure that the program works correctly in all cases, the assumption is supported by a “guard”. For instance, the JVM assumes that the polymorphism is pretty much never used when a productive program is running. Polymorphism is obviously a good idea in general, but the practical application scenarios are usually limited to testing or to decoupling a codebase – and usually to enable use by various programs or for future expansion possibilities. Nonetheless, during the runtime of a specific productive program – and this is the scope of the JVM – the polymorphism is rarely used. The problem here is that, when calling an interface module, it takes a relatively long time for the existing object to find the appropriate method implementation, which is why the method calls are traced. If, for example, the method “java.util.List.add(…)” is called multiple times on an object of the type “java.util.ArrayList”, the JVM makes a note of this. For the subsequent method calls “List::add”, it is speculated that they are ArrayLists again. At first, the assumption is supported by a guard, and a check is made to determine whether the object is actually of the ArrayList type. This is usually the case, and the method that has already been determined multiple times is simply called directly using the “noted” reference.

Over two decades have passed since the JIT compiler was integrated into the JDK. During this time, a great many runtime optimisations have been integrated. The polymorphism speculation presented here is just one small example intended to illustrate the fact that a large number of optimisations have been devised – in addition to the compilation of machine code – that only work at runtime in a complex language like Java. If, for example, an instance is generated using reflection, it is difficult or even impossible for an AOT compiler to identify the specific type and implement speculative optimisation. The speed advantages of the current JIT compilers are therefore primarily based on the fact that they can monitor the program during execution, identify regular operations and ultimately integrate shortcuts.

GraalVM

GraalVM is a JDK from Oracle based on OpenJDK. It offers a virtual machine and a host of developer tools – but the same is also true of the other JDKs. So why is GraalVM generating much more attention that the other JDKs?

Firstly, GraalVM provides a GraalVM compiler, which was developed in Java. In addition, the entire JVM is to be rewritten in Java. In the last section, we showed that current JVMs offer high performance primarily because various optimisations have been included over the decades that are now adding up. These optimisations are mainly Java-specific and are mostly developed by people with a background in Java. If the execution environment is implemented in Java rather than C++, it is possible to make optimisations without any knowledge of C++. This puts the developer community on a broader footing over the medium and long term.

Another exciting aspect of GraalVM is that it supports more than just Java-based languages. The “Truffle language implementation framework” is a starting point for developing domain-specific languages (DSL). The GraalVM compiler supports languages developed with the Truffle framework, meaning these can also be executed in GraalVM and enjoy all the corresponding benefits. Certain languages, such as JavaScript, Python or Ruby, are already supported by GraalVM as standard. Since all Truffle languages can be executed jointly and simultaneously in GraalVM, it is referred to as a polyglot VM.

LLVM-based languages are also supported. LLVM is a framework project for optimising compilers [4][5]. In the LLVM project, compiler components and technologies for external compiler developments are provided, and compilers for many programming languages, like C/C++ or Fortran, are offered too. LLVM runtime is another component of GraalVM that can be used to execute LLVM-based languages on the basis of the Truffle framework in GraalVM. However, we will not go into the polyglot aspect any further as it deserves its own blog post.

GraalVM native image

The innovation that is most relevant to this blog post is native image technology. Native image is a GraalVM developer tool that uses bytecode to generate an executable file. It aims to achieve better performance and reduced main memory usage at runtime. However, we have said that Java is getting faster, with the JIT compiler translating all commonly executed (i.e. relevant) parts of the program into native machine code. The program is monitored during execution, and runtime optimisations are continuously made. So this might lead us to ask: What exactly could be improved here by orders of magnitude?

The answer is incredibly simple: the start time. Even with JIT compilers, the bytecode is initially interpreted. Firstly, the program start is not usually a frequently executed part of the program. Secondly, these parts of the program usually need to run a couple of times first for the JIT compiler to recognise that they are a worthwhile translation target. With runtime optimisations, the behaviour is the same: The program first needs to be monitored at runtime so that the appropriate optimisations can be recognised and integrated. Another complicating factor is that all required objects and their classes, including the complete inheritance hierarchy, need to be initialised at start-up.

Since we now have an idea of “what”, we are interested in knowing “how”: How can our program be made to start faster?

When generating the native image, the bytecode initially undergoes highly extensive static analysis. Amongst other things, this checks which parts of the code can actually be executed at runtime. This includes not only the classes provided by the user but the entire classpath, with the Java class libraries provided by the JVM. Only the identified source code fragments are included in the native image, so the scope is significantly reduced at this stage. However, a “closed-world assumption” is also proposed: As soon as anything is loaded dynamically at runtime, the native image tool has a problem. It does not recognise that these source code parts can also be executed and are thus required. For this reason, anything more than a simple HelloWorld program would not work this way, so when creating the native image you can – and must – give the tool more information about anything that can be called dynamically.

Following the static analysis, the first element that increases the start-up speed is implemented: Since the JIT compiler would start with interpretation, an AOT compiler is used to create machine code. The generated native image is, as the name suggests, machine code that can be executed natively. However, this means that platform independence is lost.

In addition to the natively compiled program, the Substrate VM is included in the native image. This is a stripped-down VM containing only the components required to execute the native image, like thread scheduling or garbage collection. The Substrate VM has its own limitations, with no support provided for a security manager, for example.

An additional increase in the start-up speed is achieved by initialising the native image in advance during creation. Following compilation, the program is started until the key initialisations have completed but no external input needs to be processed. On the basis of this started state, a disk image is created and included in the native image.

We have looked at the “what” and the “how”, and now we turn to a rather critical “why”: The AOT compiler has been known for half a century, and Java has now existed for a quarter of a century. Particularly in early days of Java, various AOT approaches were tried but none ever became established. Why should it be different this time? Why is a reduced start time now of interest, when it goes hand-in-hand with certain disadvantages? Why are high-performance response times in operation over consecutive days or weeks suddenly less important?

The answer can be found in cloud computing, where the services are provided in a different form. Previously, the services were primarily operated in an application container executed day and night and in which the program had already been fully optimised for days. It was usually the case that the application container was not eventually shut down, even when used sparingly (e.g. depending on the time of day). By contrast, the service infrastructure in the cloud can be shut down without problems when not used, enabling capacity to be preserved. At the next call, the infrastructure is started up again and the call is executed. This means that the programs in the cloud may execute a cold start for each call rather than running continuously. As a consequence, the “all at once” start time is extremely crucial. And as it can be expected that even more Java programs will be executed in the cloud rather than in an application container in future, there is likely to be an increased focus on the start time.

Hands on: HelloWorld

After all of that theory, let’s take a look at the JDK in operation. First, we will use the HelloWorld class shown in listing 1.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
package de.zeiss.zdi.graal;
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Moin!");
}
}
package de.zeiss.zdi.graal; public class HelloWorld { public static void main(String[] args) { System.out.println("Moin!"); } }
package de.zeiss.zdi.graal;

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Moin!");
    }

}

Listing 1

Here is the classic variant: We are on a Linux VM and an OpenJDK is installed:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
> java --version
openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
java 11.0.12 2021-07-20 LTS
Java(TM) SE Runtime Environment GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08)
Java HotSpot(TM) 64-Bit Server VM GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08, mixed mode, sharing)
> java --version openjdk 11.0.11 2021-04-20 OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04) OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing) java 11.0.12 2021-07-20 LTS Java(TM) SE Runtime Environment GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08) Java HotSpot(TM) 64-Bit Server VM GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08, mixed mode, sharing)
> java --version

openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

java 11.0.12 2021-07-20 LTS
Java(TM) SE Runtime Environment GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08)
Java HotSpot(TM) 64-Bit Server VM GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08, mixed mode, sharing)

With this setup, we compile the HelloWorld class (javac) and execute the generated bytecode on a JVM:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
time java -cp target/classes de.zeiss.zdi.graal.HelloWorld
time java -cp target/classes de.zeiss.zdi.graal.HelloWorld
time java -cp target/classes de.zeiss.zdi.graal.HelloWorld

This gives us the following output:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
Moin!
real 0m0.055s
user 0m0.059s
sys 0m0.010s
Moin! real 0m0.055s user 0m0.059s sys 0m0.010s
Moin!

real    0m0.055s
user    0m0.059s
sys     0m0.010s

The total of the two lines “user” and “sys” is relevant to the evaluation here. This is the computing time required to execute the program – in this case, approx. 69 ms.

One note on the 55 ms: From the start to the end, the HelloWorld program required 55 ms of “real time” (the time perceived by the user), which is less than the 69 ms of computing time required. This is due to the Linux system having multiple processors. However, for the purposes of our measurements, we will analyse the computing time applied by the system. Firstly, the computing time is less dependent on the number of processors that have executed the program. And secondly, in the cloud, for example, this is the time that must be paid for by the application operator.

Now we are curious about GraalVM, which is available to download from its website [3]. The Enterprise version (“free for evaluation and development”) is suitable for our evaluation, as most of the performance optimisations are only found here.

The installation is very well documented for Linux and works with virtually no problems. GraalVM is then available for use as JDK.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
> java --version
java version "11.0.12" 2021-07-20 LTS
Java(TM) SE Runtime Environment GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08)
Java HotSpot(TM) 64-Bit Server VM GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08, mixed mode, sharing)
> java --version java version "11.0.12" 2021-07-20 LTS Java(TM) SE Runtime Environment GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08) Java HotSpot(TM) 64-Bit Server VM GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08, mixed mode, sharing)
> java --version

java version "11.0.12" 2021-07-20 LTS
Java(TM) SE Runtime Environment GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08)
Java HotSpot(TM) 64-Bit Server VM GraalVM EE 21.2.0.1 (build 11.0.12+8-LTS-jvmci-21.2-b08, mixed mode, sharing)

We can now compile and execute our HelloWorld program in the same manner with the GraalJDK (javac). This gives us the following output:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
Moin!
real 0m0.084s
user 0m0.099s
sys 0m0.017s
Moin! real 0m0.084s user 0m0.099s sys 0m0.017s
Moin!

real    0m0.084s
user    0m0.099s
sys     0m0.017s

Interestingly, the JVM of the GraalJDK needs almost 70% more computing time to execute our HelloWorld example as bytecode. However, the significant performance benefits promised by GraalVM primarily relate to the use of native image technology, rather than to the execution of bytecode.

The native image (the developer tool) is not contained in the downloaded GraalVM, but the command-line tool “gu” (GraalVM Updater) exists for this purpose. This enables us to load, manage and update additional components. In this case too, we find very good support in the GraalVM documentation. Once the developer tool is loaded, we can now generate the native image from the bytecode. With such a trivial program as our HelloWorld example, a single command with the fully qualified class name as the argument is sufficient:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cd ~/dev/prj/graal-eval/target/classes
native-image de.zeiss.zdi.graal.HelloWorld
cd ~/dev/prj/graal-eval/target/classes native-image de.zeiss.zdi.graal.HelloWorld
cd ~/dev/prj/graal-eval/target/classes
native-image de.zeiss.zdi.graal.HelloWorld

Creating the HelloWorld native image requires a good three minutes of computing time, and the executable program is approx. 12 MB in size. At first glance, we might compare the size with the bytecode: HelloWorld.class is only 565 bytes. However, the native image contains not only the compiled class but also all relevant parts of the Java class library and the Substrate VM. As a rough estimate, the native image is only 10% of the size of a JRE.

But let’s return to our native image, which we have now managed to create. We can then execute it and obtain the following output.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
time ./de.zeiss.zdi.graal.helloworld
time ./de.zeiss.zdi.graal.helloworld
time ./de.zeiss.zdi.graal.helloworld
Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
Moin!
real 0m0.004s
user 0m0.003s
sys 0m0.001s
Moin! real 0m0.004s user 0m0.003s sys 0m0.001s
Moin!

real    0m0.004s
user    0m0.003s
sys     0m0.001s

For now, we can consider this result to be a relevant speed gain.

Hands on: HelloScript

One of the features of GraalVM that is highlighted again and again is that it is a polyglot VM, rather than just a Java VM. For this reason, we will expand our HelloWorld program to include a short digression into the world of JavaScript. The relevant source code is shown in listing 2. The key difference here is the transition required from the world of Java to the world of JavaScript.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
package de.zeiss.zdi.graal;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class HelloScriptEngine {
public static void main(String[] args) throws ScriptException {
ScriptEngine jsEngine = new ScriptEngineManager().getEngineByName("javascript");
System.out.print("Hello ");
jsEngine.eval("print('JavaScript!')");
}
}
package de.zeiss.zdi.graal; import javax.script.ScriptEngine; import javax.script.ScriptEngineManager; import javax.script.ScriptException; public class HelloScriptEngine { public static void main(String[] args) throws ScriptException { ScriptEngine jsEngine = new ScriptEngineManager().getEngineByName("javascript"); System.out.print("Hello "); jsEngine.eval("print('JavaScript!')"); } }
package de.zeiss.zdi.graal;

import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;

public class HelloScriptEngine {

    public static void main(String[] args) throws ScriptException {

        ScriptEngine jsEngine = new ScriptEngineManager().getEngineByName("javascript");

        System.out.print("Hello ");
        jsEngine.eval("print('JavaScript!')");
    }

}

Listing 2

Alongside this universal JavaScript connection via javax.script.ScriptEngine, we also want to try out the Graal-specific JavaScript connection using org.graalvm.polyglot.Context. The source text is shown in listing 3.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
package de.zeiss.zdi.graal;
import org.graalvm.polyglot.Context;
public class HelloScriptPolyglot {
public static void main(String[] args) {
System.out.print("Hello ");
try (Context context = Context.create()) {
context.eval("js", "print('JavaScript!')");
}
}
}
package de.zeiss.zdi.graal; import org.graalvm.polyglot.Context; public class HelloScriptPolyglot { public static void main(String[] args) { System.out.print("Hello "); try (Context context = Context.create()) { context.eval("js", "print('JavaScript!')"); } } }
package de.zeiss.zdi.graal;

import org.graalvm.polyglot.Context;

public class HelloScriptPolyglot {

    public static void main(String[] args) {

        System.out.print("Hello ");
        try (Context context = Context.create()) {
            context.eval("js", "print('JavaScript!')");
        }
    }

}

Listing 3

The two HelloScript programs are translated into bytecode in the same way as the HelloWorld program. When the native images are created, the developer tool must be informed that the world of JavaScript will be used. This is done with the following call:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
cd ~/dev/prj/graal-eval/target/classes
native-image --language:js de.zeiss.zdi.graal.HelloScriptEngine
native-image --language:js de.zeiss.zdi.graal.HelloScriptPolyglot
cd ~/dev/prj/graal-eval/target/classes native-image --language:js de.zeiss.zdi.graal.HelloScriptEngine native-image --language:js de.zeiss.zdi.graal.HelloScriptPolyglot
cd ~/dev/prj/graal-eval/target/classes
native-image --language:js de.zeiss.zdi.graal.HelloScriptEngine
native-image --language:js de.zeiss.zdi.graal.HelloScriptPolyglot

The bytecode can then be executed natively on the VMs or the native images. Since the HelloScriptPolyglot is Graal-specific, we cannot simply execute it on the OpenJDK.

A look at the measured values

Each of the three scenarios was executed as bytecode on the OpenJDK, bytecode on the GraalJDK and as a native image. The average program execution times are listed in Table 1.

Hello WorldHelloScriptEngineHelloScriptPolyglot
Bytecode OpenJDK69 ms1321 msX
Bytecode GraalJDK116 ms2889 ms2775 ms
Native Image4 ms13 ms11 ms

Table 1: Example of average program execution times

At first glance, we can see that the execution as a native image is much faster in all three scenarios than the conventional bytecode execution.

However, at second glance, we notice that the bytecode execution with GraalJDK requires much more computing time than with OpenJDK: In the HelloWorld example it needs nearly 70% more time, and in the HelloScriptEngine example it needs over 100% more. This was not communicated by Oracle, but in general it is not such a big problem since the faster bytecode execution is probably not the motivation for using GraalVM . Nevertheless, we should keep this fact in the back of our minds when we want to determine the relevant speed-up from the native image, since GraalVM must be installed in order to create the native image. If we measure the bytecode execution for comparison purposes and execute “java -jar …”, the bytecode is executed via GraalVM. However, since it is most likely that OpenJDK has tended to be used until now in productive operations, we should use this for comparison – and this means the speed-up would be “only” just over half as high.

Things to consider

If we want to achieve the promised performance gains, it is not enough to simply install GraalVM instead of a conventional JDK. During bytecode execution, it was not possible to achieve any performance gains – at least with our examples and setup. This is only possible if we use a native image, but we must keep in mind that this has several disadvantages when compared with bytecode execution.

  • In the native image, Substrate VM is used as JVM. This comes with certain restrictions. Aside from the fact that not all features are currently implemented, there are some things that are not even on the agenda, like a security manager.
  • We should also keep in mind the duration of the build process: For the native image, the start time does not simply disappear. With different approaches, the computing time is “simply” shifted, from the execution time to the build time. In our environment, it took more than three minutes to create our HelloWorld example, and the process of creating the HelloScript program took more than 20 minutes (HelloScriptEngine: 1291 s, HelloScriptPolyglot: 1251 s).
  • However, the biggest challenge is the “closed world assumption”. When the native image is created, a static code analysis is run and only the parts of the code that are run through are compiled in the native image. Although this works for our HelloWorld program, command line parameters had to be input for the JavaScript examples. Classes loaded via “reflection” are only recognised if the fully qualified class name is hard-wired in the source code. This results in problems with any technology that uses dynamic class loading in any form, including JNDI and JMX.

The parts of the program that are loaded dynamically can (and must) be explicitly specified when the native image is created. This includes all parts of the program, from the actual project code to all the libraries used, right up to those of the JRE. Since this configuration is a real challenge for “genuine” programs, tools are provided that are likely to be needed for it to work in practice. For example, the tracing agent monitors a program executed as bytecode. It detects all reflective access operations and uses them to generate a JSON configuration. This can now be used to create the native image.

In practice, the build pipeline would therefore initially create the bytecode variant. All automated tests can then be run with this bytecode variant, and the tracing agent detects the reflective access operations. Assuming that every program path is really executed in this process, the native image can then be generated in a further build step. This takes us directly to the next point: When working with native image technology, the build process becomes longer and more complex overall.

In summary, this means that some things are impossible or close to impossible when using native image technology (e.g. security manager). Although many other things generally work, they require extensive configuration. Tool support is available for this and is undergoing extremely dynamic development. The hope here is that the tools will be able to compensate for the additional work (aside from the build duration). However, this will also make the build process more complex and thus more susceptible to errors.

Pitfalls in Windows

Finally, we will take a look at the Windows platform, which is now also supported. In preparation for this blog post, the “GraalVM Enterprise 20.3.0” and “GraalVM Enterprise 21.0.0.2” versions were tested on a Windows system. Unfortunately, the relevant documentation was still a little lacking here and the tooling does not mesh quite as well as in the Linux environment, so there were some obstacles that were not noticeable in Linux. For instance, there was a problem creating a native image when the underlying bytecode was generated by a different JDK (in this case, by OpenJDK). The error message that appeared was not very helpful either, as it gave no indication of the actual cause.

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
native-image de.zeiss.zdi.graal.HelloWorld
[de.zeiss.zdi.graal.helloworld:20764] classlist: 947.02 ms, 0.96 GB
[de.zeiss.zdi.graal.helloworld:20764] (cap): 3,629.54 ms, 0.96 GB
[de.zeiss.zdi.graal.helloworld:20764] setup: 5,005.98 ms, 0.96 GB
Error: Error compiling query code (in C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.c). Compiler command ''C:Program Files (x86)Microsoft Visual Studio2019BuildToolsVCToolsMSVC14.28.29333binHostX64x64cl.exe' /WX /W4 /wd4244 /wd4245 /wd4800 /wd4804 /wd4214 '-IC:Program FilesJavagraalvm-ee-java11-21.0.0.2includewin32' '/FeC:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe' 'C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.c' ' output included error: [JNIHeaderDirectives.c, Microsoft (R) Incremental Linker Version 14.28.29337.0, Copyright (C) Microsoft Corporation. All rights reserved., , /out:C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe , JNIHeaderDirectives.obj , LINK : fatal error LNK1104: Datei "C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe" kann nicht ge?ffnet werden.]
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1
native-image de.zeiss.zdi.graal.HelloWorld [de.zeiss.zdi.graal.helloworld:20764] classlist: 947.02 ms, 0.96 GB [de.zeiss.zdi.graal.helloworld:20764] (cap): 3,629.54 ms, 0.96 GB [de.zeiss.zdi.graal.helloworld:20764] setup: 5,005.98 ms, 0.96 GB Error: Error compiling query code (in C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.c). Compiler command ''C:Program Files (x86)Microsoft Visual Studio2019BuildToolsVCToolsMSVC14.28.29333binHostX64x64cl.exe' /WX /W4 /wd4244 /wd4245 /wd4800 /wd4804 /wd4214 '-IC:Program FilesJavagraalvm-ee-java11-21.0.0.2includewin32' '/FeC:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe' 'C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.c' ' output included error: [JNIHeaderDirectives.c, Microsoft (R) Incremental Linker Version 14.28.29337.0, Copyright (C) Microsoft Corporation. All rights reserved., , /out:C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe , JNIHeaderDirectives.obj , LINK : fatal error LNK1104: Datei "C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe" kann nicht ge?ffnet werden.] Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception Error: Image build request failed with exit status 1
native-image de.zeiss.zdi.graal.HelloWorld

[de.zeiss.zdi.graal.helloworld:20764]    classlist:     947.02 ms,  0.96 GB
[de.zeiss.zdi.graal.helloworld:20764]        (cap):   3,629.54 ms,  0.96 GB
[de.zeiss.zdi.graal.helloworld:20764]        setup:   5,005.98 ms,  0.96 GB

Error: Error compiling query code (in C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.c). Compiler command ''C:Program Files (x86)Microsoft Visual Studio2019BuildToolsVCToolsMSVC14.28.29333binHostX64x64cl.exe' /WX /W4 /wd4244 /wd4245 /wd4800 /wd4804 /wd4214 '-IC:Program FilesJavagraalvm-ee-java11-21.0.0.2includewin32' '/FeC:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe' 'C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.c' ' output included error: [JNIHeaderDirectives.c, Microsoft (R) Incremental Linker Version 14.28.29337.0, Copyright (C) Microsoft Corporation.  All rights reserved., , /out:C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe , JNIHeaderDirectives.obj , LINK : fatal error LNK1104: Datei "C:UsersxyzAppDataLocalTempSVM-13344835136940746442JNIHeaderDirectives.exe" kann nicht ge?ffnet werden.]

Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1

There was another pitfall when it came to operating across drives, as it is unfortunately not possible in Windows to install GraalVM on one drive (in this case, C:\Program Files) and execute it on another drive (in this case, D:\dev\prj\…):

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
native-image de.zeiss.zdi.graal.HelloWorld
[de.zeiss.zdi.graal.helloworld:10660] classlist: 3,074.80 ms, 0.96 GB
[de.zeiss.zdi.graal.helloworld:10660] setup: 314.93 ms, 0.96 GB
Fatal error:java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: 'other' has different root
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[]
native-image de.zeiss.zdi.graal.HelloWorld [de.zeiss.zdi.graal.helloworld:10660] classlist: 3,074.80 ms, 0.96 GB [de.zeiss.zdi.graal.helloworld:10660] setup: 314.93 ms, 0.96 GB Fatal error:java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: 'other' has different root at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) […]
native-image de.zeiss.zdi.graal.HelloWorld

[de.zeiss.zdi.graal.helloworld:10660]    classlist:   3,074.80 ms,  0.96 GB
[de.zeiss.zdi.graal.helloworld:10660]        setup:     314.93 ms,  0.96 GB

Fatal error:java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: 'other' has different root
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[…]

In addition, it was not possible to identify any performance benefits with the native image in the Windows environment. At present, Windows support (both the GraalVM toolchain itself and the generated native images) is thus rather experimental.

Summary

This blog post has looked primarily at how the start time for Java programs can be massively improved with GraalVM native image technology. It has shown what approaches and technologies GraalVM uses to do this. The results are backed up by measurements from example programs. However, certain challenges were mentioned that arise when native image technology is used.

Virtually no performance improvements can be expected for longer-running programs since the optimisations in conventional JVMs would also apply in such cases. For now, this is just an assumption. An investigation of this aspect would be beyond our scope here and has enough potential to merit its own blog post.

Let’s now turn to the questions from the introduction. In principle, “conventional” Java has not been slow for a very long time; in fact, it is extremely fast. Its use in the (computationally intensive) big data environment is enough of an indication that this is the case. The main prerequisite for a high level of performance is a certain warm-up time. The reverse conclusion is that starting a conventional Java program leaves a lot to be desired, and this is exactly where native image technology comes in. On the other hand, this technology comes with a number of drawbacks, particularly for large, technology-heavy applications.

In summary, GraalVM has the potential to establish itself in various fields of application. Applications in multiple languages could make use of the polyglot properties, and the use of the native image technology that we covered is definitely a viable option for cloud services in particular. However, the use of GraalVM is probably not worthwhile for applications that are computationally intensive (usually those that are longer-running) and non-trivial.

Finally, we should mention the fact that the compiler and optimiser are implemented in Java as a benefit of GraalVM. Although this is initially not better or worse than the previous implementations, it increases the chances of making better use of the potential of the Java community in the medium and long term.

Overall, it remains exciting. At the moment, it is not possible to foresee OpenJDK being replaced entirely. And we should remember that developments in that area are also continuing apace. Nevertheless, GraalVM certainly has the potential to establish itself (at least initially) in specific fields of application.

Security and compliance in software projects – getting dependencies under control

Symbolic picture: White keyboard with key and lock.

This blog post addresses the high standards of security and compliance that we have to meet in every software project. Trained security engineers are responsible for ensuring that we achieve this within any given project. An especially persistent challenge they face is dealing with the countless dependencies present in software projects, and getting them – and their variety of versions – under control.

tree diagramm showing dependencies
Figure 1: An excerpt from the dependency graph of an npm package, taken from npmgraph.js.org/?q=mocha

Challenges in software projects

For some time now, large-scale software projects have consisted of smaller components that can each be reused to serve their particular purpose. Components with features that are not intended to be kept clandestine are increasingly being published in the form of free and open-source software – or FOSS for short – which is freely licensed for reuse.

To assess and prevent security vulnerabilities, it is vital that we have a complete overview of all the third-party libraries we are integrating, as any of our imported modules may be associated with multiple dependencies. This can result in the overall number of dependencies that we are aware of stretching into the thousands – making it difficult to maintain a clear picture of licences and security vulnerabilities among the various versions.

Based on reports of incidents in recent years, such as supply chain attacks and dependency hijacking, there is no mistaking the significant impact that issues like these can have. For an interesting meta-analysis of breaches of this kind, we would recommend Ax Sharma’s article “What Constitutes a Software Supply Chain Attack” (https://blog.sonatype.com/what-constitutes-a-software-supply-chain-attack). Here, we’re going to delve deeper into how to handle components in both large-scale and small-scale software projects, working from the perspective of a security engineer.

FOSS scanning tool solutions

Over time, some projects have managed to overcome the issues associated with identifying FOSS components. Today, there are programs available for creating bills of materials (BOMs) and overviews of security risks, and we have tried these out ourselves.

There are also large catalogues such as Node Package Manager (npm), containing detailed information about the components available in any given case.

Open-source components of this kind might be free to use, but they still involve a certain amount of work, particularly in cases where they are being used in major and long-term software projects.

To perform our own evaluations, we have combined the OWASP Dependency-Check (DC) tool and the OSS Review Toolkit in order to create a solution for identifying security problems through DCs and checking that licensing conditions are being adhered to. Compared with commercial solutions such as Black Duck, these tools provide a free, open option for gaining an overview of FOSS components in projects and evaluating the risks associated with them.

That said, our experience has shown that these tools also involve additional work in the form of configuration and ongoing reviews (in other words, re-running scans in order to identify new security issues).

What software engineers are responsible for

Our guidelines for ensuring secure development and using open-source tools outline the processes we require and the goals that our security engineers have to keep in mind when they are approaching a project. Below is probably the most important part of those guidelines:

It is our responsibility that the following so called Essential FOSS Requirements are fulfilled:

  • All included FOSS components have been identified and the fitness for purpose has been confirmed.
  • All licenses of the included FOSS have been identified, reviewed and compatibility to the final product/service offering has been verified. Any FOSS without a (valid) license has been removed.
  • All license obligations have been fulfilled.
  • All FOSS are continuously – before and after release – monitored for security vulnerabilities. Any relevant vulnerability is mitigated during the whole lifecycle.
  • The FOSS Disclosure Statement is available to the user.
  • The Bill of Material is available internally.

For that it must be ensured that

  • the relevant FOSS roles are determined and nominated.
  • the executing development and procurement staff is properly trained and staffed.

These guidelines form the basis for developing mandatory training, equipping subject matter experts with the right knowledge and putting quality control measures in place.

The processes involved

  • Investigation prior to integration (licences and operational risks such as update frequency)
  • Update monitoring (operational risks)

Let’s say that a new function needs to be built into a software project. In many cases, developers will already be aware of FOSS tools that could help introduce the function.

Where feasible, it is important that whichever developer is involved in the project knows how to handle package managers and the potential implications of using them so that they know how to account for the results produced by tools or analyses. As an example, developers need to be able to visualise how many parts are involved in a top-level dependency, or evaluate various dependencies associated with the same function in order to maintain security in any future development work. In other words, they must be able to assess operational risks. More and more nowadays, we are seeing projects that aim to keep the number of dependencies low. This needs to be taken into account when selecting components so that, wherever possible, additional dependencies only provide the functions that are really needed.

Before integration, the security engineer also has to check potential imports for any security vulnerabilities and verify that they have a compatible licence. An equally important job is reviewing the operational risks, involving aspects such as the following:

  • How up-to-date the import is
  • Whether it is actively maintained or has a keenly involved community
  • Whether the update cycle is agile enough to deal with any security vulnerabilities that crop up
  • How important secure handling of dependencies is considered to be
  • Whether the number of additional dependencies is reasonable and whether it is reduced where possible

During the development process and while operation is taking place further down the line, the project team also has to be notified whenever new security vulnerabilities are identified or closed. This may involve periodic scans or a database with security vulnerability alerts. Periodic scans have the advantage of running more independently than a database, which requires hardware and alerts to be provided. However, alerts are among the benefits offered by software composition analysis solutions such as Black Duck.

As the number of well-marked FOSS tools rises, the amount of time that needs to be invested in curating them manually is becoming comparatively low. The work that does need to be done may involve declaring a licence – and adding easy-to-find, well-formatted copyright details to components, as these have often been given highly unusual formats or left out altogether in older components. Cases in which no licence details are provided should never be misconstrued as carte blanche invitations to proceed – without a licence, a component must not be used without the author’s consent.

Example of a security vulnerability

An example of a complex security vulnerability was published in CVE-2021-32796. The module creating the issue, xmldom, is indirectly integrated via two additional dependencies in our example project here.

Black Duck shows the following security warning related to the module:

screenshot black duck
Figure 2: Black Duck example summarising a vulnerability

This gives a security engineer enough information to make a broad assessment of the implications that this vulnerability has. Information is also provided with the patch in version 0.7.0.

The importance of having enough time in advance when it comes to running updates/replacing components

Before issuing new publications under @xmldom/xmldom, we have had the time to check how much work would be involved if we were to do without this dependency.

To benefit from this kind of time in a project, it is useful to gain an overview of potential issues right at the development stage, and ensure that there is enough of a time buffer leading up to the point at which the product is published.

This makes it easier for developers to evaluate workarounds for problematic software libraries, whether they are affected by security vulnerabilities, incompatible licences or other operational risks.

Summary

This post has provided an overview of the project work we do involving the large variety of open-source software out there, and has outlined what security engineers need to do when handling open-source software. By using the very latest tools, we are able to maintain control over a whole range of dependencies and establish the transparency and security we need. Dependencies need to be evaluated by a trained team before they are integrated and then monitored throughout the software lifecycle, with the team responding to any issues that may arise.

Test automation tool from the catalogue – How do I find the perfect tool?

Software systems are becoming increasingly complex due to the constantly growing number of applications on different platforms. A decisive factor for the success of a software product is its quality. For this reason, more and more companies are carrying out systematic checks and tests, if possible, on the various platforms, in order to be able to ensure a given quality standard. In order to be able to keep short release cycles despite the higher testing effort, it becomes necessary to automate the tests. This in turn leads to the need to define a test automation strategy. One of the first steps in introducing a test automation strategy is to evaluate suitable test automation tools. Since each project is unique, both the requirements and the choice of tools vary. This blog series is intended to provide guidance in selecting the appropriate solution.

Symbolical image: Two people are sitting at a table and looking at a tablet display, under the tablet some printed statistics and some notebooks are lying on the desk.
Figure 1: Evaluating suitable test automation tools is an important part of introducing a test automation strategy.

Introduction

As part of my final thesis, I took on the task of providing the Scrum teams at ZEISS Digital Innovation (ZDI) with an aid to help them find the right test automation tool quickly and flexibly. The challenge here was that the projects have specific scenarios and the requirements may have to be weighted separately. I would like to present the status of the work and the results to you in this and the next blog articles.

Software development has long been an area of rapid change. But while in the past these developments have mainly taken place at the technological level, we are currently observing major changes in the area of processes, organisation and tools in software development. However, these new trends come with challenges such as changes in requirements management, shortened timelines and especially the increased demands on quality. Today, the development of needs-based solutions and the optimisation of quality increase both efficiency and effectiveness within software development.

In addition, software systems are becoming more and more complex due to the constantly growing number of applications on different platforms. Since quality is a decisive factor for the success of a software product, more and more companies are carrying out systematic checks and tests, if possible, on the various platforms, in order to ensure a given quality standard. A SmartBear survey conducted in early 2021 found that many companies, regardless of the sector, already perform different types of testing, with web application testing leading the way at 69% and API/web services testing second at 61%. Desktop testing is done by 42% of the respondents. A total of 62 % of the respondents state that they perform mobile testing, 29 % of which for native applications (apps) (SmartBear, 2021, p. 12). In order to be able to maintain short release cycles despite the higher testing effort, it is becoming necessary to automate the tests.

As ZDI, we support our customers inside and outside the ZEISS Group in their digitisation projects. We also offer individual test services. That is why we have a large number of projects with different technologies and different solutions. As a small selection, keywords such as Java, .Net, JavaFX, WPF, Qt, Cloud, Angular, Embedded etc. should be mentioned here. In such complex projects, quality assurance is always in the foreground and the projects are dependent on the use of modern test tools that support the project participants in the manual, but especially in the automated tests. It would be desirable to have a tool that supports this automation effectively and efficiently. However, testers face a variety of challenges when selecting a test automation tool.

Challenges

In the research and interviews conducted as part of my work, the greatest challenges in test automation were named as the variety of test tools available. Due to this fragmentation a variety of tools is available for the same purpose.

The choice becomes even more difficult because the tools differ not only in the technology they can test, but also in their work approach. When automation is mentioned, it is always associated with scripting. In recent years, however, a new approach to GUI testing has been developed called NoCode/LowCode. This approach basically requires no programming knowledge and is therefore popular with automation beginners. Nevertheless, scripting remains the dominant method, although sometimes both approaches are used to involve as many of the quality assurance team as possible (SmartBear, 2021, p. 33).

The type of test automation approach, and therefore the use of a test automation tool, depends on the requirements in the project. This means that the requirements must always be re-examined for each new project.

Inventory

In the course of the interviews, the analysis of the current approach and the evaluation of an internal survey, I was able to identify the following procedures for the selection of tools in the projects, which have become established as a “quasi-standard”:

  • Das The tool is Open Source and costs nothing.
  • The tool has been used before and is considered trustworthy.

One aim of the survey was to find out to what extent the project situation has an influence on the tool selection. Therefore, the next step was to examine the project situations and the process models used in the projects and their influence. It turned out that it is not the process models that have a direct influence on the tool selection, but the people or groups who use the tool as future users or who approve its use.

When examining the influence of these operationally-involved participants, it became apparent that there are other interest groups that have a direct or indirect influence on the selection of a tool. These are, for example:

  • Software architects, who define the technology or tool chains within the project,
  • The management, which sets guidelines for the purchase or use of tools (OpenSource, GreenIT, strategic partnerships, etc.) or
  • The company’s IT, which prevents the use of certain tools through the infrastructure used (firewall, cloud, etc.).

Furthermore, the first approaches of checklists for the selection of test tools were already found during the research. They were mostly based on a few functional criteria and were determined in workshops by the project members. The evaluation showed that there was no uniform method and that the tools selected in this way were very often later replaced by other tools. This became necessary because essential requirements for the tool were overlooked during the selection.

In the majority of cases, the forgotten requirements were of a non-functional nature, such as usability criteria or performance requirements. Therefore, a next step in checking relevant criteria was to refer to ISO/IEC 25000 Software Engineering (Quality criteria and evaluation of software products.

Schematic representation of the criteria of ISO/IEC 25000 Software Engineering
Figure 2: Criteria for software according to ISO/IEC 25010

The next blog article in this series will show how the criteria catalogue is structured and how the final list of criteria is composed.


Literature

SmartBear (2021): State of Software Quality | Testing.

This article was technically supported by:

Kay Grebenstein

After graduating in computer sciences, Kay Grebenstein was drifting over to “The Dark Side” of software development and he is now working as a software tester and agile QA coach at ZEISS Digital Innovation, Dresden. Over the last few years, he was responsible for quality assurance and software testing in many projects in different business domains (telecommunications, industry, commerce, energy sector, …). He has experience with both the classical as well as the agile process models.

Alle Beiträge des Autors anzeigen

Patient care of the future – Digital Health Solutions with Azure Health Data Services

Since the beginning of the Covid pandemic the healthcare sector has been under enormous pressure. The demographic development, the change in the spectrum of diseases, legal regulations, cost pressure and a shortage of specialists combined with the increasing demands of patients, present healthcare organisations with a number of challenges. Here, digitalisation and the use of modern technologies such as artificial intelligence or machine learning offer numerous opportunities and potentials for increasing efficiency, reducing errors and thus improving patient treatment.

Doctor uses cloud-based medical application on smartphone, healthcare professionals talking in the background
Figure 1: Digital Health Solutions with Azure Health Data Services for optimal and future-proof patient care

Use of medical data as the basis for optimised patient care

The basis for the use of these technologies and for future-oriented predictive and preventive care is medical data. This can already be found everywhere today. However, most healthcare professionals and the medical devices in use still store this on-premise, resulting in millions of isolated medical data sets. In order to get a fully comprehensive overview of a patient’s medical history and, based on this, to create treatment plans in terms of patient-centred therapy and to be able to derive overarching insights from these data sets, organisations need to integrate and synchronise health data from different sources.

To support the development of healthcare ecosystems, the major global public cloud providers (Microsoft Azure, Amazon Web Service and Google Cloud Platform) are increasingly offering special SaaS and PaaS services for the healthcare sector that can provide companies with a basis for their own solutions. Through our experience at ZEISS Digital Innovation as an implementation partner of Carl Zeiss Meditec AG and of customers outside the ZEISS Group, we recognised early on that Microsoft offers a particularly powerful healthcare portfolio and is continuing to expand it strongly. This became clear again at this year’s Ignite.

Screenshot of a video in which two people are talking virtually about a certain topic
ZEISS Digital Innovation (right) at Ignite 2021 talking about how to get long-term value from healthcare data with Microsoft Cloud for Healthcare. (Click here for the full video)

Medical data platforms based on Azure Health Data Services

One possibility for building such a medical data platform as the basis of an ecosystem is the use oAzure Health Data Services. With the help of these services, the storage, access and processing of medical data can be made interoperable and secure. Thousands of medical devices can be connected to each other and the data generated in this way can be used by numerous applications in a scalable and robust manner. As Azure Health Data Services are PaaS solutions, they can be used out of the box and are fully developed, managed and operated by Microsoft. They are highly available with little effort, designed for security and are in compliance with regulatory requirements. This significantly reduces the implementation effort and thus also the costs.

Carl Zeiss Meditec AG also relies on Azure Health Data Services to develop its digital, data-driven ecosystem. The ZEISS Medical Ecosystem, developed together with ZEISS Digital Innovation, connects devices and clinical systems with applications via a central data platform, creating added value at various levels to optimise clinical patient management.

The DICOM service within Azure Health Data Services is used here as the central interface for device connection. As DICOM is an open standard for storing and exchanging information in medical image data management, the majority of medical devices that generate image data communicate using the DICOM protocol. Through an extensible connectivity solution based on Azure IoT Edge, these devices can connect directly to the data platform in Azure using the DICOM standard. This allows a wide range of devices that have been in use with customers for years to be integrated into the ecosystem. This increases acceptance and ensures that more data can flow into the cloud and be further processed to enable clinical use cases and develop new procedures.

Azure API for FHIR® serves as the central data hub of the platform. All data of the ecosystem are stored there in a structured way and linked with each other in order to make them centrally findable and available to the applications. HL7® FHIR® (Fast Healthcare Interoperability Resources) offers a standardised and comprehensive data model for healthcare data. Not only can it be used to implement simple and robust interfaces to one’s own applications, but it also ensures interoperability with third-party systems such as EMR systems (Electronic Medical Record), hospital information systems or the electronic patient record. The data from the medical devices, historical measurement data from local PACS solutions and information from other clinical systems are automatically processed, structured and aggregated centrally in Azure API for FHIR® after upload. This is a key factor in collecting more valuable data for clinical use cases and providing customers with a seamlessly integrated ecosystem.

Schematic representation of building a medical data platform with Azure Healthcare APIs
Figure 2: Building a medical data platform with Azure Health Data Services

Successful collaboration between ZEISS Digital Innovation and Microsoft

As early adopters of Azure Health Data Services, our development teams at ZEISS Digital Innovation work closely with the Azure Health Data Services product group at Microsoft headquarters in Redmond, USA, helping to shape the services for the benefit of our customers. In regular co-creation sessions between the ZEISS Digital Innovation and Microsoft teams, the solution design for currently implemented features of the Azure Health Data Services is discussed. In this way, we can ensure that even the most complex use cases currently known are taken into account.

We are working very closely with ZEISS Digital Innovation to shape Azure’s next generation health services alongside their customer needs. Their strong background in the development of digital medical products for their customers is a core asset in our collaboration and enables the development of innovative solutions for the healthcare sector.

Steven Borg (Director, Medical Imaging at Microsoft)

You too can benefit from our know-how and contact us. Together, we will develop the vision for your innovative solution and support you during implementation.

This post was written by:

Elisa Kunze

Elisa Kunze has been working at ZEISS Digital Innovation since 2013. During her various sales and marketing activities she supported lots of different projects, teams and companies in various sectors. Today she supports her clients in the health sector as a key account manager and supports them in implementing their project vision.

See author’s posts

Evaluate with matrices – The perfect decision matrix

A not inconsiderable part of the work of a software architect consists of comparing different solution alternatives with each other. Decision tables or evaluation matrices are often used for this purpose, whereby both terms are usually used synonymously. This article aims to provide an insight into two basic approaches and to evaluate them according to their suitability. 

Symbolical image showing a persons's feet in white sneakers standing on a painted white arrow on an asphalt street pointing in two directions
Figure 1: Software architects often have to compare different solution alternatives with each other. To do this, they often use certain evaluation matrices. 

Types of evaluation matrices 

Evaluation methods for comparing several options range from simple rankings based on voting to complicated evaluation methods based on matrices calculations. The challenge is to find the most appropriate methodology for an objectified evaluation of the comparison of two options available. Criteria for this are:  

  • Fast comparison 
  • Simple, uncomplicated procedure 
  • Little to no training time
  • Any number of people 

After a brief comparison of the evaluation methods based on the criteria mentioned, the very widespread and well-known utility value analysis turns out to be the method of choice here. Why? Because it is an uncomplicated and simple way of comparing options on the basis of various criteria without the need for a great deal of mathematical effort. In the utility value analysis, the variants are evaluated with a score using weighted criteria, the weighting and score are multiplied together and all the evaluations for an option are added up (see example).

table showing weighted criteria
Figure 2: Evaluation of the individual variants in a utility value analysis 

This procedure should be familiar to almost everyone and is also used in practice in many areas. In addition to the evaluation by the person giving the score, there is a potential source of subjective error: the weighting of the criteria. Since score allocation cannot be made more objective, a way of calculating the weighting objectively must be found. This is absolutely essential if a meaningful utility value analysis is required. Objective weighting ensures that no “mindless” decisions are made, e.g. due to time pressure, and that the weighting is as independent of the observer as possible. 

Procedure for calculating weighting 

As with the utility value analysis, the procedure should be uncomplicated, quick and require no training. Under these premises, two procedures in particular emerge, which are briefly explained in the following below.

Graduated weighting

In the graduated weighting calculation, all criteria are compared with each other in terms of their importance. Thereby, the scale comprises five gradations from -2: “significantly lower importance” to 2: “significantly higher importance”. This granular assessment must therefore be made for each pairing of criteria. The weighting is then calculated using a method similar to matrices calculation.

table showing the scale of five gradations
Figure 3: Example of a scale with graduated weighting 

Priority weighting 

Here, the granular evaluation is dispensed with and a distinction is only made between “more important” and “less important”. Thus, each criterion is compared with each other and the more important one is noted in the table. The relative proportion of the number of a criterion is then used to determine the weighting. This procedure can be integrated well in a team, because after the ranking by the individuals, all results can be combined and thus a representative weighting is obtained. 

table showing priority weighting
Figure 4: Example of priority weighting of individual criteria 

When is which procedure suitable? 

In principle, any weighting procedure can be used in any situation. However, the following table gives indications for which calculation method can be advantageous in which circumstances. 

Graduated weighting Priority weighting 
Low number of criteria High number of criteria 
Low number of evaluating persons High number of evaluating persons 
Sufficient time available Little time available 
Especially important decision  

In summary, it can be said that the procedure of graduated weighting is often too labour-intensive if one assumes in practice that decisions are sometimes not even accompanied by a representative evaluation procedure. Priority weighting, on the other hand, is an uncomplicated option that can be quickly understood and also implemented in a team and is therefore particularly recommended. 

In addition to the utility value analysis, other methods and indicators can be used to compare variants, such as standard deviation, number of gains/losses, etc., which should make the final decision easier, but which are not part of this article. 

Generate test data with Oracle SQL

Many projects work with databases, and of course the performance of the system has to be tested. Usually, smaller amounts of data are entered by means of manual tests, but how do you get bulk data? 

The usual answer is, of course: “Buy one of the tools” – and usually rightly so. These tools have an (often) fairly easy-to-use UI, are quick and above all they can generate semantically meaningful data.  

Colleagues working on a screen
Figure: To perform a meaningful performance test for databases, testers need to simulate the processing of both small and large amounts of data.

Generating data with SQL

However, with a little SQL you can also generate such data yourself. It gets interesting when linking multiple tables, which plays an important role in the following example. 

Scenario: Our project manages companies and their plants and for them invoice data is to be generated. There are different views and queries for this data in the project. We want to have some areas with little data (to test the functioning of the system), but also bulk data. 

First, we create the companies (here only three of them) with a single instruction. This is the first important point to achieve good performance in the generation of the data. Of course, you could write three INSERT statements one after the other or use a loop 1-3 with PL/SQL. But after a certain amount of data, that you want to generate, it will become massively slower.

Sub-select „company_seq“: Creates a sequence of 1-3 

Insert Statement: The company will get a meaningful name (high number = more data).

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
insert into company (name)
select
'company '||company_seq.company_number as name
from (
select rownum as company_number
from dual
connect by level<=3
) company_seq
;
insert into company (name) select 'company '||company_seq.company_number as name from ( select rownum as company_number from dual connect by level<=3 ) company_seq ;
insert into company (name) 
  select
    'company '||company_seq.company_number as name
  from (
    select rownum as company_number 
    from dual 
    connect by level<=3
  ) company_seq
;

Next, we would like to assign the plants to the companies. Remember: We want some areas with little data, others with a lot of data. Therefore, three plants are to be assigned to the first company, six plants to the second company, etc.

Sub-select “Company“: The ranking gives you a running number 1-N, so you can easily set the desired number of plants per company. This query produces exactly one line per company. 

Sub-select “plant_seq“: We need a sequence to generate a row by joining the “company” subquery and desired number of attachments. To do this, we use a generously estimated sequence. The join is limited by the previously determined number of plants per company. 

Insert Statement: The plant name should indicate which company the plant belongs to.  

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
insert into plant (firma_id, name)
select
company.company_id,
'plant '||company.company_number||'.'||plant_seq.plant_number
from (
select
company_id,
dense_rank() over (order by company_id) as company_number,
dense_rank() over (order by company_id) * 3 as amount_plants
from company
) company
join (
select rownum as plant_number from dual connect by level<=1000
) plant_seq on (plant_seq.plant_number <= company.amount_plants)
;
insert into plant (firma_id, name) select company.company_id, 'plant '||company.company_number||'.'||plant_seq.plant_number from ( select company_id, dense_rank() over (order by company_id) as company_number, dense_rank() over (order by company_id) * 3 as amount_plants from company ) company join ( select rownum as plant_number from dual connect by level<=1000 ) plant_seq on (plant_seq.plant_number <= company.amount_plants) ;
insert into plant (firma_id, name) 
  select 
    company.company_id,
    'plant '||company.company_number||'.'||plant_seq.plant_number
  from (
    select 
      company_id, 
      dense_rank() over (order by company_id) as company_number,
      dense_rank() over (order by company_id) * 3 as amount_plants
    from company
  ) company
  join (
    select rownum as plant_number from dual connect by level<=1000
  ) plant_seq on (plant_seq.plant_number <= company.amount_plants)
;

Finally, a billing per calendar month is to be generated for each company and plant.

The core element for random data in Oracle is the dbms_random package. It offers several functions for generating numbers, with the default range 0-1 or with self-defined ranges. However, the decimal places of the generated numbers have to be checked by rounding up or down. There is also a generator for characters available, with some modes like Large, Small, Alphanumeric, …  

Column “Invoice number“: Three capital letters followed by six digits 

Column “Amount“: A random value in the range € 100.900.  

Column “Surcharges”: For approx. 10% of invoices a surcharge of € 50 is to be charged, this is achieved by comparing a random number in the range 0-1 to < 0.10. 

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
insert into billing (company_id,plant_id,billing_month,invoice_number,amount,surcharges)
select
company.company_id,
plant.plant_id,
to_date('01.'||month||'.2021', 'dd.mm.yyyy') as billing_month,
dbms_random.string('U',3) || round(dbms_random.value(10000,99999), 0) as invoice_number,
round(100 + dbms_random.value(100,900),2) as amount,
case when dbms_random.value<0.10 then 50 else null end as surcharges
from company
join plant
on plant.company_id = company.company_id
join (
select rownum as month from dual connect by level<=12
) month_seq on (1=1)
order by 1,2,3
;
insert into billing (company_id,plant_id,billing_month,invoice_number,amount,surcharges) select company.company_id, plant.plant_id, to_date('01.'||month||'.2021', 'dd.mm.yyyy') as billing_month, dbms_random.string('U',3) || round(dbms_random.value(10000,99999), 0) as invoice_number, round(100 + dbms_random.value(100,900),2) as amount, case when dbms_random.value<0.10 then 50 else null end as surcharges from company join plant on plant.company_id = company.company_id join ( select rownum as month from dual connect by level<=12 ) month_seq on (1=1) order by 1,2,3 ;
insert into billing (company_id,plant_id,billing_month,invoice_number,amount,surcharges)
  select 
    company.company_id,
    plant.plant_id,
    to_date('01.'||month||'.2021', 'dd.mm.yyyy') as billing_month,
    dbms_random.string('U',3) || round(dbms_random.value(10000,99999), 0) as invoice_number,
    round(100 + dbms_random.value(100,900),2) as amount,
    case when dbms_random.value<0.10 then 50 else null end as surcharges
  from company
  join plant 
    on plant.company_id = company.company_id
  join (
    select rownum as month from dual connect by level<=12
  ) month_seq on (1=1)
  order by 1,2,3
;

Conclusion

These data do not win a beauty prize, so it may be better to resort to the mentioned products. But in our project, this approach has helped to provide data of sufficient quality and, above all, quantity from day one. This enabled us to ensure the performance of the system right from the start. In addition, the end result is that the system can process more data with better performance than similar systems.


Source Code

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
--------------------------------------------------------------------------------------------------------------
-- Create data model
--------------------------------------------------------------------------------------------------------------
drop table billing;
drop table plant;
drop table company;
create table company (
company_id number(9) generated as identity,
name varchar2(100) not null,
constraint pk_company primary key (company_id)
);
create table plant (
plant_id number(9) generated as identity,
company_id number(9) not null,
name varchar2(100) not null,
constraint pk_plant primary key (anlage_id),
constraint fk_plant_01 foreign key (company_id) references company (company_id)
);
create table billing (
billing_id number(9) generated as identity,
company_id number(9) not null,
plant_id number(9) not null,
billing_month date not null,
invoice_number varchar(30) not null,
amount number(18,2) not null,
surcharges number(18,2),
constraint pk_billing primary key (billing_id),
constraint fk_billing_01 foreign key (company_id) references company (company_id),
constraint fk_billing_02 foreign key (plant_id) references plant (plant_id)
);
--------------------------------------------------------------------------------------------------------------
-- Generate data
--------------------------------------------------------------------------------------------------------------
-- Delete all data
truncate table billing;
truncate table plant;
truncate table company;
whenever sqlerror exit rollback;
-- Generate Companies (here 3)
insert into company (name)
select
'company '||company_seq.company_number as name
from (select rownum as company_number from dual connect by level<=3) company_seq
;
commit;
-- Insert attachments per company. The first company will receive three plants, the next six plants, ...
insert into plant (company_id, name)
select
company.company_id,
'plant '||company.company_number||'.'||plant_seq.plant_number
from (
-- Determine the serial number for each company and the desired number of plants
select
company_id,
dense_rank() over (order by company_id) as company_number,
dense_rank() over (order by company_id) * 3 as amount_plants
from company
) company
join (
-- A sequence of attachments by which to join the required rows per company with N attachments
select rownum as plant_number from dual connect by level<=1000
) plant_seq on (plant_seq.plant_number <= company.amount_plants)
-- order by 1,2
;
commit;
-- Generate an invoice per calendar month for each company and plant.
-- The invoice number is a random string with three letters and six digits.
-- The amount is a random value in the range 100-900 euros.
-- The surcharge of 50 euros is applied to approx. 10% of the invoices.
insert into billing (company_id,plant_id,billing_month,invoice_number,amount,surcharges)
select
company.company_id,
plant.plant_id,
to_date('01.'||month||'.2021', 'dd.mm.yyyy') as billing_month,
dbms_random.string('U',3) || round(dbms_random.value(10000,99999), 0) as invoice_number,
round(dbms_random.value(100,1000),2) as amount,
case when dbms_random.value<0.10 then 50 else null end as surcharges
from company
join plant on plant.company_id = company.company_id
join (select rownum as month from dual connect by level<=12) month_seq on (1=1)
-- order by 1,2,3
;
commit;
-------------------------------------------------------------------------------------------------------------- -- Create data model -------------------------------------------------------------------------------------------------------------- drop table billing; drop table plant; drop table company; create table company ( company_id number(9) generated as identity, name varchar2(100) not null, constraint pk_company primary key (company_id) ); create table plant ( plant_id number(9) generated as identity, company_id number(9) not null, name varchar2(100) not null, constraint pk_plant primary key (anlage_id), constraint fk_plant_01 foreign key (company_id) references company (company_id) ); create table billing ( billing_id number(9) generated as identity, company_id number(9) not null, plant_id number(9) not null, billing_month date not null, invoice_number varchar(30) not null, amount number(18,2) not null, surcharges number(18,2), constraint pk_billing primary key (billing_id), constraint fk_billing_01 foreign key (company_id) references company (company_id), constraint fk_billing_02 foreign key (plant_id) references plant (plant_id) ); -------------------------------------------------------------------------------------------------------------- -- Generate data -------------------------------------------------------------------------------------------------------------- -- Delete all data truncate table billing; truncate table plant; truncate table company; whenever sqlerror exit rollback; -- Generate Companies (here 3) insert into company (name) select 'company '||company_seq.company_number as name from (select rownum as company_number from dual connect by level<=3) company_seq ; commit; -- Insert attachments per company. The first company will receive three plants, the next six plants, ... insert into plant (company_id, name) select company.company_id, 'plant '||company.company_number||'.'||plant_seq.plant_number from ( -- Determine the serial number for each company and the desired number of plants select company_id, dense_rank() over (order by company_id) as company_number, dense_rank() over (order by company_id) * 3 as amount_plants from company ) company join ( -- A sequence of attachments by which to join the required rows per company with N attachments select rownum as plant_number from dual connect by level<=1000 ) plant_seq on (plant_seq.plant_number <= company.amount_plants) -- order by 1,2 ; commit; -- Generate an invoice per calendar month for each company and plant. -- The invoice number is a random string with three letters and six digits. -- The amount is a random value in the range 100-900 euros. -- The surcharge of 50 euros is applied to approx. 10% of the invoices. insert into billing (company_id,plant_id,billing_month,invoice_number,amount,surcharges) select company.company_id, plant.plant_id, to_date('01.'||month||'.2021', 'dd.mm.yyyy') as billing_month, dbms_random.string('U',3) || round(dbms_random.value(10000,99999), 0) as invoice_number, round(dbms_random.value(100,1000),2) as amount, case when dbms_random.value<0.10 then 50 else null end as surcharges from company join plant on plant.company_id = company.company_id join (select rownum as month from dual connect by level<=12) month_seq on (1=1) -- order by 1,2,3 ; commit;
--------------------------------------------------------------------------------------------------------------
-- Create data model
--------------------------------------------------------------------------------------------------------------
drop table billing;
drop table plant;
drop table company;

create table company (
  company_id number(9) generated as identity,
  name varchar2(100) not null,
  constraint pk_company primary key (company_id)
);

create table plant (
  plant_id number(9) generated as identity,
  company_id number(9) not null,
  name varchar2(100) not null,
  constraint pk_plant primary key (anlage_id),
  constraint fk_plant_01 foreign key (company_id) references company (company_id)
);

create table billing (
  billing_id number(9) generated as identity,
  company_id number(9) not null,
  plant_id number(9) not null,
  billing_month date not null,
  invoice_number varchar(30) not null,
  amount number(18,2) not null,
  surcharges number(18,2),
  constraint pk_billing primary key (billing_id),
  constraint fk_billing_01 foreign key (company_id) references company (company_id),
  constraint fk_billing_02 foreign key (plant_id) references plant (plant_id)
);

--------------------------------------------------------------------------------------------------------------
-- Generate data
--------------------------------------------------------------------------------------------------------------

-- Delete all data

truncate table billing;
truncate table plant;
truncate table company;

whenever sqlerror exit rollback;

-- Generate Companies  (here 3)
insert into company (name) 
  select
    'company '||company_seq.company_number as name
  from (select rownum as company_number from dual connect by level<=3) company_seq
;
 
commit;

-- Insert attachments per company. The first company will receive three plants, the next six plants, ...
insert into plant (company_id, name) 
  select 
    company.company_id,
    'plant '||company.company_number||'.'||plant_seq.plant_number
  from (
    -- Determine the serial number for each company and the desired number of plants
    select 
      company_id, 
      dense_rank() over (order by company_id) as company_number,
      dense_rank() over (order by company_id) * 3 as amount_plants
    from company
  ) company
  join (
    -- A sequence of attachments by which to join the required rows per company with N attachments
    select rownum as plant_number from dual connect by level<=1000
  ) plant_seq on (plant_seq.plant_number <= company.amount_plants)
  -- order by 1,2
;
 
commit;

-- Generate an invoice per calendar month for each company and plant.
-- The invoice number is a random string with three letters and six digits.
-- The amount is a random value in the range 100-900 euros.
-- The surcharge of 50 euros is applied to approx. 10% of the invoices.
insert into billing (company_id,plant_id,billing_month,invoice_number,amount,surcharges)
  select 
    company.company_id,
    plant.plant_id,
    to_date('01.'||month||'.2021', 'dd.mm.yyyy') as billing_month,
    dbms_random.string('U',3) || round(dbms_random.value(10000,99999), 0) as invoice_number,
    round(dbms_random.value(100,1000),2) as amount,
    case when dbms_random.value<0.10 then 50 else null end as surcharges
  from company
  join plant on plant.company_id = company.company_id
  join (select rownum as month from dual connect by level<=12) month_seq on (1=1)
  -- order by 1,2,3
;
 
commit;

Distributed Work – An Experience Report on the Practical Application of Remote Mob-Testing

Within ZEISS Digital Innovation (ZDI) there is an internal training event at regular intervals – the so-called ZDI Campus Event. As employees, we present software development topics by means of lectures, workshops or discussion rounds. 

Katharina had already reported on collaborative testing methods at the last ZDI Campus, and also published blog articles about Remote Pair Testing and Remote Mob Testing. Therefore, we wanted to continue the topic at the next ZDI Campus and offer a workshop on the topic of collaborative testing methods. However, due to the Covid-19 pandemic, the next campus had to be held online. Nevertheless, we wanted to offer several participants the opportunity to apply a test method using a practical example, and therefore decided to hold a remote Mob-Testing Workshop.  

But there was a challenge: We had never worked with so many people in the mob remotely! 

Structure of the workshop 

As described in the blog post about Remote Mob Testing, it makes more sense to supervise small groups of four to six people. Distributed work may otherwise lead to more frequent delays due to technical problems (e. g. poor connection, poor sound quality), which could reduce the time required for the current navigator. As a facilitator, you can also keep an overview of smaller groups and the participants can take on the role of navigator or driver more often.
(Zur Erläuterung der verschiedenen Rollen siehe ebenfalls Blogbeitrag Remote Mob Testing

Our setting looked like this:

  • Microsoft Teams as a communication platform  
  • Easy to understand test object (website)
  • 3 Facilitators  
  • 39 participants from the fields of QA, Development and Business Analysis 
  • Timeframe: 1.5 hours  

Because we wanted to give everyone the opportunity to get to know the Mob-Testing themselves by means of a practical example, we did not set a limit on participants in the run-up to the workshop. In the end, all three facilitators had more than 12 participants.  

person in front of a screen having a video conference
In remote mob testing, all participants take on the active role of the navigator once.

As a test object, we chose a simple website so that everyone could concentrate on communicating or getting to know the test method and not have to acquire additional knowledge about the application. 

Feedback 

Already during the course of the workshop, we noticed that waiting times for an active role (Driver or Navigator) could be perceived as unpleasant.   

This was also mentioned in the feedback round. Therefore, we recommend that the mob assists with test ideas, because it doesn’t mean that you can sit back and wander with your thoughts as a mob member. This also avoids double testing or having to admit that you were not paying attention.  

In some cases, it was difficult for some participants to focus purely on the role of the driver – executing the instructions of the navigator, and holding back on their own test ideas. However, the participants got used to it after some remarks by the facilitator. This aspect of Mob-Testing was considered to be very positive, because everyone in the role of the navigator can speak and contribute their own test ideas.  

However, the question arose as to why the roles Navigator and Driver could not be combined. The following can be said: It promotes the learning process when a member articulates step-by-step what he or she intends to do. In this way, more participants are involved in this active role. Communicating the destination makes it easier for the participants to follow the ideas of the navigator. Otherwise, some steps may be carried out too quickly, and with too little explanation. This would lose traceability and make it difficult for the mob to get actively involved.  

There was other positive feedback on the division of roles and the whole process. According to the respondents, the test approaches are partly obtained by the path taken by the previous navigator, and therefore viewed the test object from a variety of angles. For this reason, it is always useful to invite participants with different skills, depending on the purpose of the Mob-Testing session. This increases the learning process, and the number of test cases. The simple sample test object also helped to focus on the method and to internalize it. 

The collaborative work in Mob-Testing was highlighted very positively. This is how the agile thought is experienced.  

Solution approach for a large number of participants 

The problem of too many participants could be solved by limiting the size of the group in advance or by introducing a new role: the role of spectator. A separation would then be made between active participants on the one hand and spectators on the other. Participants would follow the procedure described above, and follow the roles distribution (navigator, driver, mob) and the role change. The spectators would only observe and not participate. Comments by them would not be allowed either, because this could disturb the active participants with a large number of spectators. 

Conclusion 

All in all, the workshop was very well received at the Campus event, and showed that it is very possible to use Mob-Testing remotely, and therefore for distributed work. This means that cooperation can be maintained, even if it is not always possible to meet on site.  

This post was written by:

Maria Petzold

Maria Petzold has been working at ZEISS Digital Innovation since 2010. As a test manager, her focus is on software quality assurance. She was able to gain her test experience especially in medical software projects.

See author’s posts

UI-Dev Session (Part 2) – Experience report from everyday project life

The first post of this blog article series describes a lightweight method to eliminate discrepancy between the default user interface design and the implemented application. Developers as well as specialists for User Interface (UI) / User Experience (UX) work in this area by means of Pair Programming, in order to optimize the usability and appearance of an application in an uncomplicated way. 

In our project we regularly conduct such “UI-Dev Sessions.” In this blog post I want to report on our experiences – away from theory to practice. 

Why are the UI-Dev Sessions important to us?  

Within the scope of the project a standalone software is being developed, which serves ophthalmic specialists as a support tool for collecting patient data and analyzing the results of surgical laser vision corrections. Also in the medical environment, user experience is becoming an important purchasing criterion. In addition to safety, which is probably the most important criterion for medical devices, soft criteria such as “Joy of Use” and appearance are gaining in importance. The UI-Dev Sessions are a way for us to give the finishing touch to our application. 

„The details are not the details. They make the design.” 

Charles Eames, Designer
screenshot of the software, which serves ophthalmic specialists as a support tool for collecting patient data and analyzing the results of surgical laser vision corrections
Figure 1: Screenshot of Software

How do the UI-Dev Sessions work with us?  

Our project team works agilely and uses Scrum as a framework for its approach. Like most of the teams at ZEISS Digital Innovation (ZDI), the team members work in a distributed way, i. e. they are not in the same location and therefore not in the same office. Our project team is spread over four locations in two countries. The roles of Scrum Master, Developer, Tester, Business Analyst and UI/UX Specialist are represented.

Usually, people from the fields of UI/UX and development take part in a UI-Dev Session. The UI/UX specialists focus on two different aspects, which makes them ideally complementary: on the one hand, on the visual design of the UI and on the other hand, on the behavior of the UI components. The participating developers have a high affinity for front-end development. One of these people participates in each UI-Dev Session and has an overview of the items to be completed. A few days before, the UI/UX specialists in the Daily remind us that there will be a UI-Dev Session and that one more person from the development team is needed to assist. Depending on availability, we then decide who can support. The four-eyes principle applies on both sides (design and development), thus avoiding errors and extensive review rounds.

The list of UI errors to be solved is maintained, structured and prioritized by the UI/UX experts in project Wiki and is accessible to all team members. Confluence from Atlassian is used as a tool for this purpose. A selection of the themes is shown in Figure 2.

Example for list of UI errors
Figure 2: Excerpt from the list of UI errors 

Since our list of possible topics is currently quite extensive, regular sessions are necessary. A UI-Dev Session takes place once per sprint – i. e. once every three weeks – for two hours. If other topics are priorities in the sprint, the date can also be postponed at short notice, but ideally within the same sprint. The event will be held remotely with the help of Microsoft Teams, as the participants are distributed over the locations Dresden, Leipzig and Miskolc.

One or two days before the UI-Dev Session, the developers pick out a few points from the list in project Wiki and start preparing them. This includes, for example, marking the appropriate places in the code with to-dos in order to efficiently use the time in the UI-Dev Session for the actual adjustments.

At the beginning of the UI-Dev Session, all participants briefly go through the selected UI errors, which are to be corrected in the session. The topics are then dealt with from top to bottom. A person from the development department transmits the screen, opens the development environment, and the style guide in Figma. The other participants also open the style guide. One of the advantages of Figma is that the attendees can see where the other participants currently are in the style guide. In this way, the relevant positions can be quickly found by everyone. The specialists for UI/UX help the developers to find their way through the style guide faster and to find the relevant information. It is important that the people in the development team are able to look at the relevant places themselves and, for example, color values are not just “predicted”. This also trains the handling of the style guide.

The selected points will be dealt with gradually. If the selected UI errors are fixed faster than expected, new topics are added within the event. If selected topics remain open, they will be dealt with at the beginning of the next appointment.

During the preparation, or during the UI-Dev Session, it turns out that topics are more complex than initially thought, the developers communicate this to the specialists for UI/UX. These then move the theme from project Wiki to a separate backlog item in Jira, for example as an improvement or new-user story.

The results are presented to the testers in a follow-up appointment, which usually takes place one to two days after the UI-Dev Session and lasts a maximum of 30 minutes. This is important to determine if test cases are affected by the changes. Afterwards, the list of topics in project Wiki is updated by the specialists for UI/UX. The completed items are documented in tabular form in order to make it possible to track the changes made.

Excerpt from the list of fixed UI errors
Figure 3: Excerpt from the list of fixed UI errors

There doesn’t have to be a catch everywhere 

In our project, the use of UI-Dev Sessions has proven itself to optimize the appearance of the application quickly and easily. For us, the sessions primarily bring the following advantages: 

  • It fixes UI errors which have been known for a long time, but have been given only a low priority compared to the development of new features.
  • The lightweight method with little documentation effort can be easily integrated into our sprints.
  • We achieve a high level of compliance with the ZEISS style guide for user interfaces

In addition, the sessions strengthen collaboration and knowledge sharing within the team:

  • The collaboration between the development and UI/UX departments enables UI errors to be efficiently corrected, as developers can focus on the implementation and the UI/UX specialists can directly pass on design-specific specifications (e. g. font color, spacing) orally.
  • The experts for UI/UX get to know the implementation problems of the developers, which, for example, result from the technologies used.
  • With the experience gained from the UI-Dev Sessions, the UI/UX specialists will be able to make future design decisions even better based on the development effort.
  • The development team gets to know the Figma design tool, including the style guide, better.
  • The team of UI/UX specialists have the opportunity to explain design decisions and give developers an insight into what is important in the design.
  • The development team is working out a better awareness of subtleties in design and will thus be able to avoid UI defects in the future.

The list of benefits that the method brings to us is therefore long. But what’s the catch? For us there is currently none, and we are convinced of the method. We therefore recommend it to any team that has accumulated a large number of smaller UI errors in the project over time. The procedure is flexible and can be adapted according to the needs of the team. For example, the number of participants can be minimised or the timeframe expanded. 

The outlook 

Our goal is to continuously reduce the list of existing UI errors with the help of the UI-Dev Sessions. In order to keep the number of new UI errors as low as possible, we plan to integrate the UI-Dev Sessions into the Sprint during the implementation of a user story. In this way, new deviations from the design can be avoided from the outset.