Tester Teatime (Part 1): The myth of “historical software growth” put to the test

The “Tester Teatime” is a post format on this blog, in which topics are taken up that testers deal with on a daily basis. Certain problems or themes recur again and again, so we want to create a basis for explaining such phenomena and finding solutions to them. Discussions and new ways of thinking are also to be stimulated. In testing, we can learn a lot from each other by observing our daily lives! 

Moderator: Welcome to the first tester teatime! In an interview with testers from ZEISS Digital Innovation (ZDI), we will discuss exciting topics. 

Let us now turn to today’s subject. We talk to Sandra Wolf (SW), tester at ZDI. What is the myth of “historical growth” all about?  

SW: Well-versed testers will probably roll their eyes in the title of the article unwittingly. Because what sounds so steeped in history here is for many one of the most common answers that cross their paths in everyday work. As soon as a deviation is found in the software that cannot or should not be corrected immediately, this phrase is very popular. Examples of such deviations are poor user-friendliness and a constant inconsistency in the software to be tested. If the testers draw attention to an improvement here, they will be glad to receive the answer from team members from the areas of development, project management, etc. that this particular abnormality has grown so historically and that there is therefore no need for action. Often, the quality-driven testers are left puzzled and frustrated – after all, they are not shown any perspective for action in the sense of their role. What, then, is the meaning of the phrase “historical growth”? And how can this be handled professionally and effectively? That’s what this is about. 

two people having an interview
Image: Hendrik Lösch and Sandra Wolf during the interview at the Tester Teatime.

Moderator: Okay, that sounds interesting. What effects does this so-called “historical growth” have on the project and its software? 

SW: Many project participants are not aware of the importance of the problem. The “historical growth” is an avoidance, because in fact it hides the fear of a renewal of the respective software, which, however, inevitably moves closer with further ignorance. Especially the software components that have been expanded over the years develop more and more interactions, which means that the maintenance effort and the maintenance costs continue to rise. The different extensions may no longer fit together, have neither a uniform operation nor a uniform design. The quality of the software thus decreases congruently to customer satisfaction. At this point, innovative further development becomes increasingly difficult or no longer exists at all. Looking at all these facts, it quickly becomes clear that “historical growth” goes beyond mere words and can have serious consequences for the company, project, and software itself. Something must therefore be done here and a rethink must take place. 

Moderator: How can the project challenge “historical growth”? 

SW: The fear of many project members about “historical growth” is that the only way to solve this problem is by stomping software and re-build it completely. That this seldom makes sense, however, is something everyone can imagine. A complete renewal of software would also be very costly. The question now arises as to how this problem can be solved. Here I would like to point out that the task of the tester is not to eliminate or control the “historical growth” itself. As testers, we can only provide the impetus for change and should definitely do so, as we act as an important interface between development, technical department and project management. Our perspective on the whole process is worth gold here. We have a comprehensive overview of all steps of the process, as we act at the end of it. 

Moderator: What might a solution for “historical growth” look like? 

SW: Given this wide-ranging problem, it is advisable to advance the solution in smaller steps first. First of all, contact should be sought with a suitable contact person, e. g. the project manager, since a tester can never decide on such overarching project topics. In conversation with this person, attention can then be drawn to the consequences and risks of “historical growth”. The next step should then be defined jointly by the entire team. Perhaps it is not necessary to renew the entire software, but rather to limit it to certain parts of the application. Here, too, testers can contribute important knowledge from their daily lives. The development of the entire project must be considered and also which sub-areas arose first. The aim of the discussion is to draw attention to the problem and thus to provide the impetus for changes to restore the quality of the software. The form in which this happens – also by external companies, for example – must also be decided. For us as testers, however, this is an opportunity to contribute to the overall quality of the software to be tested. After all, the most important thing is that the very far-reaching theme of “historical growth” is no longer used as an excuse, but is finally placed in the project. 

Moderator: A solution can only be found together. In this context, we are also interested in the perspective of a developer. We now talk to Hendrik Lösch (HL), software architect at ZDI. What solutions do you see for “historical growth”? 

HL: I myself would not call it “historical growth”, because that implies that growth is in the past. I prefer to speak generally of “software evolution”. Because growth is always there, it only becomes problematic when the structures of the software mutate unintentionally. We also refer to these mutations as technical debts because they require future investments in restructuring and thus tie up funds that cannot be used for value-added measures. The longer you delay these restructurings, the harder they will be to carry out. This is probably where the “historical growth” comes from. It has grown over time and no one knows exactly why anymore. To solve this problem, a thorough analysis of the structures is required, in which experienced software architects determine the actual architecture. A target architecture is then created and a path is sketched to get from one to the other. With the Health Check, we as ZDI offer a suitable procedure. 

Moderator: Thank you, Sandra and Hendrik for your insightful presentations. So, we can sum up that the problem of “historical growth” is more of a myth. Growth is always there, but the lack of structure can make it problematic. However, this requires not only action, but also concrete strategies to deal with it. ZDI itself already offers them in its portfolio today. 

In the following articles, we will take up further problems from the everyday life of testers and discuss possible solutions. 

Distributed Work – An Experience Report on the Practical Application of Remote Mob-Testing

Within ZEISS Digital Innovation (ZDI) there is an internal training event at regular intervals – the so-called ZDI Campus Event. As employees, we present software development topics by means of lectures, workshops or discussion rounds. 

Katharina had already reported on collaborative testing methods at the last ZDI Campus, and also published blog articles about Remote Pair Testing and Remote Mob Testing. Therefore, we wanted to continue the topic at the next ZDI Campus and offer a workshop on the topic of collaborative testing methods. However, due to the Covid-19 pandemic, the next campus had to be held online. Nevertheless, we wanted to offer several participants the opportunity to apply a test method using a practical example, and therefore decided to hold a remote Mob-Testing Workshop.  

But there was a challenge: We had never worked with so many people in the mob remotely! 

Structure of the workshop 

As described in the blog post about Remote Mob Testing, it makes more sense to supervise small groups of four to six people. Distributed work may otherwise lead to more frequent delays due to technical problems (e. g. poor connection, poor sound quality), which could reduce the time required for the current navigator. As a facilitator, you can also keep an overview of smaller groups and the participants can take on the role of navigator or driver more often.
(Zur Erläuterung der verschiedenen Rollen siehe ebenfalls Blogbeitrag Remote Mob Testing

Our setting looked like this:

  • Microsoft Teams as a communication platform  
  • Easy to understand test object (website)
  • 3 Facilitators  
  • 39 participants from the fields of QA, Development and Business Analysis 
  • Timeframe: 1.5 hours  

Because we wanted to give everyone the opportunity to get to know the Mob-Testing themselves by means of a practical example, we did not set a limit on participants in the run-up to the workshop. In the end, all three facilitators had more than 12 participants.  

person in front of a screen having a video conference
In remote mob testing, all participants take on the active role of the navigator once.

As a test object, we chose a simple website so that everyone could concentrate on communicating or getting to know the test method and not have to acquire additional knowledge about the application. 

Feedback 

Already during the course of the workshop, we noticed that waiting times for an active role (Driver or Navigator) could be perceived as unpleasant.   

This was also mentioned in the feedback round. Therefore, we recommend that the mob assists with test ideas, because it doesn’t mean that you can sit back and wander with your thoughts as a mob member. This also avoids double testing or having to admit that you were not paying attention.  

In some cases, it was difficult for some participants to focus purely on the role of the driver – executing the instructions of the navigator, and holding back on their own test ideas. However, the participants got used to it after some remarks by the facilitator. This aspect of Mob-Testing was considered to be very positive, because everyone in the role of the navigator can speak and contribute their own test ideas.  

However, the question arose as to why the roles Navigator and Driver could not be combined. The following can be said: It promotes the learning process when a member articulates step-by-step what he or she intends to do. In this way, more participants are involved in this active role. Communicating the destination makes it easier for the participants to follow the ideas of the navigator. Otherwise, some steps may be carried out too quickly, and with too little explanation. This would lose traceability and make it difficult for the mob to get actively involved.  

There was other positive feedback on the division of roles and the whole process. According to the respondents, the test approaches are partly obtained by the path taken by the previous navigator, and therefore viewed the test object from a variety of angles. For this reason, it is always useful to invite participants with different skills, depending on the purpose of the Mob-Testing session. This increases the learning process, and the number of test cases. The simple sample test object also helped to focus on the method and to internalize it. 

The collaborative work in Mob-Testing was highlighted very positively. This is how the agile thought is experienced.  

Solution approach for a large number of participants 

The problem of too many participants could be solved by limiting the size of the group in advance or by introducing a new role: the role of spectator. A separation would then be made between active participants on the one hand and spectators on the other. Participants would follow the procedure described above, and follow the roles distribution (navigator, driver, mob) and the role change. The spectators would only observe and not participate. Comments by them would not be allowed either, because this could disturb the active participants with a large number of spectators. 

Conclusion 

All in all, the workshop was very well received at the Campus event, and showed that it is very possible to use Mob-Testing remotely, and therefore for distributed work. This means that cooperation can be maintained, even if it is not always possible to meet on site.  

This post was written by:

Maria Petzold

Maria Petzold has been working at ZEISS Digital Innovation since 2010. As a test manager, her focus is on software quality assurance. She was able to gain her test experience especially in medical software projects.

See author’s posts

MAUI – More than an island

In May 2020, Microsoft released a new, unified UI platform for all systems under the resounding name MAUI or in full “. NET Multi-Platform App UI”, which is expected to be launched in November 2021. But what exactly is behind this name? This question is to be answered in this article. The platform will be presented, the technical background explained and, above all, the potential of the technology will be highlighted.

To understand MAUI as a platform, you need to know Xamarin. Xamarine (and in particular Xamarine.Forms is a platform for developing native apps for iOS, Android, and UWP with C#, XAML, and .NET. You can create an app from a code base for all supported operating systems. Thus, the development effort for different operating systems is significantly lower compared to the native coding of the applications. Currently, the various SPA frameworks for the browser are actually the only technology that offers comparable portability at a similar overall effort.

graphic explaining MAUI
Figure 1: Xamarin.Forms allows you to create native apps for all supported operating systems from a single code base. MAUI is the direct successor of Xamarin.Forms.

But what is this mysterious MAUI and what does it have to do with Xamarin? The answer to this question is quite simple: MAUI is the new Xamarin or, more precisely, its direct successor, which is shipped with .NET 6 for the first time.

Just like Xamarin.Forms, MAUI can create apps for all supported operating systems from a single project and with the same code base. Installation packages are generated from the code, which can then be installed on the different platforms. Officially, Microsoft supports Android from version 10, iOS, macOS and of course Windows, both natively and as UWP app. In addition, there will be a community implementation for Linux operating systems and an implementation for their Tizen platform provided by Samsung. The project and build system will be standardized for all platforms and the creation of the apps will be possible via both Visual Studio and the .NET CLI

Another feature will be the sharing of resources such as images or translations. These should be automatically converted by MAUI into the corresponding native formats and integrated into the created packages. You will also be able to access the APIs of the respective operating system at any time. For this purpose, there should be a special folder in the project, under which the native code hooks are stored and which are then automatically integrated into the package when compiling.

All functionalities available in the .NET standard, such as Dependency Injection, should also be able to be used for a MAUI app. By using C# and XAML it will also be possible to use corresponding design patterns such as the widely used MVVM pattern. Also new is the support for the Model View Update Pattern, a pattern borrowed from Elm, which is intended to be able to display a unidirectional data flow analogous to Redux. Also, Microsoft’s web-based client technology Blazor is to be supported.

Unfortunately, MAUI will not be officially available until the launch of .NET 6 in November 2021. In addition, parts of the framework have been moved to .NET 7 and thus to 2022. The official support for Blazor and the MVU pattern should be mentioned here. Since MAUI is the official successor of Xamarin, it will be supported with the release of .NET 6 for another year and then discontinued.

Microsoft’s strategy for the future of UI development with the .Net Framework seems clear: MAUI is the new “First Class Citizen” when it comes to creating native user interfaces.

At first it all sounds like Microsoft wants to integrate native cross-platform support into the .NET framework. However, this plan will only work if MAUI is accepted by the developers. Due to the very late release and a lot of excellent open-source alternatives such as Uno might not be able to establish MAUI in the end. At the moment, therefore, one can only wait and see what the future holds.

However, MAUI would actually have the potential to become the only UI technology in the .NET framework and eventually even replace WPF. It offers a variety of functionalities to increase productivity in development – “Write once, run everywhere” is the motto of the hour. This could significantly reduce costs and development time for apps on all supported platforms. Just imagine the joy of the customer if you could create not only a Windows application, but also the corresponding mobile apps for only a small amount of development effort. Of course, this is an idealized idea, the devil is in the detail as you know. Migration of existing WPF applications is difficult, for example, because the long-awaited XAML standard, which was supposed to define a standardization of elements and tags for all XAML dialects, has apparently been dropped and MAUI and WPF will not be able to be exchanged seamlessly due to different elements.

But if the technology actually delivers on its promises, developers deploy it widely, and MAUI becomes as high-class as Microsoft promises, a cross-platform revolution under .NET could be in the offing.

Exploratory Testing (Part 1) – The Test for Lazy People or the Crown of Creation in Terms of Testing?

We are testers and love test methods, where the focus is on human strengths like creativity and collaboration. Nevertheless, we do not want to act without rules, or rather without framework conditions, or plead for an abstinence of documentation.

We have already encountered some prejudices about exploratory testing, therefore we have asked ourselves the question how these come about. In this article we will talk about those prejudices in more detail in order to get rid of them. But first we will take a closer look at exploratory testing.

What is exploratory testing?

Exploratory testing can do much more than just working off test cases in a monotonous way: It promotes creative thinking during a test session, blasts limitations and satisfies curiosity. At the same time, it requires discipline regarding reaching the set goal during a test implementation and with regard to documentation.

ISTQB (independent panel of experts for standardized training for software testers) describes exploratory testing in its glossary as a „an approach to testing whereby the testers dynamically design and execute tests based on their knowledge, exploration of the test item and the results of previous tests“ (archived version 3.4 of the ISTQB® GTB Glossary). Unlike the “classic“ testing with prepared test cases, it is therefore an intuitive test approach where each previous idea leads to the next one. There are various tools to achieve a new starting point, to develop ideas and create more structure. These include for example the test charter, the heuristics, the test oracle or also the test tours.

Hand holding a lightbulb from which interconnected dots rise
Figure: Exploratory testing – an intuitive test approach

In summary, the tester creates the tests through experience, intuition, curiosity and structured approach. Nevertheless, this type of testing is not only well-suited for experienced testers but also for project newcomers who are just getting to know the test object and/or the procedure. To take the person by the hand a goal should be defined or an overview of the object should be created. This can be done either in direct exchange via pair testing or in groups via the mob-testing approach.

Prejudices against exploratory testing

But what prejudices are there now against which exploratory testing must stand its ground?

Exploratory testing is often notorious for having no structure; for not being comprehensible; and being unquantifiable – plain and simple “I-will-click-around-and-see-what-happens”. The statement of obsolete documentation further complicates it to charge for exploratory testing in client projects. This is an understandable reason, if it is assumed that exploratory testing is documentation-free and thus the test performance is not traceable. The following statement and the killer argument are therefore also made quickly that there is no time available for this test.

In the following, we will take a closer look at the allegations just presented as well as other negative statements and comment on them.

Prejudice Number 1: “Exploratory testing is not comprehensible.” / “I cannot test exploratively-tested content again.” / “How am I supposed to do a bug retest?” These statements made us curious. When asked what is meant by exploratory testing, we often received the answer that it is used for spontaneous clicking around in the test object. It should promote getting to know the test object or test further variants that were not considered in the previous test-case creation. And thus, it is carried out in an unplanned as well as unstructured way. No documentation was mentioned. This explains how the prejudice comes about that this test method is not comprehensible, and that tests and bug fixes are difficult to retest. It is absolutely true that exploratory tests support spontaneous test taking. However, care must be taken to always document the tests steps, expected results, or even deviations, and thus make the tests traceable. The depth of documentation must be defined individually for each project according to its framework conditions. As a conclusion we maintain that: Exploratory testing does not mean not to document.

To counteract the first prejudice and to bring structure into exploratory testing, session-based test management can be used. During a session, the test object is checked with maximum flexibility and freedom. The framework conditions are set by means of a defined standpoint and a charter (purpose and procedure for a session). In addition to some other components, this also includes documentation by means of session reports in which the test steps carried out, deviations, the test object, and other information can be recorded. These can be adjusted depending on the person or project being tested. Various examples of session reports can be found on the Internet.

At this point, a special application should be mentioned: Exploratory testing in a regulated environment. Documentation plays a central role here. The prejudice here: Exploratory testing is impossible or very difficult to implement in a regulated environment due to regulatory requirements. In a further article, we will show how to implement it in spite of the requirements.

Prejudice Number 2: “We have no time for exploratory testing” / “First of all, we have to work through our prepared test cases before we can test exploratively.“

In order to implement the exploratory test method actively in the project, one must be prepared to adapt the previous test procedure and recognize the advantage of saving time. Because if you see the exploratory test only as a supplement, without an alignment of the test structure used up to now, this procedure is seen only as a burden or as an annoying addition, which results in the no-time prejudice. However, if we proceed as envisaged in exploratory testing, and if the test case creation takes place at the same time as the test case execution, the now superfluous creation of test cases that are not executed at the end, and thus represent a waste of time, is eliminated. The time gained allows the tester to develop new test ideas and thus to test the test object more intensively. For example, debriefings can be held to share lessons learned with other team members. This time saving can also be used to increase the test automation rate.

Prejudice Number 3: “Documentation is all well and good, but exploratory testing is not sufficient.”:
This statement is correct, if it is meant that exploratory testing should be considered as a complete test approach in its own right. In addition to the exploratory tests, at least automated tests should be defined and executed (for example unit tests, integration tests and GUI tests). If the project has a high degree of automation, it is even possible to perform the manual testing purely exploratively. Alternatively, the exploratory method can be used as a complement to the classical testing approach to make manual testing easier, more flexible and freer. How much exploratory testing is applied in a project must always be carefully examined. It is not always possible to make all manual tests exploratory. This can be due to various factors, e. g. an insufficient degree of automation, or a lack of skills on the part of the testers. In our view, they should have skills such as creativity, conscientiousness, initiative, personal responsibility, an understanding of quality, and social aspects such as communication and teamwork skills.

Conclusion

In this article, we have shown the advantages of exploratory testing and put together arguments to refute the existing prejudices against it. From our point of view, it is worthwhile to rethink the previous test procedure and to engage in exploratory testing in order to exploit its positive effects. Of course, each project should be considered on its own merits and the possibilities for exploratory testing should be determined – without, however, ruling this out completely from the outset. Negative statements or experiences should be questioned where appropriate. When choosing the method and its execution, the skills of the testers must always be considered. At this point we would like to point out that documentation is also essential for exploratory testing.

This post was written by:

Katharina Warak

Katharina Warak works as a senior software test engineer at ZEISS Digital Innovation. She is responsible for automation at the API and GUI level. In addition, she helps customers in the projects to learn more about the collaborative test methods and to use these methods to improve their way of working. She also likes to share this knowledge at international and national conferences.

See author’s posts

Christmas party in times of Corona: The 2nd ZEISS Digital Innovation Online Campus Event

As early as spring we at ZEISS Digital Innovation (ZDI) started to plan our Christmas party which normally takes place on the Friday before the 1st Sunday of Advent. In 2020 we quickly came to realize that a traditional Christmas party where all the employees normally meet in Dresden and celebrate together would unfortunately not be possible this time. Cancelling the Christmas party was not an option for us either, so we started to search for a new format.

We came back to the idea of our 1st ZDI online campus event which had already received a lot of positive feedback. It was especially important to us this time that we created an all-encompassing festive mood, in addition to the intern knowledge sharing. But how could we bring this home to our colleagues?

A map showing all the locations from which our employees were connected.
Figure 1: Our colleagues participated in the online campus event from a wide variety of locations.

A small Christmas surprise for all employees

To enhance a Christmas atmosphere all employees received a small Christmas parcel by mail with the request that it should only be opened on the day of our common online campus event.

All 345 parcels were personally packed by the organizational team to a background of festive Christmas music. We were happy that the Christmas surprise arrived in time at the homes of all our colleagues in Germany and Hungary.

So all were on time for our 2nd ZDI online campus event with Christmas goodies, mulled wine and traditional Christmas spices to create a delicious punch, an Advent calendar, a Christmas party hat and other small gifts and off we go!

2nd online campus event including online team game and Christmas celebration

Using the positive experiences from our 1st ZDI online campus event this time again we offered different lecture slots from and for our colleagues during the whole day. The technical realization was done again via Microsoft Teams. We had 29 lectures altogether covering a variety of different topics such as Java, .NET, Cloud, Usability, Agile and Web. Through lessons learned the first time around we adapted the length of the lecture slots increasing them to 45 minutes instead of 30 with 15-minute breaks in between to change. So everyone had time to breathe and mentally prepare for the next one.

Timetable of all presentations at our 2nd ZEISS Digital Innovation online campus event.
Figure 2: A total of 29 presentations in 6 parallel slots were given on related topics.

After a longer lunch break our online team game took place, 28 teams took part, consisting of nine to ten players in each one. When we put together the teams we paid great attention to the make-up of each team ensuring that they were as heterogeneous as possible regarding the level of maturity, location and business area. Each group had their own team room for solving the tasks, it was a super way to get to know each other. The activities included among other things general knowledge questions, charades and quizzes as well as practicing the Christmas team contribution, which every team presented live at the end of our online campus event within the framework of our collective Christmas party.

We kicked off our Christmas Celebration together by drinking a toast! After that each team started to show their live Christmas contributions. There were musical and poetic presentations and performances including Christmas greetings in different languages together with Christmas film tips and recommendations. All teams were extremely creative and the team chat was also available and used actively adding another dimension to our party. So despite the wide distribution and accompanying physical distance, there was nevertheless a contemplative Christmas mood that was felt by all.  

Conclusion

We all knew that an online format and a virtual Christmas celebration could never replace the feeling of a real on-site Christmas party with ice skating, delicious food, dancing and the relaxed atmosphere amongst one other. Still, we were pleasantly surprised how much festive, community spirit could nonetheless be felt online. It has shown us once more that we are a strong team acting in unison and despite Corona we are ready to make the best of the current situation breaking new ground together.

It is uncertain if we may celebrate a summer or Christmas party with all the colleagues in Dresden on-site this year. But one thing is already for sure: our new online campus event format with team games is certain to outlast Corona.

Capturing and Streaming Database Changes with Debezium and Apache Kafka (Part 1) – Theory

Databases, and relational databases in particular, play an important role in just about every business environment. They are used to manage both the master data of employees, clients, etc. and the constantly changing transaction data of the company. The requirements for the respective companies may change for a variety of reasons so that the changes in the transaction data, such as bookings in a system, have to be processed in real time in different processes and applications. Regular polling of a source table by an application is no longer sufficient for this purpose. Instead, the application has to be notified of change events in a table, and the information has to be provided to the application in processed form immediately.

Using Debezium and the Apache Kafka streaming platform makes this possible.

Capturing changes in the data

First, let us briefly lay out the basic possibilities of capturing changes in the data in relational databases. The relevant literature describes four methods summarized under the term Change Data Capture. These methods will be presented below.

The easiest way to capture changes is a line-by-line and column-by-column comparison of the database table with an older version of the same. Obviously, this algorithm is not particularly efficient, especially for larger tables, but easy to implement. Selecting only the changed datasets by means of a well-executed SQL query is another option. In practice, time stamps are most commonly used to indicate the data and time of the last change for each dataset. When you search for data changes in the table, you can select all the datasets where the time stamp is more recent than the date and time of the last comparison. A minimal SQL statement could, for example, look like this:

SELECT * FROM [source_table] 
WHERE last_updated < [datetime_of_last_comparison]

The problem with this approach is that the database table must already include a column for such time stamps, which will likely not always be the case. In other words, there are certain requirements for the data schema. Furthermore, it is very difficult to capture deleted datasets with this method because their time stamps are deleted as well.

A third option that can solve this problem is implementing a database trigger that is activated by every INSERT, UPDATE and DELETE on the table. In addition, monitoring changes to the database schema by means of a trigger is also conceivable. Such a trigger could then write the captured data changes to a table specifically dedicated for this purpose.

The fourth and last option, the so-called log scanning, plays a particularly important role in this blog post. Most database management systems (DBMS) keep a transaction log that records all the changes made to the database. This is primarily used to restore the consistent state of the database after a failure, e.g. due to a power outage or damage to the physical data carrier. For the change data capture, we have to somewhat repurpose the transaction log: By reading the log file, it is possible to capture changes in the datasets of a specific source table and to process them.

The big advantage of log scanning is the fact that reading the transaction log does not generate any overhead in the database the way querying or executing a trigger after each change does. Consequently, it does not compromise the performance of the database. Furthermore, all kinds of changes in the data can be captured: inserts, updates and deletions of datasets as well as changes to the data schema. Unfortunately, the structure of such transaction logs is not standardized. The look of the log files of different database vendors vary considerably, and sometimes even different versions of the same database management system differ. If you want to capture data changes from several databases provided by different vendors, several algorithms need to be implemented.

This is where Debezium comes in.

What is Debezium?

Debezium is not actually stand-alone software, but a platform for connectors that implement the change data capture for various databases. The connectors use log scanning to capture the data changes and forward them to the Apache Kafka streaming platform.

Each connector represents the software of a specific DBMS that is able to establish a connection to the database and read the change data there. The exact operating principle of each connector differs depending on the DBMS, but generally speaking, it first creates a snapshot of the current database and then waits for changes in the transaction log. To date, Debezium has developed connectors for the MongoDB, MySQL, PostgreSQL and SQL Server databases. Additional connectors for Oracle, Db2 and Cassandra are still at the development stage, but already available.

The most common way of using Debezium is in combination with Apache Kafka and the corresponding Kafka Connect framework. Apache Kafka is an open-source, distributed streaming platform that is based on the publisher-subscriber model. It enables applications to process data streams in real time and to exchange messages between several processes. With Kafka Connect, it is possible to implement matching connectors that can both write data to Kafka (source connectors) and read from Kafka (sink connectors). As you may have guessed, Debezium provides source connectors that stream the captured change events to Kafka. Applications that are interested in the change data can read them from the Apache Kafka logs and process them.

Outline of the architecture for the use of Debezium
Figure: Outline of the architecture for the use of Debezium 1

Recently, it has become possible to use Debezium not only in combination with Kafka Connect, but as a standalone application. This means that the users are no longer bound to Kafka and can now forward change events to other streaming platforms such as Kinesis, Google Cloud Pub/Sub, or Azure Event Hubs. The change data that Debezium captures from various databases have to be provided to the applications that want to process them (consumers) in a standardized format. Debezium uses the JSON format for this. Each change event comprises two parts: a payload part and a preceding schema that describes the structure of the payload. The payload part contains, for example, the operation that was carried out on the dataset and, optionally, the content of the line before and after the change occurred. The primary key of the database is given a specific designation which is important for the allocation of the event to a partition in Kafka. We will elaborate on this in the second part.

Furthermore, Debezium places a high value on fault tolerance for the change data capture: If a connector crashes, reading continues after the reboot at the position in the transaction log that has been processed last. Similarly, if the connection to Apache Kafka fails, the change data are stored in the cache until the connection has been reestablished. This concludes our overview of the basics of the use of Debezium. In the next part of the series, we will present a concrete example where Debezium streams change events from a sample table in the SQL Server.


1 The outline is based on the architecture outline provided in the Debezium documentation.

WCF Alternatives (Part 1) – Introduction

The Windows Communication Foundation (WCF) is a communication platform for the creation of distributed applications developed by Microsoft for the .NET Framework. It was introduced with the .NET Framework 3.0 in 2006, replacing .NET Remoting. With WCF, the various aspects of distributed communications are logically separated, and different communication technologies are combined into a standardized programming interface. This makes it possible to focus on the business logic without having to deal with the connection of the different communication technologies.

The structure of WCF

The following communication technologies were unified with the release of WCF.

WCF alternatives
Figure 1: Unified communication technologies

The structure of a WCF application is based on three questions (where, how, what).

The first question, “Where”, describes the address at which the application can be found in order to communicate with it, e. g.:

  • http://localhost:81/DataInputService
  • net.tcp://localhost:82/TcpDataInputService
  • net.pipe://localhost/PipeDataInputService
  • net.msmq://localhost/MsMqDataInputService

The second question, “How”, describes which protocols and which encoding, the so-called bindings, are to be used for the communication. They are defined in the *.config of the application, allowing them to be modified at any time without the need to recreate the application. WCF supports nine different bindings:

  • Basic Binding (BasicHttpBinding)
  • TCP Binding (NetTcpBinding)
  • Peer Network Binding (NetPeerTcpBinding)
  • IPC Binding (NetNamedPipeBinding)
  • Web Service Binding (WSHttpBinding)
  • Federated WS Binding (WSFederationHttpBinding)
  • Duplex WS Binding (WSDualHttpBinding)
  • MSMQ Binding (NetMsmqBinding)
  • MSMQ Integration Binding (MsmqIntegrationBinding)  

The last question, “What”, uses various contracts to define the endpoints and data types to be provided. The endpoints are defined by ServiceContracts, and the data types by DataContracts.

Example of a WCF ServiceContract and DataContract:

[ServiceContract]
public interface IDataInputService
{
    [OperationContract]
    int CreateUser(User user);
 
    [OperationContract]
    int Login(User user);
 
    [OperationContract]
    List<Time> GetTimes(int userId);
 
    [OperationContract]
    void AddTime(Time time, int userId);
 
    [OperationContract]
    List<string> Projects();
}
 
[DataContract]
public class User
{
    [DataMember]
    public string Name { get; set; }
 
    [DataMember]
    public string Passwort { get; set; }
}
 
[DataContract]
public class Time
{
    [DataMember]
    public DateTime Start { get; set; }
 
    [DataMember]
    public DateTime End { get; set; }
 
    [DataMember]
    public string Project { get; set; }
 
    [DataMember]
    public int uId { get; set; }
 
    [DataMember]
    public int Id { get; set; }
}

WCF is very popular because of its flexibility achieved with the separation and its versatility, which is why the platform is readily used in numerous projects.

Why is a migration necessary?

When .NET Core was first announced in 2016, WCF was already no longer included. WCF is not part of the subsequent .NET Core releases, including the most recent .NET 5.0.

For Microsoft, the “W” in WCF, which stands for Windows, would probably be the greatest issue in porting. In order to do justice to .NET Core, a cross-platform solution would have to be found for this purpose. One of the problems in this context are the Windows-specific operating system libraries currently used, e.g. for Socket Layers or cryptography.

Even though the developer community is asking for the integration of WCF in .NET Core, it is unlikely that Microsoft will provide this in the foreseeable future.

The future with gRPC and Web API

To make an existing project sustainable, or generally use the advantages of .NET Core, porting to .NET Core should be the aim. This is particularly useful for projects that are being actively and continuously developed. If WCF is used in a project, this poses an additional challenge in porting. First, an alternative needs to be found, and then a preparatory transition of WCF is required. Microsoft generally recommends two alternatives, gRPC and Web API, to replace WCF.

We are going to present these two alternatives in a series of blog posts, discussing the respective particularities and challenges of the migration. More articles will follow shortly.

Serverless Computing: An Overview of Amazon’s and Microsoft’s Services

With the serverless model, application components such as databases or data processing components are provided and operated, automated and demand-based, by the cloud service provider. The cloud user is responsible for configuring these resources, e.g. with their own code or application-specific parameters, and combining them.

The costs incurred depend on the capacities used, and scaling takes place automatically based on the load. The cloud service provider is responsible for the provision, scaling, maintenance, high availability and management of the resources.

Serverless computing is particularly convenient for workloads that are difficult to anticipate or are short-lived, for automation tasks, or for prototypes. Serverless computing is less suitable for resource-intensive, long-term, and predictable tasks because in this case, the costs can be significantly higher than with self-managed execution environments.

Building blocks

Within the framework of a “serverless computing” Advent calendar, we compared the cloud services of AWS and Azure. The windows open under the hashtag #ZEISSDigitalInnovationGoesServerless.

CategoryAWSAzure
COMPUTE
Serverless Function
AWS LambdaAzure Functions
COMPUTE
Serverless Containers
AWS Fargate
Amazon ECS/EKS
Azure Container Instances / AKS
INTEGRATION
API Management
Amazon API GatewayAzure API Management
INTEGRATION
Pub-/Sub-Messaging
Amazon SNSAzure Event Grid
INTEGRATION
Message Queues
Amazon SQSAzure Service Bus
INTEGRATION
Workflow Engine
AWS Step FunctionsAzure Logic App
INTEGRATION
GraphQL API
AWS AppSyncAzure Functions mit Apollo Server
STORAGE
Object Storage
Amazon S3Azure Storage Account
DATA
NoSQL-Datenbank
Amazon DynamoDBAzure Table Storage
DATA
Storage Query Service
Amazon Aurora ServerlessAzure SQL Database Serverless
SECURITY
Identity Provider
Amazon CognitoAzure Active Directory B2C
SECURITY
Key Management
AWS KMSAzure Key Vault
SECURITY
Web Application Firewall
AWS WAFAzure Web Application Firewall
NETWORK
Content Delivery Network
Amazon CloudFrontAzure CDN
NETWORK
Load Balancer
Application Load BalancerAzure Application Gateway
NETWORK
Domain Name Service
Amazon Route 53Azure DNS
ANALYTICS
Data Stream
Amazon KinesisAnalytics
ANALYTICS
ETL Service
AWS GlueAzure Data Factory
ANALYTICS
Storage Query Service
Amazon AthenaAzure Data Lake Analytics

We compiled an overview of the above-mentioned services and their characteristics, including some exemplary reference architectures on a poster (english version follows). This overview offers a simple introduction to the topic of serverless architecture.

Figure 1: Preview poster “Serverless Computing”

We will gladly send you the poster in original size (1000 x 700 mm). Simply send us an e-mail with your address to info.digitalinnovation@zeiss.com. Please note our privacy policy.

Best practices for serverless functions

Each function should be responsible for a single task (single responsibility principle):  this improves maintainability and reusability. Storage capacity, access rights and timeout settings can be configured more specifically.

As the allotted storage space of a Lambda function is increased, the capacity of the CPU and the network increases as well. The optimal ratio between execution time and costs should be determined by way of benchmarking.

A function should not call up another synchronous function. The wait causes unnecessary costs and increased coupling. Instead, you should use asynchronous processing, e.g. with message queues.

The deployment package of each function should be as small as possible. Large external libraries are to be avoided. This improves the cold start time. Recurring initializations of dependencies should be executed outside of the handler function so that they have to be executed only once upon the cold start. It is advisable to define operative parameters by means of a function’s environment variables. This improves the reusability.

The rights to access other cloud resources should be defined individually for each function and as restrictively as possible. Stateful database connections are to be avoided. Instead, you should use service APIs.

Amazon Athena: SQL – Without the Database

Many companies face the problem that data may be important for new applications years later, but when that time comes, they have long been deleted, or their structure has since been changed several times. Furthermore, data have often been selected, aggregated or transformed before they are first saved, i.e. they are no longer complete when they are to be used later.

For data-intensive projects in the field of data science or AI in particular, suitable data must therefore first be collected again, causing significant delays in the planned projects.

How can data lakes help?

Data lakes are an architectural pattern that aims at making data from various applications available in a centralized ecosystem in the long term. Data from every segment and department of a company are stored in a central location if possible. Unlike with traditional data warehouses, however, the raw data are always stored as well, often in an object storage system such as S3.

The advantage of this method is the fact that the information is available in its entirety, without being reduced or transformed when they are first stored like they are in traditional data warehouses. Consequently, the central data pool does not have a structure that is tailored for specific user requirements, i.e. in this case, the consumers have to deduce the meaning of the data themselves.

In order to be able to efficiently exploit the advantage of data lakes, they should be provided on a cross-departmental level. This way, the data can be retrieved anywhere they are needed.

It is possible to store the data in different zones, allowing access with different levels of abstraction. For data scientists, for example, low-level tools such as Athena are used to gain in-depth, detailed insight into the data pool, whereas more specialized data marts are preferable for technical departments.

What does Amazon Athena offer?

Amazon Athena allows for SQL queries to be executed directly on (semi-)structured data in S3 buckets, without the need for a database with a defined structure. Preparatory ETL (Extract Transform Load) processes as we know them from traditional data warehouses are not required for the work with the raw data, either.

As Amazon Athena is a serverless service, no infrastructure has to be provided. This happens automatically in the background, and is transparent for the user. On the one hand, this reduces the effort and specialist knowledge required, and on the other hand, using this service only causes costs per gigabyte of the data read from S3.

Lecture at online campus event (German only)

The following video of our first online campus event gives more detailed insight into the technical background and the possibilities of application and optimization. It shows discussions about practical experiences and a brief live demonstration in the AWS console.

Web Components (Part 1) – Building Your Own Components

So-called “web components” are one way of building reusable UI components for web applications. Unlike common single-page app frameworks such as React or Angular, the component model is based on web standards. Since SPA frameworks can, in fact, do far more than just build components, web components do not compete directly with the established frameworks. They can, however, be a useful addition. Whenever components are meant to be reused for applications with different technology stacks, web components can be very useful indeed.

Still, using web components in single-page applications presents some difficulties when you go into detail: While the integration into Angular applications is relatively simple, a few things have to be observed, in particular when using React applications.

Whether the “fault” lies on React or the web component standard depends on one’s point of view and is not easily answered. Furthermore, there are some aspects where web components are disadvantageous even with respect to their core competency of building components because they are unnecessarily complicated or inflexible, e.g. compared to React.

Figure 1: Web components and SPA frameworks

This series of blog posts deals with these and other aspects regarding the interaction of web components and SPA frameworks, in particular React. The first part of the series focuses only on web components, what the term means, and how to build web components.

What are web components, and how do you build your own components?

The term “web components” refers to several separate HTML specifications that deal with various aspects of the development of one’s own components. Consequently, there is no such thing as “one” standard for web components; rather, it is a combination of several specifications.

The two most important ones are “Custom Elements” and “Shadow DOM”. The Custom Elements specification describes the JavaScript base class “HTMLElement”, among others, from which individual components have to be derived. This class provides numerous lifecycle methods that allow you to respond to various events in the component’s life cycle. You can, for example, program a response to the component being integrated into a document or attributes of the component being set. The developers of a component can then update the presentation of the component. Custom Elements furthermore comprise the possibility to register individual component classes under a specific HTML tag so that the component is then available throughout the entire document.

“Shadow DOM” means a method that allows for a separate DOM tree, which is largely isolated from the rest of the document, to be created for a component. This means, for example, that CSS properties set globally in the document do not take effect in the Shadow DOM, and on the other hand, CSS definitions within a component do not affect other elements in the document. The goal is to better encapsulate the components and avoid unwanted side effects when integrating foreign web components.

The following code block shows a simple Hello World component that comprises a property for the name of the person to be greeted.

class HelloWorld extends HTMLElement {

    person = ""

    constructor() {
        super();

        this.attachShadow({mode: "open"})

        this.shadowRoot.innerHTML = `
            <div>
                <p>Hello <span id="personGreeting"></span></p>
            </div>
        `
    }

    static get observedAttributes() {
        return ['person']
    }

    attributeChangedCallback(name, oldValue, newValue) {
        if(name === "person") {
            if(this.person !== newValue) {
                this.person = newValue
                this.update()
            }
        }
    }

    update() {
        this.shadowRoot.querySelector("#personGreeting").textContent = this.person
    }

}
window.customElements.define("app-hello-world", HelloWorld)

Firstly, a separate Shadow DOM tree is created for the component in the constructor of the component. “Mode: open” ensures that it is possible to access the DOM tree of the component from the outside with JavaScript despite the Shadow DOM barrier.

Then, the “shadowRoot”, i.e. the root node of the Shadow DOM, is created according to our requirements—in this case, with “innerHTML”.

With “observedAttributes”, we describe which attributes the component is supposed to have and/or which attributes we want to be notified of (we can also specify standard attributes such as “class” at this point).

The notification is done by means of the “attributeChangedCallback” method, with the name of the changed attribute and both the old and the new value as parameters. Since we only specified one attribute in “observedAttributes” in this case, checking the name of the attribute is not really necessary. In the case of several attributes, however, it is important to always check which attribute has been changed in each case.

In our case, we first check whether the new value is actually different from the previous one (we will see how this can happen later on). Then, we set the “person” property that we created as class variable to the value of the submitted attribute.

To update the presentation of the component, the “update” method was created in this example. This method is not part of the Custom Elements standard, but only serves to gather the update logic in one place in this case. We retrieve the previously created span element with the ID “person” from the Shadow DOM and set its text to the value of the “person” property.

Figure 2: Shadow DOM

The code example shows how, in a final step, our component class is registered with the tag name “app-hello-world”. It is important that the name comprises at least one minus sign. This rule was defined in order to avoid possible name collisions with future standard HTML tags. Choosing a meaningful prefix for one’s own components has also proven to be useful to prevent collisions with other component libraries as far as possible (the prefix “app” used in the example is, in fact, not a very good example in this respect). However, there is no sure way to prevent conflicts.

We are now able to submit simple data to the component by means of attributes. “Attributes” have a few more particularities and pitfalls, but we are going to deal with those in the next part of this series of blog posts. For this general introduction, we will leave it at that.

Slots

The so-called “slots” are another important feature of web components that will be dealt with again in a later part of this series. Slots allow for HTML snippets to be submitted to a component. The component then decides how to present the submitted elements. For example, if we want to build a message box that presents both a text and an icon inside a frame, it is advisable to submit the message text to the component by means of a slot instead of an attribute. This way, we are not limited to plain text, but we can use any HTML content we want.

Here is an example of how this can look in the application:

<app-notification-box>
	<p>Some Text with additional <strong>tags</strong></p>
</app-notification-box>

We only have to write the HTML tags we want as child elements. Within the component, there has to be a <slot> element in the Shadow root for this purpose. When the component is rendered, the submitted content is then displayed instead of the slot element.

<div>
    <div>Icon</div>
    <div id="content">
        <slot></slot>
    </div>
</div>

A component can also contain several slots. In order for the browser to be able to decide which HTML elements to assign to which slot, so-called “named slots” have to be used in this case, i.e. the slots are given a specific name attribute. A component must not contain more than one slot without a name attribute. This one is called the “default slot”. Here is an example of how this can look in the component:

<div>
    <div id="header">
        <h1>
            <slot name="header"></slot>
        </h1>
    </div>
    <div id="icon">
        <slot name="icon"></slot>
    </div>
    <div id="content">
        <slot><slot>
    </div>
</div>

This is an example of how this could look when used:

<app-notification-box>
    <p>Some Content</p>
    <span slot="header">A meaningful Header</span>
    <img slot="icon" src="..." alt="notification icon"/>
</app-notification-box>

You can see how the “slot” attribute is used here. The values have to match the “name” attributes of the slots within the component. Consequently, this is part of a component’s public API and has to be documented accordingly.

Events

So far, we have only considered how data can be submitted to the components, but we have not yet looked at the opposite direction. In order to be truly interactive, developers must also be able to respond to certain events and accept data from the component.

This is the purpose of HTML events. We are only going to take a brief look at this aspect in this post and address it in more detail later.

Web components can generate both standard events and custom events.

Standard events are useful if the type of event also appears with standard HTML elements and does not need to be invented, e.g. a KeyboardEvent. Custom events are useful if additional data are to be sent with the event as payload. For example, if we build our own interactive table component where the users can select individual lines, it may be advisable to trigger an event upon selection that contains the data from the selected line as payload.

The mechanism for triggering an event is the same for all kinds of events. It can be seen in the following code block:

class InteractiveComponent extends HTMLElement {

    triggerStandardEvent() {
        const event = new Event("close")
        this.dispatchEvent(event)
    }

    triggerCustomEvent() {
        const event = new CustomEvent("some-reason", 
            { detail: { someData: "..." }}
        )
        this.dispatchEvent(event)
    }
}

To generate an event, you either generate an instance of “event” or one of the other event classes (one of which is “CustomEvent”). Every event constructor expects the first parameter to be the type of event. This type is later used to register listeners for these events as well.

The second parameter is optional and constitutes a JavaScript object that configures the event. For CustomEvent, for example, the “detail” field is provided to submit any desired payload data.

Conclusion

This post gives a brief introduction to the topic of “web components”, and with the methods shown, you can already build your own components. There are, of course, numerous other aspects that need to be considered in the development of web components. After all, this topic fills a number of reference books. In this series of blog posts, we want to focus on some of the pitfalls that can occur with individual issues, and examine how to avoid them. This series will also include a critical analysis of the web component API. The next blog posts are going to focus on the interaction with SPA frameworks in particular.