The King is dead, long live the King! But which king? After the Angular team announced in April that it would stop development of its Protractor e2e testing framework at the end of 2022, many developers have been asking themselves what will happen next. Which test automation tool should be used? The market has a wide range of alternatives, some of which will be discussed in this article.
The test candidates
Cypress
Cypress is probably the best-known candidate in the test field. According to an Angular Community poll conducted in January 2021, it is the most widely used tool for automated UI testing, with over 60% of the votes. The first stable release was published back in 2017, and the application is currently on version 10. Cypress describes itself as fast, simple and reliable, relying on its own ecosystem of tools that, unlike Protractor, is independent of Selenium and WebDriver. Chrome, Firefox and Edge are currently supported, though not Safari (as yet). The possibility of running tests against various browser versions is offered by either connecting BrowserStack or by swapping the binaries for local execution.
| Pro | Contra |
| + fast + stable + large community + thorough, easy-to-understand documentation | – still no support for Safari |
Playwright
Developed by Microsoft since 2020, Playwright is the newest candidate in the field of automation tools. According to its own description, the library is up-to-date, fast, reliable and powerful, thus helping to eliminate common UI testing problems such as flakiness and slow execution speeds. With this aim in mind, it is not surprising that unlike the other testing tools, Playwright is not based on Selenium WebDriver and instead has its own implementation for browser control. The current browser engines from Chromium, Firefox and WebKit are supported on Windows, Linux and MacOS respectively. Older versions can be integrated using older versions of Playwright, and more recently, support has been added for generic Chromium builds (Edge and Chrome). Currently, support for mobile devices is only possible via the emulation of certain device configurations, such as resolution and user agent, and testing on real devices is not possible.
The feature set includes standard operations as well as functions that are missing in other frameworks or which can only be achieved using workarounds. Thus, in addition to clicks, scrolling, selections and so on, drag-and-drop, Shadow DOM interactions and synchronisation with animations are also available.
Im Hinblick auf Performance und Ausführungsgeschwindigkeit verhält sich Playwright wesentlich schneller als In terms of performance and execution speed, Playwright works much faster than Protractor and other Selenium-based testing frameworks. In our tests, we did not detect any problems in terms of flakiness or unexpected crashes, even with several test instances running in parallel. In conclusion, with Playwright, Microsoft provides a modern framework geared towards the requirements of modern UI testing. In its short time on the market, it has already won many advocates and will probably continue to grow in importance in the future.
| Pro | Contra |
| + fast + stable + large feature set | – currently only offers limited support for BrowserStack – no support for real mobile devices |
Webdriver.io
It is now over seven years since Webdriver.io was first released, making it one of the longest-standing automation tools in the test field. The creators describe the design of their test framework as expandable, compatible and feature-rich, claims that stand up in the face of closer inspection:
Webdriver.io is a Selenium-based test framework that implements the W3C WebDriver API, which is based on it. This ensures compatibility with all modern browsers (and IE) that provide their own driver implementation. It also enables the testing of web apps on mobile devices, as well as native apps.
The further strengths of the framework become apparent in day-to-day use: A wealth of help methods and syntactic sugar makes formulating test steps and expectations much easier and more readable than was usually the case with Protractor. Some points of note include access to ShadowRoot components via shadow$, versatile configuration options (including automatic test repetitions in the event of errors) and the ability to wait for DOM elements to appear or disappear. However, if a function is not available, it can be implemented as a custom command and then used in the same way as the functions provided.
Despite all the praise, one fly in the ointment remains: Compared to other frameworks, test execution is quite slow due to the Selenium-based implementation, but execution can also be done in parallel.
| Pro | Contra |
| + versatile compatibility + excellent documentation + numerous support methods and syntactic sugar | – test execution slower than in other frameworks |
TestCafé
TestCafé is one of the lesser-known candidates in the test field. In the aforementioned Angular community poll, it scored a usage percentage of less than 1%. The first stable release was published in 2018, and the application is currently in version 1.19. Two good reasons to take a closer look included the promise of an easy setup and its independence from WebDriver. Instead, TestCafé uses a rewritten URL proxy called Hammerhead that emulates commands via the DOM API and injects JavaScript into the browser.
All major browsers and the mobile versions of Chrome and Safari are currently supported. Of note is the ability to run tests on real mobile devices across the same network. It is also possible to run tests with BrowserStack. It should also be mentioned at this point that a dedicated development environment (TestCafé Studio) also exists, where you can create and record test cases with just a little programming knowledge.
| Pro | Contra |
| + fast + versatile compatibility (including for mobile browsers and end devices) + large feature set | – low circulation – test recorder tool is only available at a cost |
Comparison
When choosing a testing framework, it is firstly important to know your own project requirements. To this end, the following questions may prove useful:
- Which browsers are to be supported?
- Are different browser versions to be tested?
- Is support required for mobile devices?
- Can the app’s core functions be tested using the framework?
Die folgende Tabelle kann bei der Orientierung der Wahl eines TThe following table can help in selecting a testing framework.
| Criterion | Cypress | Playwright | Webdriver.io | TestCafé |
| Browser support (and versions) | o | + | + + | + + |
| Mobile support | + + | + | + + | + + |
| Range of functions | + | + + | + + | + |
| Test speed | + | + | o | + |
| Migration from Protractor | o | o | + + | – |
| Future-proof | + + | + | + | o |
Summary
Given the very wide range of testing frameworks available, it is often tough to make the right decision. For projects with low test requirements, for example, testing against a few, exclusively current browsers, where fast test execution is an advantage, we recommend using Playwright.
However, when the project requires migration from Protractor and it is important to test against a wide range of browsers in different versions and on mobile devices, we recommend using Webdriver.io.
Sources
- https://www.rainerhahnekamp.com/en/angular-e2e-testing-protractor-is-dead-long-live-cypress/
- https://github.com/angular/protractor/issues/5502
- https://docs.cypress.io/
- https://webdriver.io/
- https://playwright.dev/
- https://testcafe.io/
Dieser Beitrag wurde verfasst von:
Stefan Heinze
Since 2011, Stefan Heinze has worked in Dresden as a software developer at ZEISS Digital Innovation. He is currently working on Angular applications and Azure back-end development in the medical field. There is an increased focus on the automation of surface tests to ensure software quality.
This post was written by:
