Real Cross-Platform Workflow UI Automation

Testing workflows that span multiple applications and platforms is a challenging task. Imagine a test case that calls an API first, then an operator has to enter information at his web app so that the end user finally sees the update in his native app on a smartphone. Real End-to-End automation is nearly impossible today.

Current approaches try to automate cross-platform with the help of web apps running on (mobile) browsers. Native apps on Android or iOS can only be included in the workflow by using time-intensive techniques like picture-in-picture search because they lack code selectors which operate on the Document-Object-Model (DOM).

With the rise of machine learning/artificial intelligence, we can train models to recognize elements on User Interfaces based solely on visual attributes. With that, it becomes possible to automate any platform when you combine it with automation on the Operating System level.

In this talk, I will show a live demo that will walk the audience step-by-step through a typical 2-Factor Authentication using a physical smartphone. We will see what automation is possible today and how it enables testing currently untestable workflows.