3 alternatives to Xamarin.UITest

• Why you should write automated UI tests
• Why we need an alternative to Xamarin.UITest
• Finding alternatives to Xamarin.UITest
• Comparing the alternatives to Xamarin.UITest
• Practical evaluation
  • Appium - the allrounder
  • Maestro - the simple one
  • Waldo - the no-code solution
• Conclusion

Why you should write automated UI tests

When it comes to test automation, we often speak about a pyramid portraying the different levels of automated tests with unit tests at the base and UI tests at the top.

Unit tests as a base are now the standard in professional software development. They are used to test modules against their specification. However, they cannot be used to determine whether the interaction between all functional units works properly.

A step up from that are integration tests, forming the middle layer of the test pyramid. Imagine unit testing a function that stores data in a database or exchanges data with a web service. Integration tests go beyond the boundary of a single unit and integrate functionality of other units in the test, to ensure that the affected modules interoperate correctly.

Unit and integration tests leave out an important factor when the application has some kind of frontend. The user interface, a crucial part of your app or website that the user directly interacts with. Testing at the user-facing interface level allows you to test common flows and use cases to ensure that application or business critical functions do what they are supposed to on different resolutions, orientations or designs.

Moving higher up in the pyramid means higher costs, longer execution times, growing complexity and less reliability, while also achieving more coverage and reducing otherwise growing manual testing effort.

At all levels of the pyramid, we strive for the highest possible automation for the following reasons:

• repeatability and determinism of results
• fewer regressions
• less manual testing and time spent fixing bugs
• reduced costs

Automating most of the actual testing, especially the most common use cases that don't require complex setup, saves you a tremendous amount of time. Time that can be spent for manually testing edge cases or fixing bugs. Building a large suite of automated tests helps detect regressions quickly and save you a lot of headaches.

Why we need an alternative to Xamarin.UITest

The more .NET MAUI is approaching maturity, the more the need for a compatible and functional UI testing framework arises. For Xamarin and Xamarin.Forms apps Xamarin.UITest was the go-to option for UI tests. With it's tight integration into App Center it enables developers to run their tests on App Center Test (formerly Xamarin Test Cloud) on a wide range of physical devices. Developers can also write tests for native iOS and Android apps in C#, if they so desire.

It even has support for backdoors built in, with which you can call methods in your running app, enabling you to call upon logic that might not be accessible through your UI. For example, your backdoor method could simulate an incoming PushNotification and therefore enable you to test how your UI behaves when that happens. Although that means you are stretching the boundaries of black-box testing.

There even was a time when the Xamarin Test Recorder existed, that enabled you to record your app interactions and generating test code for it. Unfortunately that product was discontinued on March 2018.

The bad news: now that Xamarin moved to .NET, compatibility of Xamarin.UITest is on a decline. At the time of writing the last update of the NuGet package was for .NET 6, although .NET 7 is already released. I tried to write and run tests for a .NET MAUI app with Xamarin.UITest and the results where mixed.

On Android some simple queries were not working properly, while others worked perfectly. For some reason the Repl command also did not work. However, I was still able to invoke backdoor methods. So aside from some limitations you can use Xamarin.UITest to test your .NET MAUI app, at least on Android.

With iOS though, it's a whole other story. To run Xamarin.UITest on iOS, you need to install the Xamarin.TestCloud.Agent NuGet package, so that the Xamarin Test Cloud Agent backend will be run when your app launches. However, due to the fact that Xamarin.TestCloud.Agent was not updated for .NET I was not able to add it to my .NET MAUI app to run UI tests on iOS. An issue for that has been open for almost a year now on the microsoft/appcenter repository.

Because progress on App Center and especially Xamarin.UITest has been sluggish in the recent time, many people are asking for Xamarin.UITest to be made open source in the hopes that the OSS community is able to fix some of the bugs and keep it up to date. There are rumors that .NET MAUI will not use Xamarin.UITest going forward and instead is looking at an Appium based approach. Some efforts have also been made to create a new UI testing framework for .NET MAUI, namely Maui.UITesting by @Redth and Xappium, but they also became stale. The .NET MAUI team is also refraining from communicating clearly where they want to go with UI testing.

Considering all of that, it looks like UI testing is currently not high on the priority list of the .NET MAUI team, at least it cannot be found on the ongoing GitHub project board.

Regardless, we still want to benefit from automated UI tests for our .NET MAUI applications, therefore let's have a look at what alternatives exist, compare them to Xamarin.UITest and see how far we can get with them.

Finding alternatives to Xamarin.UITest

Before we look at specific options, let's think about what we are looking for when it comes to UI testing cross-platform mobile apps and broaden our view a bit. When it comes to cross-platform apps you probably don't want to learn multiple frameworks like UIAutomator or XCUITest. At least I would prefer to write the tests only once for each scenario, wherever possible, instead of once per scenario per platform. As long as most of the UI looks and behaves very similiar.

Which framework/platform fits best for you, your team and your project depends on many factors. Some characterics are stronger filters, like the programming language or supported platforms, than others. As a developer of the app it feels natural to use the same programming language for the UI tests as for the app. But that's not necessarily the only option, as some people might be exclusively responsible for writing UI tests and prefer other programming languages. So a script- or markup-based approach can also be a valid choice. Generally, text-based UI tests come with valuable advantages: tests can be easily read, maintained and versioned.

Although, not everyone who is tasked with coming up with automated test scenarios is necessarily a developer. Test recorders help, but a no-code approach might be even better in some cases.

When it comes to running the tests, a lot of times it's beneficial if they can be run locally on a simulator or a physical device. But to regularily run your automated UI tests you would want to integrate them directly into your CI/CD processes for QA purposes. In addition to that, you should also consider running your UI tests on a variety of devices instead of a few selected physical devices you have at hand. That is especially important in the mobile world where there are a plethora of different devices with varying screen dimensions and resolutions. That is where Test Clouds come in, which enable you to run your UI tests on a wide range of devices without having them physically available.

Backdoor methods are an interesting feature for developers as they allow them to test functionality within their app that cannot be reached through UI only. For example, a backdoor method could simulate an incoming PushNotification and therefore enable testing of how the UI behaves when that happens. But one should keep in mind that that's stretching the boundaries of black-box testing.

As you can see, there are many characteristics that make a good UI testing framework/platform. There are even different approaches to how UI tests are created, which each have their own right to exist. To represent each of the different approaches I specifically chose the following frameworks/platforms to explore a bit more:

• Appium - A flexible client-server based framework that uses the WebDriver protocol
• Maestro - A more recent simple and fastly evolving framework, where tests are defined with YAML
• Waldo - A no-code UI testing platform where tests are primarily recorded by navigating in your simulated app

There are many frameworks that build on top of Appium or the Selenium WebDrivers like Xappium, Legerity and many others. Even fullfledged platforms like Lambdatest. As they all rely on Selenium/Appium I focussed on Appium for this comparison.

Comparing the alternatives to Xamarin.UITest

After some research and testing I created the following table to give a quick overview over how the frameworks compare to eachother:

One of Appium's greatest advantages is that it is an open source framework with a large userbase and many contributors from lots of different technical branches. No wonder that the Client-SDK exists for multiple programming languages. In addition to Android and iOS, Appium can also UI test Windows and Web making it stand out from the alternatives. Due to the large community and it being open source there are many third party products bringing you test cloud support, test recorders or backdoor methods, among other things.

With Maestro, tests are written in YAML, which makes it very easy to learn, but can also limit flexibility. Lately, Maestro Studio, a visual UI explorer was added, which is now being extended to become a test recorder. Mobile.dev inc., the company behind Maestro, offers a test cloud, that can run UI tests on simulators only, for now. Since Maestro is a more recent framework, it leaves some things to be desired. However, it is constantly improving and features are added regularly.

Waldo works differently in that tests are not written, but recorded only while you navigate through your app, that is running on a simulator/emulator, via your browser. Waldo itself is in its core a test cloud and a test recorder. Unfortunately, tests cannot be run locally, due to it all running in the cloud. But that can also be an advantage, as a working internet connection is all you need.

Practical evaluation

Up to now we looked at the characteristics more from a theoretical perspective. Yes, UI tests with Appium and Maestro, can be run locally, but how hard is it to setup your local environment? A test recorder is nice, but can you go in and edit an existing test scenario or do you need to rerecord everything? Are the frameworks any good for UI testing .NET MAUI apps?

I tested the usability of each of these frameworks with a small .NET MAUI app on Android and iOS. Here is my experience with each of them:

Appium - the allrounder

To run UI tests with Appium you need to install:

• Appium server: a HTTP server that accepts instructions from clients
• A platform-specific driver e.g. XCUITest Driver or UiAutomator2 Driver
• An Appium client or client library

Dividing the infrastructure into client and server, where the server acts as a middle-man between client and device, makes it very flexible. The client will always use HTTP requests to communicate with the server, regardless of which programming language the client is using. The server interpretes the commands from the client and is able to transform it into a command for the specific Driver, abstracting away the driver specifics.

So far so good, but having multiple dependencies makes it harder to understand how things work and leaves room for errors. Individual drivers can also require additional setup steps. In January 2022 Appium went under a major revision and moved to Appium 2.0, making the most recent existing drivers incompatible with 1.x versions. Appium Desktop, which provided a GUI for interacting with the Appium Server, also became deprecated with the new revision. As I was using Appium Desktop years ago I was wondering why it no longer worked. So if you tried Appium before, make sure you update everything before you get started.

When connecting to the Appium server you will need to pass capabilities that describe the target platform, device, app and driver you want to use. When using the C# SDK you create a platform-specific instance of the class AppiumDriver. You can then call methods on this instance to find UI elements and interact with them. For an iOS app that might look like this:

var options = new AppiumOptions();
options.AddAdditionalCapability("appium:platformName", "iOS");
options.AddAdditionalCapability("appium:app", "de.cayas.mauiuitest");
options.AddAdditionalCapability("appium:udid", "4CC6A4D1-116F-4BE2-BA1B-B34805840401");
options.AddAdditionalCapability("appium:deviceName", "iPhone 14 Pro Max");
options.AddAdditionalCapability("appium:automationName", "XCUITest");
        
_driver = new IOSDriver<IOSElement>(new Uri("http://127.0.0.1:4723"), options);
_driver.LaunchApp();
_driver.FindElementByName("Click me").Click();

FindElementByName("Click me") to find a button with the text "Click me" worked on iOS. However, that did not work on Android with the UiAutomator2 driver. I had to resort to making sure I add an AutomationId so that I could use FindByAccessibilityId method instead. So there might be cases where you will have to write some platform-specific code in your tests.

Maestro - the simple one

Maestro claims to be 'built on learnings from its predecessors (Appium, Espresso, UIAutomator, XCTest)' making it easier to define and test your Android & iOS apps. And compared to Appium I must say it delivers on the ease of use part. All you need to do is install the Maestro CLI and for iOS make sure you install Facebooks idb-companion. One disappointing caveat though, is that Maestro can only run iOS apps in the simulator. But their support for real iOS devices is in the works and planned for Q2 of 2023.

UI tests in Maestro consist of one or more 'Flows' that are defined with YAML. Here you can see how it runs a UI test for my .NET MAUI application:

You can also have Maestro run your test continuously, so that whenever you make changes to the yaml file, the test is automatically rerun. This enables you to iteratively write your test flow. With Maestro Cloud you can also run your Flows on a test cloud, but only on simulators and emulators for now.

Recently Maestro added Maestro Studio, a graphical UI element inspector. And even more recently they added a recording feature to it, so that you can easily record your flows, although it is still very limited in it's capbilities and only supports simple click actions for now. Regardless, it looks very promising:

As great as it all looks and sounds, there are still some trade-offs. Using YAML you do not have the flexibility of a real programming language and are limited to the commands that the Maestro team adds. Currently the team is iterating fast and pumping out new features regularly, while still being open to enhancements, which is great.

Waldo - the no-code solution

As mentioned before, Waldo follows a no-code approach. To make that work, your .apk or .ipa file needs to be uploaded to Waldo. The app will then be launched on a simulator or emulator of your choice and you will be connected to a live session of your app running on it. To record your first test, all you do is interact with your app directly in your browser. After you have performed all the steps for your test scenario, Waldo will rerun your test three times to automatically make assertions based on what is changing in the UI and check whether the test is deterministic and repeatable.

When test validation is done, you can start modifying your test by rerecording individual steps or adding additional assertions. By default Waldo asserts that 85% of the UI must match with the UI that was recorded during the initial test validation. You can assert on the text of a specific element or on its position.

This way of defining tests is very close to how your users will interact with your app. And of course, there is no coding knowledge required to create the tests. Unfortunately, that can also make the process of creating or modifying tests tedious and slow, since you cannot copy/reuse bits from existing tests and you will have to wait for your Waldo session to get ready everytime you want to (re)record a test. But keep in mind that Waldo only runs your tests on simulators too. The fact that Waldo is an online platform, means you don't need any complicated local setup and you can create your tests from anywhere, as long as you got an internet connection.

Conclusion

We have learned that the current state of Xamarin.UITest is not fully compatible with .NET MAUI and that there is uncertainty about where UI testing in .NET MAUI is heading. There are multiple different approaches to solve UI testing and each of them has its pro's and con's. The same applies to the many UI testing frameworks out there. We looked at 3 viable alternatives and how they compare to Xamarin.UITest.

Appium is probably the feature richest and most flexible of them all. Although, setting it up can be a challenge.
Meastro's simpleness is charming and due to it rapidly evolving it's worth to give it a shot and keep on your radar.
Waldo is fantastic for protecting against visual regressions and getting testing teams fast on track without any programming knowledge.

I am curious to what UI testing frameworks will emerge for .NET MAUI and especially if the MAUI team will officially introduce a properly compatible framework that can target all of MAUI's target platforms. Generally, most .NET MAUI apps can be UI tested using frameworks native to the target platform, as long as you set the AutomationId on you UI elements.