Automated mobile UI testing with Xamarin.UITest and Visual Studio App Center
Systematic automated testing of mobile apps is unfortunately still not a part of every project. I've seen many projects that only rely on manual exploratory testing. This is, in part, because the tools for automated testing of mobile apps haven't always been there.
At their first Evolve conference in 2013, Xamarin announced that they had acquired LessPainful, a company mostly known for the creation of the popular Calabash mobile testing toolkit. The other big testing-related announcement at Evolve was that of the closed beta of Xamarin Test Cloud, a very impressive cloud-based platform for testing apps on a wide range of Android and iOS devices.
Insight in brief
- Xamarin.UITest is a very powerful tool for UI testing of any kind of mobile app.
- The C#/NUnit approach allows testing almost any type of user interaction.
- If you combine these possibilities with Visual Studio App Center Test you possess the tools to provide your users with the maximum level of quality across a multitude of Android and iOS devices, OS versions and form factors.
Calabash tests are typically written using a combination of Ruby and Gherkin. Because of my .NET background, I wanted to try the new Xamarin.UITest framework to write tests with C# and NUnit. To make things more complicated, I decided to test an Ionic/Cordova hybrid app instead of a fully native or Xamarin C# app.
Calabash works for iOS and Android. Operating the actual UI controls is accomplished through a Calabash component (Instrumentation Test Server) on the device or emulator that is controlled remotely by a test runner on the development machine or continuous integration server. For Android, the Instrumentation Test Server is deployed as a separate app. Due to more strict sandboxing on iOS, the Instrumentation Test Server needs to be integrated into the app in the form of the calabash.framework. While it's unfortunate that it's not possible to test the binary that's actually being deployed to the iOS App Store (calabash.framework uses private APIs that will not pass the automated App Store acceptance tests) the influence of the library on the app will typically not show in the tests.
To get started with Xamarin.UITest you'll have to set up a .NET test project. This can be done in Visual Studio for Android tests and in Xamarin Studio for iOS and Android (Xamarin is working on supporting testing iOS apps from Visual Studio). When the test starts it needs to know the path to the IPA or APK file to be used by all tests.
The first unit test is typically started off like this:
This starts up a typically very useful C# REPL that can be used to identify the UI elements. When outputting all elements on iOS you will see something like this for a hybrid app:
Working with hybrid apps
The only thing to see here is the full-screen UIWebView. Since all UI elements are rendered inside this web view there is no relevant information that can be gathered through this. To see the actual view elements we'll need to look inside the web view. This is possible through Safari for iOS and through Chrome for Android. You can use the browser developer tools to visually identify the elements inside the DOM.
Once you've found the UI element you want to address you'll need to formulate a CSS selector to uniquely identify the element you want to call that works on both Chrome and Safari. A call to tap an element looks like this:
Entering a text looks like this:
It's important to make sure that an element you want to interact with is already available if it's on a subsequent page. This can be achieved by waiting for the element:
Xamarin.UITest offers swiping methods to trigger a swipe gesture but the performed swipe may not be enough to actually navigate between two full-screen pages. A manual swipe implementation may look like this:
Enter Visual Studio App Center Test
The fun starts when you upload your app and corresponding test script to Visual Studio App Center Test, the successor of Xamarin Test Cloud and give it a spin on a multitude of devices. To prepare for this it makes sense to insert a number of named screenshots into each test case.
In the App Center portal you can define a set of devices to deploy to and run the test on all of them. I did this for a simple test of the Ahoi! train timetable app by my coworker Christian Kohler. Ahoi! is based on Ionic and has a wonderful user experience that allows you to search European train timetables with only two taps.
The results are very impressive and allow deep insights into the behavior of an app on a large number of mobile devices.
Xamarin.UITest is a very powerful tool for UI testing of any kind of mobile app. The C#/NUnit approach allows testing almost any type of user interaction. If you combine these possibilities with Visual Studio App Center Test you possess the tools to provide your users with the maximum level of quality across a multitude of Android and iOS devices, OS versions and form factors.