8 minutes to read Systematic automated testing of mobile apps is unfortunately still not a part of every project. I've seen many projects that only rely on manual exploratory testing. This is, in part, because the tools for automated testing of mobile apps haven't always been there. At their first Evolve conference in 2013, Xamarin announced that they had acquired LessPainful, a company mostly known for the creation of the popular Calabash mobile testing toolkit. The other big testing-related announcement at Evolve was that of the closed beta of Xamarin Test Cloud, a very impressive cloud-based platform for testing apps on a wide range of Android and iOS devices. Calabash Calabash tests are typically written using a combination of Ruby and Gherkin. Because of my .NET background, I wanted to try the new Xamarin.UITest framework to write tests with C# and NUnit. To make things more complicated, I decided to test an Ionic/Cordova hybrid app instead of a fully native or Xamarin C# app. Calabash works for iOS and Android. Operating the actual UI controls is accomplished through a Calabash component (Instrumentation Test Server) on the device or emulator that is controlled remotely by a test runner on the development machine or continuous integration server. For Android, the Instrumentation Test Server is deployed as a separate app. Due to more strict sandboxing on iOS, the Instrumentation Test Server needs to be integrated into the app in the form of the calabash.framework. While it's unfortunate that it's not possible to test the binary that's actually being deployed to the iOS App Store (calabash.framework uses private APIs that will not pass the automated App Store acceptance tests) the influence of the library on the app will typically not show in the tests. Xamarin.UITest To get started with Xamarin.UITest you'll have to set up a .NET test project. This can be done in Visual Studio for Android tests and in Xamarin Studio for iOS and Android (Xamarin is working on supporting testing iOS apps from Visual Studio). When the test starts it needs to know the path to the IPA or APK file to be used by all tests. Zühlke Engineering AG 1 2 3 4 5 [SetUp] public void SetUp() { _app = ConfigureApp.iOS.AppBundle(PathToIPA).StartApp(); } or Zühlke Engineering AG 1 2 3 4 5 [SetUp] public void SetUp() { _app = ConfigureApp.Android.ApkFile(PathToAPK).StartApp(); } The first unit test is typically started off like this: Zühlke Engineering AG 1 2 3 4 5 [Test] public void FirstUnitTest() { _app.Repl(); } This starts up a typically very useful C# REPL that can be used to identify the UI elements. When outputting all elements on iOS you will see something like this for a hybrid app: Zühlke Engineering AG 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 Full log file: /var/folders/wg/m0xyxrr10s5b4nz83dvgjssh0000gn/T/uitest/log-2014-12-13_22-37-58-452.txt iOS test running Xamarin.UITest version: 0.6.7 Initializing iOS app on http://127.0.0.1:37265/. Test server version: 0.12.0.pre1 Running activated version with api key from ConfigureApp App has been initialized to the 'app' variable. Exit REPL with ctrl-c or see help for more commands. >>> app.Query(c => c.All()) [ { "Id":null, "Description":"<UIWindow: 0x7a7d2060; frame = (0 0; 320 568); gestureRecognizers = <NSArray: 0x7a7d2950>; layer = <UIWindowLayer: 0x7a7d2620>>", "Rect":{ "Width":320.0, "Height":568.0, "X":0.0, "Y":0.0, "CenterX":160.0, "CenterY":284.0 }, "Label":null, "Text":null, "Class":"UIWindow", "Enabled":true }, { "Id":null, "Description":"<UIView: 0x7b8be890; frame = (0 0; 320 568); autoresize = W+H; layer = <CALayer: 0x7b8bdd70>>", "Rect":{ "Width":320.0, "Height":568.0, "X":0.0, "Y":0.0, "CenterX":160.0, "CenterY":284.0 }, "Label":null, "Text":null, "Class":"UIView", "Enabled":true }, { "Id":null, "Description":"<UIWebView: 0x7baa5660; frame = (0 0; 320 568); autoresize = W+H; layer = <CALayer: 0x7baa5770>>", "Rect":{ "Width":320.0, "Height":568.0, "X":0.0, "Y":0.0, "CenterX":160.0, "CenterY":284.0 }, "Label":null, "Text":null, "Class":"UIWebView", "Enabled":true }, { "Id":null, "Description":"<_UIWebViewScrollView: 0x7baaaed0; frame = (0 0; 320 568); clipsToBounds = YES; autoresize = H; gestureRecognizers = <NSArray: 0x7baab610>; layer = <CALayer: 0x7baab200>; contentOffset: {0, 0}; contentSize: {320, 568}>", "Rect":{ "Width":320.0, "Height":568.0, "X":0.0, "Y":0.0, "CenterX":160.0, "CenterY":284.0 }, "Label":null, "Text":null, "Class":"_UIWebViewScrollView", "Enabled":true }, { "Id":null, "Description":"<UIWebBrowserView: 0x7b2a3000; frame = (0 0; 320 568); gestureRecognizers = <NSArray: 0x7baa75e0>; layer = <UIWebLayer: 0x7b8c0e40>>", "Rect":{ "Width":320.0, "Height":568.0, "X":0.0, "Y":0.0, "CenterX":160.0, "CenterY":284.0 }, "Label":null, "Text":null, "Class":"UIWebBrowserView", "Enabled":true }, { "Id":null, "Description":"<UIImageView: 0x7a7d7700; frame = (0 565.5; 320 2.5); alpha = 0; opaque = NO; autoresize = TM; userInteractionEnabled = NO; layer = <CALayer: 0x7a7d7af0>> - (null)", "Rect":{ "Width":320.0, "Height":2.5, "X":0.0, "Y":565.5, "CenterX":160.0, "CenterY":566.75 }, "Label":null, "Text":null, "Class":"UIImageView", "Enabled":false }, { "Id":null, "Description":"<UIImageView: 0x7a7d82d0; frame = (317.5 0; 2.5 568); alpha = 0; opaque = NO; autoresize = LM; userInteractionEnabled = NO; layer = <CALayer: 0x7a7d8350>> - (null)", "Rect":{ "Width":2.5, "Height":568.0, "X":317.5, "Y":0.0, "CenterX":318.75, "CenterY":284.0 }, "Label":null, "Text":null, "Class":"UIImageView", "Enabled":false } ] Working with hybrid apps The only thing to see here is the full-screen UIWebView. Since all UI elements are rendered inside this web view there is no relevant information that can be gathered through this. To see the actual view elements we'll need to look inside the web view. This is possible through Safari for iOS and through Chrome for Android. You can use the browser developer tools to visually identify the elements inside the DOM. Once you've found the UI element you want to address you'll need to formulate a CSS selector to uniquely identify the element you want to call that works on both Chrome and Safari. A call to tap an element looks like this: Zühlke Engineering AG 1 _app.Tap(c => c.WebView().Css("#element")); Entering a text looks like this: Zühlke Engineering AG 1 _app.EnterText(c => c.WebView().Css("#username"), "testuser"); It's important to make sure that an element you want to interact with is already available if it's on a subsequent page. This can be achieved by waiting for the element: Zühlke Engineering AG 1 _app.WaitForElement(c => c.WebView().Css("#username"), "Timed out waiting for login view"); You can also execute arbitrary JavaScript on the web view: Zühlke Engineering AG 1 2 var backgroundColor = _app.Query(c => c.WebView().InvokeJS("document.getElementById('element-1').currentStyle.backgroundColor")).Single(); Assert.AreEqual("#0c9462", backgroundColor); Xamarin.UITest offers swiping methods to trigger a swipe gesture but the performed swipe may not be enough to actually navigate between two full-screen pages. A manual swipe implementation may look like this: Zühlke Engineering AG 1 2 3 4 5 6 7 8 9 10 private void SwipeLeft() { var bodyRect = _app.Query(c => c.WebView()).Single().Rect; float fromCenterX = bodyRect.CenterX * 0.9f; _app.DragCoordinates( bodyRect.CenterX + fromCenterX, bodyRect.CenterY, bodyRect.CenterX - fromCenterX, bodyRect.CenterY); } Enter Visual Studio App Center Test The fun starts when you upload your app and corresponding test script to Visual Studio App Center Test, the successor of Xamarin Test Cloud and give it a spin on a multitude of devices. To prepare for this it makes sense to insert a number of named screenshots into each test case. Zühlke Engineering AG 1 _app.Screenshot("After swipe"); In the App Center portal you can define a set of devices to deploy to and run the test on all of them. I did this for a simple test of the Ahoi! train timetable app by my coworker Christian Kohler. Ahoi! is based on Ionic and has a wonderful user experience that allows you to search European train timetables with only two taps. The results are very impressive and allow deep insights into the behavior of an app on a large number of mobile devices. Summary Xamarin.UITest is a very powerful tool for UI testing of any kind of mobile app. The C#/NUnit approach allows testing almost any type of user interaction. If you combine these possibilities with Visual Studio App Center Test you possess the tools to provide your users with the maximum level of quality across a multitude of Android and iOS devices, OS versions and form factors. Kerry Lothrop Principal Consultant Kerry W. Lothrop is a Principal Consultant at Zühlke with over 20 years of experience in software development. He joined Zühlke in March 2008. As a consultant and tech lead, he accompanies projects from the initial vision to market success. His project projects typically connect things with each other, from cloud backends to mobile phones to IoT devices. Kerry holds a degree in Aerospace Engineering and was recognized as a Microsoft MVP (Most Valuable Professional) by Microsoft and as a Xamarin MVP by Xamarin. He is a frequent public speaker and runs a tech blog and podcast. Contact kerry.lothrop@zuehlke.com +49 6196 777 54 770 Your message to us You must have JavaScript enabled to use this form. First Name Surname Email Phone Message Send message Leave this field blank Your message to us Thank you for your message.