Automated testing of smartphone and tablet apps is becoming more and more important. Regularly new devices and new versions of mobile operating systems are getting released on which apps have to be tested. Of course apps can also be tested on the simulator or emulator of their respective platform, but these tests are not always significant: On the one hand hardware can behave differently, especially when it comes with manufacturer-modified firmware, and on the other hand it is not possible to test integration with external hardware on a simulator most of the time.
Then again, when testing apps on the target hardware, developers face new challenges. First of all, automated deployment on a physical device can be tricky, especially on iOS. And secondly not all kinds of test input can be realized through the testing frameworks: Taps and touch gestures can be triggered from the testing scripts, but device movement like rotation or shaking cannot. Therefore it is also impossible to test landscape presentation on the target hardware.
So, how do you combine testing on the target hardware and testing device movement for landscape presentation? The solution: A mobile testing rig backed by servos!
At the Zühlke Camp, our annual job training event, a team of software engineers has built such a testing rig in a three-day track. The rig was supposed to be used for smartphones and tablets using the Android and iOS platforms and to provide a CI environment for automated execution of the tests on a defined set of devices.
The basis for the testing rig was created from wooden plates. The main plate was fitted with RC modeling servos, and car mounts for the phones were attached to the servo heads. We installed an Arduino Mega 2560 microcontroller board which we programmed using the Arduino Servo Library in order to control the servos through the Arduino’s I/O pins. To control the Arduino in turn, we attached an Ethernet Shield to it and implemented a REST interface through which we could send commands to the servos over LAN.
On the software end we set up a Jenkins server for continuous integration. To execute the app tests we first had to find a suitable testing framework. The final candidates were Zucchini, Frank and Calabash.
Zucchini is designed for testing on iOS only. The views and actions are defined using CoffeeScript and are executed using Apple’s UIAutomation library. The tests are defined in a syntax close to natural language. Execution of the tests is possible on the simulator as well as on the target hardware. However, because of poor documentation and especially because of the missing options for evaluating the test results, Zucchini was not given closer consideration.
Frank uses the Gherkin Syntax known from Cucumber. Like Zucchini it can be used for testing on physical devices and it can also be integrated into a CI environment. Custom test steps can be written in Ruby. The tools that come with Frank are well-engineered and make writing the tests easy. Unfortunately though, Frank can only be used for testing on iOS.
The framework we rolled with in the end was Calabash. Compared to Frank it’s missing the tools, but it is available for iOS and for Android. Calabash also uses the Gherkin Syntax, can be integrated into a CI environment and it allows for writing test steps in Ruby to be used in the test scripts. This way we were able to define a custom test step for triggering servo rotation on the test rig via HTTP.
While integrating the tests into the CI environment we faced a challenge trying to automatically deploy the app to Apple devices. While the Android platform supports automated deployment to devices very well, the iOS platform only allows for manual deployment using Instruments and Xcode. For building apps and deploying them to devices we finally used the Xcode Plugin and iOS Device Connector Plugin for Jenkins. The iOS Device Connector plugin uses the fruitstrap library which was created by reverse-engineering Apple’s proprietary deployment protocol. With these tools the automated deployment works without any issues. However, this way we’re running the risk that any update of the Apple software involved (iOS, Xcode, MacOS or Instruments) might break the automation and make reconfiguration necessary. This potential issue can possibly be resolved using the newly-featured Bots in the upcoming Xcode 5.
With this setup we were able to get a first working version of the test rig up and running which ran automated tests of four different projects on three iPhones and three Android phones, including device rotation and integration with Bluetooth peripherals (we used the app EATON easyRemote Display developed by Zühlke and the EATON easy800 control relay for the showcase). The automated testing rig is now being integrated into regular development activities at Zühlke in order to comprehensively test Android and iOS apps on the target hardware.