Mobile applications are growing in use and sophistication along with the speed of the networks and the power of the devices they run on. The multiplicity of devices and operating systems alone make for a very challenging testing matrix.
Testing of mobile applications can be done with minimal automation, but a high level of test automation will help assure the best quality mobile applications. For any automation effort the primary key to achieving success will be the approach. As a rule of thumb test automation is successful when the “5% rules of test automation” can be achieved, which means that no more than 5% of test cases should be executed manually, with no more than 5% of testing resources applied to reach that level of automation. This is aggressive, but achievable. Having this as a goal puts the focus on effective testing, not on test execution or automation scripting, and establishing this as a goal will help drive efficiency throughout the entire test team.
The most significant key to successful mobile testing is understanding the mobile ecosystem in order to design tests appropriately.
Android and iOS are the two dominant code bases for mobile devices that most applications will be designed to run on: Even on the same platform, it is necessary to test different hardware configurations, for example iPhone vs. iPad, Nexus S vs. Galaxy Tab, etc. This means there is a substantial opportunity for bugs to appear between different versions, and every time a code change is made there is risk of breaking something that used to work. In addition to this every device has its own set of operational nuances. These include what are best described as interruption events; things like incoming call and message notification, change in network speed, and power warning notices. Without understanding the ecosystem and placing a significant emphasis on up-front test design and structure, the risk of having tests that work for a while and then inexplicably break is high. What seems as basic as automatically installing an app and launching it without manual intervention can be surprisingly complex. For example when testing an application download function, the outcome of the test is dependent on the application, the device and the OS version.
Mobile UI Testing
Mobile devices rely heavily on object oriented user interfaces with screens and controls. Routines are increasingly executed by taps, swipes and pinches and less on keypad inputs. Test cases that verify the flow and dialogs of the UI should be created and executed early in the test cycle to catch UI issues that could impact automation in the later stages. This will eliminate hours of investigative time to find the tests that need to be corrected.
There are two methods to test the UI actions–image-based and object-based testing. Object access to controls is a much more powerful and a maintainable option. It will expose virtually all properties of a screen or control to the test tool and also be visible to a developer. These properties can be used to identify a control or to verify the control values. For example with this approach the caption on a button will be visible as the “text” property of that button.
All things being equal, object oriented testing is the preferred method as it isn’t impacted by UI dynamics such as changes in background, image size or orientation. Automating the testing requires access to the OEM provisioning files (Apple’s is ipa and Android is APK). The downside is that not all apps can access the provisioning files, for example Android security will not allow access to built-in applications.
Image-based automation will provide a true “end user approach” that verifies the tests sees the same image as the user. The test automation engine acquires the images from the on-screen display to perform the tasks required for the test. It doesn’t matter what the underlying technology is, if it is visible on the screen image-based automation can interact with it.
Since it appears that image-based testing would work more often than object-based testing it begs the question “why not just use image-based testing?”. The answer is that it is generally more sensitive to even minor changes in the UI or the application under test, and it is harder to let the same test run across different devices with different screen resolutions. In order to verify the app continues to work as expected when any of the parameters change, test cases with images for those specific parameters are needed. Here again, the modular test structure in TestArchitect makes it possible to minimize maintenance by creating a module just for the images. By just updating the test in the image module, all tests containing the images will be updated.
Testing multiple devices
Creating test suites for all the different devices can be a daunting task. The ideal solution is to create one set of tests that can be used for multiple devices and applications. TestArchitect Mobile Plus is one tool that makes this possible. Using action keywords and a modular test design makes it possible to separate tests from the underlying code. The automation tool assembles the test scripts from the actions, freeing test developers from tedious coding. The abstraction engine in TestArchitect translates the test cases into the appropriate code for the target device(s). There is some work required by automation engineers to map the appropriate interface, but this is a fairly minor task. The result is that 100 or 1000 test cases can be multiplied by the number of devices and OS, with a fractional use of testing resources.
A good mobile test automation strategy will include testing multiple devices simultaneously. This can involve running one test on many devices in parallel, and depending on the application it may require testing devices while interacting with each other. These types of tests can be run with agents or simulators, or a combination of both.
Mobile emulators can be essential for developing and testing a quality application. Emulators are good from the standpoint of running standard UI interactions and being able to do exploratory and benchmark testing economically. The technology enables developers to verify certain functionality that is not specific to any device, carrier or operating system, allows data input, button use, etc. – all with the convenience of a full keyboard and mouse. However, emulators and simulators don’t operate in the real world and it is always recommended to test real devices in the native operating environment. Once satisfied the test parameters are correct, automation agents can execute tests on the actual devices (and run tests in parallel with emulators) for specific versions of devices and or OS so that testing effort can be maximized.
Cloud based services are also an option for multiple device testing. The benefit is that these services have a wide range of devices and you pay only for use. The risk is uncertainty of the validity of the outcome. Many of the phones used in cloud based services have been jail-broken which can result in unintended consequences. This can be the case when an application relies on a specific carrier’s network security–what worked fine in the lab may not work when released to customers. Again, a good way to validate this is to test one or more actual devices.
There are device connectivity options as well. Devices can be connected to the host machine via USB and Wi-Fi. In most cases a USB connection is necessary to set up the automation agent, but once operational the tests can be driven over Wi-Fi. This has the benefit of testing the sensitivity of the application and device to changes in speed and or network signal strength–which users will encounter in the real world.
Summary
The demands for mobile test automation will continue to grow along with the increasing adoption of mobile devices. Due to the multiple OS standards and the multitude of device providers testers face a highly complex testing environment. By adopting the right techniques and methods and employing the appropriate tools, it is possible to tackle complex automation and be able to effectively scale as the need requires. It all starts with planning. Part of that planning should include learning and implementing new methods and tools that will improve efficiency. Those who are committed to long-term success in their field recognize this and embrace it.
Request More Information
