A Primer on Mobile Testing: The Risks of Bringing a Company into Mobile

A few weeks ago I was discussing with an ex-colleague of mine who is responsible for enterprise application development. He mentioned that all clients from his portfolio are looking for a mobile solution. Almost everybody in the company is talking about mobile development – how to build, what is the usability, how to test, how to release etc. The most important element of creating a mobile solution, he said was about assessing risk of the devices and data during the development. Modern enterprises are being run, or intended to run, on mobile devices. Enterprises pride themselves on the ability to implement their strict policies on their data, and security systems across distributed teams all across the globe. What sort of testing strategies would help the enterprise to mitigate such potential business risks and increase return on investment (ROI)?

Donut charts about interest in accessing enterprize systems via mobile.

Source: SAP-HBR White Paper: “How Mobility is changing the Enterprise”, by Sanjay J. Poonen, SAP.

What is Test Strategy

A test strategy is an outline that describes the testing approach which includes: the types of tests that are to be performed, how the product risks of the stakeholders are mitigated with these tests, and which entry and exit points we should be focusing on. In this document we shall discuss different types of testing with respect to enterprise application testing.

What are the Risks

One of the biggest considerations when bringing enterprise to the mobile landscape is understanding the possibility that the user may lost this device, and all its precious data! This scenario is not even considered in the desktop situation. UI standardization across mobile platforms means our application must not only have a nice design, an excellent user experience, and quick response time, but high-level data and network security. while remaining privacy compliant. Phew! Now that’s a big task.

Most enterprises are running on stable licensed softwares and network and access to data are quite secure and required multilevel access control. The biggest thing to nail down at the beginning of any enterprise app are the security protocols. You want the focus of your next app iterations to be on design, UX, features etc. but not security. Example, an estimated 114,000 Apple iPad 3G owner’s e-mail addresses and the identification number that those iPads were using when they communicate over AT&T’s network known as an ICC-ID were compromised few years ago. Similarly in Feb 2012, Citibank detected a potential threat through their iOS application. These kinds of exploits generate the risk of losing customers and diluting your brand’s reputation.

In the era of mobile communications and growing number of mobile platforms and devices, risk assessment and management are significantly complex tasks to achieve.


Source: www.lovehatedata.com, SAP

Enterprise Mobility Testing Requirements

1. Device Coverage

It’s almost impossible to actively test on EVERY SINGLE mobile device and use case. This where you really need to do your due diligence and figure out which market(s) your application is targeted for. Using the 80/20 rule, (20% of devices are used by 80% of target audience) you can quickly figure out which technological use cases you should be focusing on. You can mitigate a huge amount of risk simply by doing your market research. This will also save you lots of time (and money) in research and development!

2. Device Governance

Securing mobile devices and its data is a major challenge, due to their importance and portability. This goes threefold for enterprises in the financial, defense and health sectors. Mobile devices use (2G/3G/4G), Wifi, Bluetooth, IR etc. and they are exposed to potential malware infections and general security threats from all these incoming signals. WiFi hotspots or intrusion through BT are common scenarios in accessing the data of mobile devices. Even connecting mobile phones to the desktop using USB often put the corporate network at risk.

By figuring out what our market’s biggest threats are, we can begin to put locks in place. We need a testing strategy that detects these potential threats opened by enterprise mobile applications. Find these security points and  provide recommendation to the product management team with a constant feedback loop to make sure these security points are being taken care of.

During testing and development an efficient device management tool (like DeviceAnyWhere, PerfectoMobile, etc.) could be used to control, manage and audit the usages of the mobile phones and a set of processes and SLAs should be established to ensure no misuse of the mobile devices and it’s data during development, testing and post-production.

3. OS and Application version control

The number of devices and their models are determined based on the target market and applications are tested thoroughly on all those selected devices before releasing to the market. The problem starts when there are second or subsequent releases.. When the new devices and OS versions launch in the market, the existing market share ratios all become history. This may not be an  issue for the internal applications but is intolerable for business critical applications where audiences are external.

Assessment of the application ready for enterprise

To maximise the ROI on your product and security testing, here is a quick list of items we go through to make our QA teams as efficient and cost effective as possible::

Device Strategies during testing

  •     Optimum usages of the devices: Device sharing amongst the test engineers reduce overall  cost. Multiple projects and devices distributed in multiple testing teams will help to improve the utilization and subsequently help to achieve faster ROI. As described earlier the 80-20 rule should be used to achieve the maximum device coverage in target market.
  •     Devices should be secured: A fully managed device bank and their centralized access control and tracking can improve the security of the data, devices and usage. Make sure there’s a proper schedule in place for releasing all equipment.
  •     Scalable: Start with a small set of devices but have flexibility to procure more as the development and testing is expands. Various virtual device platforms and cloud based testing environments (example: DeviceAnyWhere, PerfectoMobile, TestDroid Cloud, etc.) can be used but with proper agreements and NDA signed.
  •     Device bank: The test organisation should maintain the device bank with the legacy devices to test. This will help the testing team to achieve different Interoperability testing.

Functional Testing Strategies

  •     Requirement Traceability: Make sure product testing goals are clear to each test engineer.  This helps the product management take right decision during releases and saves lots of time.
  •     Coverage Analysis: This could be achieved through various testing mechanisms, like Mind Map. Here the testing scope against the functional and nonfunctional requirements can be analysed in details and reviewed along with the project stakeholders to reduce risk.

Non-functional Testing Strategies

  •     Multiple locations, users and concurrency: Often, enterprise-ready mobile applications are designed to work in distributed complex environments. Performance testing tools like LoadUI, JMeter or Cloud based performance testing tools, offered by different service organisations, (example: ApicaSystem, SOASTA, etc.) can be used to ensure this.
  •     Memory and resource usage: One of the critical factor of testing in mobile applications is how the application is consuming power, memory and other device resources. Significant testing is required to ensure that the app is not hogging the device’s resources.
  •     Network testing: Network is not consistent, and they are of different forms (2G/3G/Wifi etc.). Testing should ensure good user experience during low or no network coverage or during network switching.

Test Automation Strategies

When development cycles are released quickly to end users and changes are unavoidable, then automation of the most popular  test cases become a critical success factor.

Mobile testing teams should consider different testing tools to automate test cases. Main factors to consider while choosing a tool are:

  •     Maintainability of the scripts.
  •     Easy to use by the test engineers who are from non-programming background.
  •     Knowledge Transfer timeline to ensure others can understand and run the scripts quickly.
  •     Device/Platform independent scripting: This will help to execute the same scripts on multiple platforms, devices, versions and returns quick ROI.
  •     Unattended execution: 24×7 execution of test scripts ensures the testing even at night.
  •     Easy and quick analysis of test reports.


To maximize the value from the investment of building an enterprise mobile application, one needs to ensure that the risk is reduced through a robust test strategy. This will satisfy the security needs as well as the adaptability of the application and will make everyone, especially your end-users, very happy!

Please feel free to leave us any questions or comments and Sougata will do his best to reply! Or feel free to connect with us on Twitter and we’ll answer your questions: @sourcebits

Piotr Gajos, Chief Innovation Officer

Piotr is Sourcebits Chief Innovation Officer. A 2006 Apple Design Award winner, Piotr draws much of his inspiration from film and music, and focuses on leading our Innovation Strategy Workshops, generating new ideas for Sourcebits, and consulting on projects.