Testing ROI – A Numbers Game

This is a guest blog by Bill Weinberg, Senior Executive, Mobile Practice Partner – The Olliance Group

Everyone agrees that testing software (and device testing too) is necessary and valuable. Everyone looks to testing to endow software and handsets with quality and reliability. But test engineering is often pure misery – in the quest for bugs and faults and other failures, developers are damned if they find bugs and more damned if they don’t.

Testing is a numbers game. There are probably graduate theses and tortured theorems that testify in detail as to why you can never totally test a piece of software. Similarly, there are ample handy heuristics for deciding how much testing for both software and devices is “enough”.

The mobile domain is not as unique in the testing universe – it shares much with sister disciplines on the desktop and other embedded verticals. But testing Android apps and hardware does present its own set of challenges, including:

• New Android platform releases every 3-4 months
• A vast array of published APIs
• Multiple implementation paths (high level and low, Dalvik-based and native, framework and extension)
• Platform fragmentation with behavior and performance difference across Android devices

It is the last two that are wreaking havoc with software quality and inducing the highest device return rates in the industry, especially for performance and user-experience of games and media-intensive apps on so-called “mass-market smartphones”.

In most software development scenarios, testers worry about coverage, focusing on the proportion of source or binary exercised by test suites and tools. Certification regimes aim high, target 80% or even 90% coverage, or wishfully even more. Real-world testers are usually happy to cover 50% of deployed code, and many code bases never get past 25% coverage.

Android apps are no exception.

The Android Device Jungle

But Android apps and devices also face different coverage challenges – wholesale device and application coverage. Keeping track of new Android device introduction, of handsets and tablets and other devices in volume shipment, and of legacy devices is a wicked paper chase. My friends at Apkudo invest heavily in tracking devices in the wild, and they reckon that as many as 300 different Android-based devices grace the pockets and purses and premises of the consuming public.

As with code under test, device test coverage is an 80:20 affair (or 70:30 or 60:40) – that is, 80% of the benefit comes from testing 20% of deployed kit, with the implication that 80% of bugs will surface through testing just 20% of the actual deployed device portfolio.

Do the math. If you are an application developer, in theory you will realize maximum return on your testing investment by exercising your app on 20% of deployed kit – 60 devices (gulp!). Given the dominance of a few models of current handsets from leading OEMs – HTC, Motorola and Samsung in particular – that number is realistically a more attainable dozen.

But the Android application developer community is as varied as they apps they produce. There are large ISVs, for whom incremental platform testing can be expensive, but within budget. There are smaller development houses and mobile enablement groups inside corporate IT departments, for whom test on a dozen hardware platforms is a burden, but not an unbearable one. Then there’s the population of individual developers, mythically laboring alone in attics and garages and basements, whose personal budgets are stretched by having to buy handsets and tablets they don’t actually use on a daily basis.

It turns out that developers of all types report that they test on about a half-dozen actual device models (stay tuned for a revealing survey by Apkudo on this very topic). Bare bones efforts beg and borrow kit from friends and family, but more substantial organizations also rely on 80:20 statistics.

The Long Tail of Testing

Quality of software and of devices is governed by rational statistics. Perception of this quality is anything but rational. Testing apps on the top dozen shipping devices and testing devices with the top 20 apps will yield statistically relevant results and deliver a quality user experience to a numerical majority of users.

But what about those unfortunate outliers? Unhappy apps purchasers and device owners may constitute a grumpy minority population – a mangy long tail of dissatisfaction – but they can exert a grossly disproportionate influence on the success of apps and devices. When those grumbling gamers and disillusioned down-loaders make their feelings known in widely read reviews and weigh-down rankings, developers and device OEMs can regret their reliance on majority platform rules.

The whip snap of this long tail is why I find Apkudo interesting. Their platform for Android app testing and services for device qualification give small developers, regional operators and upstart OEMs a fighting chance. Their goal of testing apps on EVERY Android device and of hammering devices with thousands of apps will reduce the incidence of nasty market surprises for these individuals and organizations. And even better, it will greatly improve the Android user experience for the rest of us.

-Bill Weinberg
Senior Executive, Mobile Practice Partner – The Olliance Group
Principal Consultant and Independent Analyst – Linux Pundit
@linuxpundit