Although device emulators are widely available for all platforms, there is no substitute for testing your mobile app on actual hardware.
Mobile app device clouds come in all shapes and sizes, with different pricing models, target audiences, feature sets and use cases. The landscape can be confusing, so we’ve summarised some of the most popular options.
What did we choose in the end?
We have 1 app on iOS and Android and thought we needed about 200 mins / month on hardware devices. We will trial Kobiton, Saucelabs, AWS Device Farm and Microsoft / Xamarin in that order.
The other services did not offer iOS, were too expensive or had very confusing pricing, so were not selected for a trial. Keep a lookout for our follow up post!
AWS Device Farm
Link – https://aws.amazon.com/device-farm/
Devices – 206 Android and Fire OS devices; 152 iOS devices. An interactive device list is available
Pricing – Three levels are offered: pay as you go, $0.17/device minute ($10.20 / hour); unlimited testing, starts at $250/month; private devices (test on dedicated devices deployed exclusively for your account), starts at $200/month.
Price of 200 device minutes / month (for comparison) – 0.17 x 200 = $34 / month
Free Trial – Your first 1000 minutes are free* (worth $170)
Microsoft / Xamarin Test Cloud
Link – https://testcloud.xamarin.com
Devices – 2923 devices – https://testcloud.xamarin.com/devices
Pricing – Plans with varying levels of limitations and capabilities include:
Newbie – 1 concurrent device and 1 device hour per day – $99/month
Small Startup – 3 concurrent devices and 5 device hours per day – $379/month
Growing Business, $799/month; and Enterprise, contact for a quote.
Price of 200 device minutes / month (for comparison) – $99/month
Free Trial – A 30-day trial is available
Kobiton
Link – https://kobiton.com
Devices – 160 devices are available in several families, including iPhone7, Google Pixel, Galaxy S7 Edge and iPads.
Pricing – Three plans are available:
Indie – Single user, 100 minutes – $10/month or $100 annually – additional minutes = $0.14
Team, $199/month or $2,000 annually; Growing Biz, $499/month or $5,000 annually.
Price of 200 device minutes / month (for comparison) – $10 + (0.14 x 100) = $24
NB One part of the site says $0.10 / min and another part says $0.14
Free Trial – A 15-day trial is available – no credit card required
Results of Trial – Trial went very well. Very easy to use. Manual testing only. Sessions were recorded and you can take screenshots very easily. Good range of test devices and always had more than 10 available.
Perfecto
Link – https://firebase.google.com/docs/test-lab/
Devices – More than 10,000 smartphones, tablets and phablets
Pricing – Three plans include: Basic, $99/month/user or $1,188 per year/billed annually; Advanced, $299/month/user or $3,588 per year/billed annually; and Enterprise, call for quote.
Free Trial – None
Results – Did not try as no free trial and others seemed better value
Firebase Test Lab for Android
Link – https://firebase.google.com/docs/test-lab/
Devices – Only Android devices
Pricing – Firebase comes in three plans: Spark, for free, with limitations; Flame, $25 per month; and Blaze, a pay-as-you-go model.
Free Trial – None
Saucelabs
Link – https://saucelabs.com/
Devices – 308 devices are available, featuring Android from version 2.2.2 to 7.1.1 and iOS from 7.1.2 to 10.0.
Pricing – https://saucelabs.com/pricing
Many different models are available for monthly or annual pricing plans. For the monthly plan, options include: Manual, $29/month; Individual, $199/month; Expert, $399/month; Professional, $499/month; Small Team, $699/month; Enterprise, contact company
Manual – $29/month – 1 concurrent user
Free Trial – 14 day free trial – You have to click on a link to testobject.com to get access to real devices. Only 2 devices in the trial – for EU and for US, which was disappointing compared to Kobiton
2 CONCURRENT VMS
100 MINUTES of Automated Testing
60 MINUTES of Manual Testing
Access to Real Devices
