Which Device Cloud Service is Best for Mobile App Testing?

Although device emulators are widely available for all platforms, there is no substitute for testing your mobile app on actual hardware.

Mobile app device clouds come in all shapes and sizes, with different pricing models, target audiences, feature sets and use cases. The landscape can be confusing, so we’ve summarised some of the most popular options.

What did we choose in the end?

We have 1 app on iOS and Android and thought we needed about 200 mins / month on hardware devices. We will trial Kobiton, Saucelabs, AWS Device Farm and Microsoft / Xamarin in that order.

The other services did not offer iOS, were too expensive or had very confusing pricing, so were not selected for a trial. Keep a lookout for our follow up post!

AWS Device Farm

Link – https://aws.amazon.com/device-farm/

Devices – 206 Android and Fire OS devices; 152 iOS devices. An interactive device list is available

Pricing – Three levels are offered: pay as you go, $0.17/device minute ($10.20 / hour); unlimited testing, starts at $250/month; private devices (test on dedicated devices deployed exclusively for your account), starts at $200/month.

Price of 200 device minutes / month (for comparison) – 0.17 x 200 = $34 / month

Free Trial – Your first 1000 minutes are free* (worth $170)

 

Microsoft / Xamarin Test Cloud

Link – https://testcloud.xamarin.com

Devices – 2923 devices – https://testcloud.xamarin.com/devices

Pricing – Plans with varying levels of limitations and capabilities include:

Newbie – 1 concurrent device and 1 device hour per day – $99/month

Small Startup – 3 concurrent devices and 5 device hours per day – $379/month

Growing Business, $799/month; and Enterprise, contact for a quote.

Price of 200 device minutes / month (for comparison) – $99/month

Free Trial – A 30-day trial is available

Kobiton

Link – https://kobiton.com

Devices – 160 devices are available in several families, including iPhone7, Google Pixel, Galaxy S7 Edge and iPads.

Pricing – Three plans are available:

Indie –  Single user, 100 minutes – $10/month or $100 annually – additional minutes = $0.14

Team, $199/month or $2,000 annually; Growing Biz, $499/month or $5,000 annually.

Price of 200 device minutes / month (for comparison) – $10 + (0.14 x 100) =  $24

NB One part of the site says $0.10 / min and another part says $0.14

Free Trial – A 15-day trial is available – no credit card required

Results of Trial – Trial went very well. Very easy to use. Manual testing only. Sessions were recorded and you can take screenshots very easily. Good range of test devices and always had more than 10 available.

Perfecto

Link – https://firebase.google.com/docs/test-lab/

Devices – More than 10,000 smartphones, tablets and phablets

Pricing – Three plans include: Basic, $99/month/user or $1,188 per year/billed annually; Advanced, $299/month/user or $3,588 per year/billed annually; and Enterprise, call for quote.

Free Trial – None

Results – Did not try as no free trial and others seemed better value

Firebase Test Lab for Android

Link – https://firebase.google.com/docs/test-lab/

Devices – Only Android devices

Pricing – Firebase comes in three plans: Spark, for free, with limitations; Flame, $25 per month; and Blaze, a pay-as-you-go model.

Free Trial – None

Saucelabs

Link – https://saucelabs.com/

Devices – 308 devices are available, featuring Android from version 2.2.2 to 7.1.1 and iOS from 7.1.2 to 10.0.

Pricing – https://saucelabs.com/pricing

Many different models are available for monthly or annual pricing plans. For the monthly plan, options include: Manual, $29/month; Individual, $199/month; Expert, $399/month; Professional, $499/month; Small Team, $699/month; Enterprise, contact company

Manual – $29/month – 1 concurrent user

Free Trial – 14 day free trial – You have to click on a link to testobject.com to get access to real devices. Only 2 devices in the trial – for EU and for US, which was disappointing compared to Kobiton

2 CONCURRENT VMS
100 MINUTES of Automated Testing
60 MINUTES of Manual Testing
Access to Real Devices

rss
facebooktwittergoogle_pluslinkedinmail

Leave a Reply

Your email address will not be published. Required fields are marked *