Close

Contact us

Call Us on 1300 727 952
Find us

First Floor, 159 Victoria Pde
Collingwood, VIC 3066
(Google Map)

1300 727 952 
or
+61 3 9910 4099

 

Contact us

Close

User testing #4: Live app testing on Browserstack and Saucelabs

Selecting a tool that matches your needs is a daunting task. In this blog we compare Browserstack and Saucelabs for native app testing. The comparison matrix will help you decide which platform will best meet your needs.

Salsa Digital 5 May 2020

About cross-browser testing

Cross-browser testing is an intrinsic part of web and mobile application testing. In the real world it's not possible to test a web app or native app on every device available in the market. We have covered the basics of this kind of testing in our blog Effective cross-browser testing.

BrowserStack and Sauce Labs

Both BrowserStack and Sauce Labs are cloud-based cross-browser testing platforms. From Google Trends, we can see that users have shown more interest in BrowserStack than Sauce Labs.

Both platforms provide the following generic features:

  • Web application testing on multiple browsers across different devices and operating systems

  • The latest and older versions of browsers

  • The latest and older versions of operating systems

  • Options for changing the resolution

  • Local testing by a tunneling feature

  • Debugging and logging options

  • Reporting bugs via screenshots with proper annotation

  • App logs and network logs for native app testing

  • Integration for raising bugs on Jira and GitHub

  • API for automation testing along with screenshots and videos of executions

  • Bug reports

  • Analytics for tests executed (pass %, error % and number of tests executed)

  • An option for uploading installation files for the app for native app testing

  • Console logs, network logs and debugging options

  • A browser or app proxy option

  • A location or GPS option for testing apps/websites from a different location

  • Network throttling for testing apps or websites under different network conditions like 4G, 3G slow or 2G

  • Testing for simulated interruptions such as no network or wifi interruptions.

BrowserStack versus Sauce Labs

At Salsa we always analyse the tools before using them or bringing them into our standard practices. Currently we’re using BrowserStack mainly for cross-browser testing across different combinations of browsers and operating systems. We test various web components and user journeys on BrowserStack and provide results to our development team using Jira.

For one of our recent projects, we needed to test native iOS apps on multiple versions of iOS devices. Before performing actual testing we analysed both BrowserStack and Sauce Labs on various parameters, including:

  1. Pricing: Various pricing and plans should be available based on the number of parallel users (number of users that concurrently use the platform). The platform should also offer a trial for testing a few important features before subscribing.

  2. App installation option: If you’re manually testing a new native application you’ll either download the beta version from the app store or you can ask the developer to give you the installation file so that you can download the app on your device. Cross-browser platforms should provide the ability to upload such files or provide access to the app store or tools that hold these apps when under development, like Hockey or TestFlight.

  3. Usability: The platform should have a good user interface so end users can easily switch between different screens with no screen lag when testing the app.

  4. Logging: Whenever issues occur, web application developers quickly check browser or server logs. Cross-browser platforms should also provide features so that developers can easily debug the application they’re building.

  5. Network throttling and location options: The platform should have an option for changing network speed, such as switching between 4G to 3G. They should also have location changing options so that users can easily test an app from different locations.

  6. Device rotation: For testing a native app, the platform should provide the ability to switch between portrait and landscape modes.

  7. Interrupts: The platform should have an option for disabling the network or killing the running app. This helps the user to test it under such conditions easily.

  8. Reporting of bugs: Tool should have integration to directly log bugs on Jira and GitHub.

  9. Gestures: Native apps support device touch gestures. Platform should let users perform touch-screen gestures such as pinch, zoom, swipe, scroll, etc.

  10. Local tunneling: Local testing creates a secure, persistent tunnel between your local development/staging environments and the cloud platform. With local testing, all your web and mobile apps can be tested out-of-the-box, including those hosted on localhost, behind proxies and firewalls, and much more. Platforms should support this feature so that developers can test things easily.

Cross-browser tools should have most of the generic features listed above.

Comparing BrowserStack and Sauce Labs

Below is the comparison matrix based on the parameters above:

BrowserStack

Sauce Labs

30-minute free trial for users 14 days’ free access
iPhone XS and iPad Pro are provided during the trial period Few iOS devices are provided during the trial period
Installation options provided from App Store, Hockey app, Test Flight and App Center along with the default file-upload option for uploading the app to the cloud device Only file-upload option for uploading app installation file to test is provided during trial period

User interface looks sleek and shows installation options and devices available at single place

App installation options and devices available are controlled from different screens.

Data centre selection for this platform was not available Sauce Labs provides two dataCentre options: US West and Europe
App live testing has following options:
  1. Console log access
  2. UI inspector
  3. Network logger
  4. Throttling
  5. Device rotation
  6. App install/uninstall feature
  7. Location and language changes
  8. App kill option

Live mode provides following options:
  1. Screenshot option
  2. ADB shell
  3. Wifi disable option
  4. Enable/disable system animations
  5. Paste content into device

Integration for logging bugs to Jira, GitHub, Trello, etc. are provided Bug logging is provided using Jira and GitHub only
Local tunneling feature is provided Local tunneling feature is provided
Provides videos for automated tests Provides video recording for automated tests, as well as reports with number of bugs found, error %, etc.
For one parallel test user price is US$29 per monthSee pricing For one user plans start at US$19 per monthSee pricing
Provides basic gestures like swipe, zoom, scroll, pinch and tap Provides basic gestures like swipe, zoom, scroll, pinch and tap

Salsa Digital's take

Overall both seem feasible, but at Salsa Digital we use BrowserStack as our preferred option because it’s pricing is slightly more competitive and the stability of the interface on devices is good, with minimal lag.

Contact us

Subscribe to the Salsa Newsletter

Subscribe to the Salsa newsletter

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×