User testing #4: Live app testing on Browserstack and Saucelabs

About cross-browser testing

Cross-browser testing is an intrinsic part of web and mobile application testing. In the real world it's not possible to test a web app or native app on every device available in the market. We have covered the basics of this kind of testing in our blog Effective cross-browser testing.

BrowserStack and Sauce Labs

Both BrowserStack and Sauce Labs are cloud-based cross-browser testing platforms. From Google Trends, we can see that users have shown more interest in BrowserStack than Sauce Labs.

Both platforms provide the following generic features:

• Web application testing on multiple browsers across different devices and operating systems

• The latest and older versions of browsers

• The latest and older versions of operating systems

• Options for changing the resolution

• Local testing by a tunneling feature

• Debugging and logging options

• Reporting bugs via screenshots with proper annotation

• App logs and network logs for native app testing

• Integration for raising bugs on Jira and GitHub

• API for automation testing along with screenshots and videos of executions

• Bug reports

• Analytics for tests executed (pass %, error % and number of tests executed)

• An option for uploading installation files for the app for native app testing

• Console logs, network logs and debugging options

• A browser or app proxy option

• A location or GPS option for testing apps/websites from a different location

• Network throttling for testing apps or websites under different network conditions like 4G, 3G slow or 2G

• Testing for simulated interruptions such as no network or wifi interruptions.

BrowserStack versus Sauce Labs

At Salsa we always analyse the tools before using them or bringing them into our standard practices. Currently we’re using BrowserStack mainly for cross-browser testing across different combinations of browsers and operating systems. We test various web components and user journeys on BrowserStack and provide results to our development team using Jira.

For one of our recent projects, we needed to test native iOS apps on multiple versions of iOS devices. Before performing actual testing we analysed both BrowserStack and Sauce Labs on various parameters, including:

1. Pricing: Various pricing and plans should be available based on the number of parallel users (number of users that concurrently use the platform). The platform should also offer a trial for testing a few important features before subscribing.

2. App installation option: If you’re manually testing a new native application you’ll either download the beta version from the app store or you can ask the developer to give you the installation file so that you can download the app on your device. Cross-browser platforms should provide the ability to upload such files or provide access to the app store or tools that hold these apps when under development, like Hockey or TestFlight.

3. Usability: The platform should have a good user interface so end users can easily switch between different screens with no screen lag when testing the app.

4. Logging: Whenever issues occur, web application developers quickly check browser or server logs. Cross-browser platforms should also provide features so that developers can easily debug the application they’re building.

5. Network throttling and location options: The platform should have an option for changing network speed, such as switching between 4G to 3G. They should also have location changing options so that users can easily test an app from different locations.

6. Device rotation: For testing a native app, the platform should provide the ability to switch between portrait and landscape modes.

7. Interrupts: The platform should have an option for disabling the network or killing the running app. This helps the user to test it under such conditions easily.

8. Reporting of bugs: Tool should have integration to directly log bugs on Jira and GitHub.

9. Gestures: Native apps support device touch gestures. Platform should let users perform touch-screen gestures such as pinch, zoom, swipe, scroll, etc.

10. Local tunneling: Local testing creates a secure, persistent tunnel between your local development/staging environments and the cloud platform. With local testing, all your web and mobile apps can be tested out-of-the-box, including those hosted on localhost, behind proxies and firewalls, and much more. Platforms should support this feature so that developers can test things easily.

Cross-browser tools should have most of the generic features listed above.

Comparing BrowserStack and Sauce Labs

Below is the comparison matrix based on the parameters above:

Salsa Digital's take

Overall both seem feasible, but at Salsa Digital we use BrowserStack as our preferred option because it’s pricing is slightly more competitive and the stability of the interface on devices is good, with minimal lag.