As more and more applications move into the Software as a Service (SaaS) model, their primary means of delivery is now the worldwide web itself, with off-the-shelf web browsers serving as their primary user interface. But although the delivery model itself has been simplified, the presentation model has become exponentially more complex. There are many different browsers based on many different rendering engines that must run not only on many different operating systems (hard enough) but across an increasing number of mobile devices, each with their own operating systems and with many different form factors.
When “mobile” device meant “mobile phone” (with the accompanying much slower network access speed) you could simply (ha ha!) create a mobile version of your site sans heavy graphics and videos. Many such sites are still live today (think ESPN mobile for example). But with the rapid adoption of tablets as the go-to mobile computing access point, the form factor world is no longer neatly divided into desktop and phone sizes. We are quite literally living in a world where display sizes can range from small smart phones to 27” retina displays.
Oh, and back to the SaaS thing… You’re no longer just serving up a digital version of your glossy literature. Your entire application is online, meaning that the UI is assembled dynamically from data that can change by the second. DevOps anyone?
But if you’re going to sell such a service, you’re going to want to keep your customers, which means that you’re going to want to make your tool meet them where they are – on whatever device using whatever browser and OS they already have. And this means that you’re going to have to test your service across all of those combinations of device, form factor, operating system, and browser. And, oh by the way, the browser publishers have gotten into a habit of releasing new versions every few hours.
It should come as no surprise then that the demand for the ability to automate at least some aspect of this testing has reached a fever pitch. And the market has responded – sort of. There are now a number of tools that are designed to allow testing across multiple browsers (or versions of the same browser) either simultaneously or via various incarnations of record-and-playback. As you might expect, such tools are of great interest to us as we are constantly seeking ways of meeting our clients’ needs more cost effectively without sacrificing quality.
In this paper, we present our findings from a survey of the state of these tools. We put specific emphasis on tools that can tunnel across a firewall, because although our clients’ applications are public once released, they are never public during development. And we (as you might expect) do almost all of our work remotely.
Alas, we have some good news and some bad news. While the recent progress in this area is encouraging, we were unable to find even one tool in this genre that we could rely on to replace manual testing across multiple native browsers. The features are definitely improving. And there are many good ideas and good approaches to this problem. But the tools at this point are themselves buggy, not to mention that they all produce false failures (which we can deal with). More importantly – and this is the poison pill – they all produced false passes of one kind or another.
Still, we’re hopeful that progress in this area continues and we’re looking forward to a time when we can reliably use one or more of these tools in our work. When that time comes, we’ll certainly let you know.
Download White Paper