Ajaxian » Testing

Shim uses node.js to test sites on multiple browsers

jvaughan — Sun, 15 Jan 2012 04:01:13 +0000

Shim was developed within the Boston Globe’s media lab as a way to study how Web sites look on various devices and browsers. A laptop intercepts all wifi traffic – this is redirected to a custom node.js server – which inserts a javascript, or “shim,” at the head of each web page that is visited.

The shim, once loaded in a device’s browser, opens and maintains a socket connection to the server, according to to Shim’s developers. Shim was written in 2011 by Chris Marstall, Creative Technologist at the Boston Globe. The software has been open sourced. Write the Shim originators on git.hub:

Whenever a new page is requested, the page’s URL is broadcast to all connected browsers, which then redirect themselves to that URL, keeping all devices in sync. Shim info is available on git.hub.

Code injection, error throwing

jvaughan — Sat, 11 Jun 2011 03:01:12 +0000

In a blog, Opera Software Developer Relations team member Tiffany B. Brown looks at code injection, error throwing and handling and mobile debugging. She notes Opera Dragonfly and its remote debug features provide a way to debug mobile sites from their desktop. Brown mentions WebKit’s recently added remote debugging capabilities, folded into Google Chrome developer tools. Pointed to as well are Bugaboo, an iOS app for Safari-based debugging; JS Console which is available on the Web or as an iOS app; and Weinre for WebKit-based browsers. In this entry, Brown looks more closely at Dragonfly remote debug and JSConsole.

Best Practices for test revisited

jvaughan — Fri, 08 Apr 2011 17:08:30 +0000

With Google and their apps like Search, Docs or GMail only a very small time is actually spent in the initial page load, writes Andreas Grabner in a recent blog post. Of course, much time is spent in JavaScript, XHR Calls and DOM Manipulations triggered by user actions. Grabner writes:

It is very important to speed up Page Load Time – don’t get me wrong. It is the initial perceived performance by a user who interacts with your site. But it is not all we need to focus on. Most of the time in modern web applications is spent in JavaScript, DOM Manipulations, XHR Calls and Rendering that happen after the initial page load. Automatic verification against Best Practices won’t work here anymore because we have to analyze individual user actions that do totally different things. The way this will work is to analyze the individual user actions, track performance metrics and automate regression detection based on these measured values.

Synthetic Event Library Syn Aims to Make Testing Easier

Rey Bango — Wed, 14 Jul 2010 16:54:17 +0000

The team at Jupiter IT have release Syn, a library which allows you to create synthetic events for use in testing. This standalone library is meant to assist in testing complex UI behavior by simulating user actions such as typing, clicking, dragging the mouse.

Testing rich, dynamic web applications sucks. At Jupiter, we've tried almost every testing solution available (qUnit, Quick Test Pro, Selenium, JsUnit, Env.js, TestCase) and all of them suffer from some fatal flaw.

Problems:

Manual - A tester has to run the tests manually on every supported browser. People are lazy.

Unit Tests Only - We need to test the app as a whole and complex UI behavior like drag-drop.

Low fidelity - We need to make sure the tests are reporting accurate results.

Difficult to write - We sling JS like a ninja monkey throws poo. We want to write tests in a nice JS API.

Expensive - A QTP license is 5k a person! I'd rather buy a vacation.

Support - We want to test Mac and Linux browsers.

We've solved all of these problems in our upcoming FuncUnit testing framework. It's a mashup of qUnit, Selenium, Rhino, and Env.js. But its core library, Syn, which does the work of simulating user actions with very high fidelity, is what we are releasing today.

So by using code like this:

PLAIN TEXT

JAVASCRIPT:

Syn.click( {},'hello' )
.type( 'Hello World' )
.drag( $('#trash') );

you can simulate clicking an element with id='hello', typing "Hello World", and then dragging your mouse from that element to an element with id='trash'.

Pretty cool stuff. Check out their demo to see how playback occurs.

Keep your vows; Keeping wed to Node

Dion Almaer — Thu, 01 Jul 2010 11:14:55 +0000

Vows can be a beautiful thing. Alexis Sellier of LESS fame, is becoming an open source star. This time around he brings us Vows an asynchronous-friendly behavior driven development framework for Node.js.

Write you BBD specs like this:

PLAIN TEXT

JAVASCRIPT:

// division-by-zero-test.js
var vows = require('vows'),
assert = require('assert');
// Create a Test Suite
vows.describe('Division by Zero').addBatch({
'when dividing a number by zero': {
topic: function () { return 42 / 0 },
'we get Infinity': function (topic) {
assert.equal (topic, Infinity);
}
},
'but when dividing zero by zero': {
topic: function () { return 0 / 0 },
'we get a value which': {
'is not a number': function (topic) {
assert.isNaN (topic);
},
'is not equal to itself': function (topic) {
assert.notEqual (topic, topic);
}
}
}
}).run(); // Run it

and you get a very nice report card out of the other end:

With macros you end up with very nice DSL syntax such as:

PLAIN TEXT

JAVASCRIPT:

{ 'GET /': {
topic: api.get('/'),
'shoud respond with a 200 OK': assertStatus(200)
},
'POST /': {
topic: api.post('/'),
'shoud respond with a 405 Method not allowed': assertStatus(405)
},
'GET /resources (no api-key)': {
topic: api.get('/resources'),
'shoud respond with a 403 Forbidden': assertStatus(403)
},
'GET /resources?apikey=af816e859c249fe'
topic: api.get('/resources?apikey=af816e859c249fe'),
'shoud return a 200 OK': assertStatus(200),
'should return a list of resources': function (res) {
assert.isArray (res.body);
}
}
}
// or even
{
'GET /': respondsWith(200),
'POST /': respondsWith(405),
'GET /resources (no key)': respondsWith(403)
}

The website itself goes into exquisite detail on the install process, sample usage, and more. The website also happens to be beautiful itself, and full of HTML5 markup to boot.

A/B testing your Cohorts with Google Analytics result views

Dion Almaer — Tue, 08 Jun 2010 11:37:28 +0000

PLAIN TEXT

JAVASCRIPT:

var header_test = new Cohorts.Test({
name: 'big_vs_small_header',
sample: 1, // we want to include all visitors in the test
cohorts: {
big: {
onChosen: function() {
$('#big').show();
}
},
small: {
onChosen: function() {
$('#small').show();
}
}
}
});
$('#big').click(function() {
header_test.event('Clicked on Header');
});
$('#small').click(function() {
header_test.event('Clicked on Header');
});

The code above shows you exactly how you could run a test that shows either a large, or small clickable header... and gives you A/B results on how many were clicked.

This is all view a new library called Cohorts by James Yu:

Cohorts is a simple, purely javascript, multivariate testing framework.

It allows you to easily run split tests for visitors on your site, showing them different designs, layouts, or whatever you want. Cohorts also allows you to track interesting events that occur for each of the cohorts. By default, it uses Google Analytics event tracking to store data, but you can customize it to use your own or another.

Very nice.

Mock Data Randomizer

Dion Almaer — Thu, 20 May 2010 11:16:50 +0000

Mike Wilcox has written about a nice little tool for folks who test their JavaScript code. The Mock Data Randomizer is fairly self-explanatory:

PLAIN TEXT

JAVASCRIPT:

rand.sentences(6,7,7,9); // # of words in sentences
rand.real = true;
rand.sentences(6,7,7,9);

would generate something like:

Whut koop mida miins knew eftor kind. Deys evin thaaght has plecu uthur melos. Eftor shaeld ha lift hend wath ets unto big. Huwovir till lergi wa kind olsa con bofare. Yaor sentonca sean cemu man in whoch man. Mi sentonca nught get fund very onswar tui plecu. Shew en geed liva knew poiplu childrun ilung gat.

Far night very land room for then we. First kind turned out is own usually back much. Read sometimes since does answer himself into much. Live not for large hand important they. People another sea across across even world. Water could things young by white own only very.

PLAIN TEXT

JAVASCRIPT:

rand.date();
rand.date({seconds:true});
rand.date({delimiter:"/", yearRange:-5});
rand.date({delimiter:"/"});
rand.date({min:new Date('11/20/1964'), max:"4/19/1969", delimiter:"-"});

would generate something like:

Mon Feb 14 2011 14:13:17 GMT-0800 (PST)
1296270214154
08/04/2005
10/10/2010
02-05-1967

Check out the live demo to see it in action and then view the source.

Put your own tests up on BrowserScope

Dion Almaer — Mon, 19 Apr 2010 12:00:36 +0000

Lindsey Simon has got a powerful update to BrowserScope, the community-driven tool to test and profile browsers. The new feature is exciting as it truly delivers the "community-driven" piece at scale: you can now add your own tests to the corpus, TestSwarm style. Now, the two are very different of course as TestSwarm is about having many browsers test *your* code, and BrowserScope is more about testing the browsers in general.

Here is what Lindsey says:

It seems like nearly every week we read about an interesting new hosted test for people to visit and run their browsers through (recent examples include html5test.com and findmebyip.com). Developers really love to poke at the black boxes they code for - and the matrix of browsers, OS, and networks is enormous. One thing I, and I presume other developers, would love to see are the aggregated results for these tests by user agent. Considering this is exactly what we built Browser to accomplish for our test categories, and that a user-hosted test feature has been on our Roadmap, the Browserscope team is happy to announce that we're opening up an alpha of our User Tests feature.

Conveniently, this past week a User Tests use case came up for me at work and so it's been a driver for building this feature. We began working on a UI component that we wanted to test for speed of task completion. After building up a test page with a timer and some deltas it dawned on me just how cool it would be to open up this test to the world, and aggregate the results. The test is kind of strange in that the the UI component is out of its context, and you can argue about the mechanics of the test itself, but I still feel like the results may be informative. Interestingly too, this test is exactly the kind of thing we would *not* want to feature on the homepage of Browserscope (it's more of a performance test than a spec/binary test). And yet, the backend system with its median aggregation, scalability, and user-agent parsing library is a perfect fit. So check it out - and see how other people are doing on the test (courtesy of Browserscope).

This is definitely a release early/often feature, and we want to be explicit that things may change or break in the API while we're in alpha mode. We may have to take the service offline briefly to fix things. But if you write tests for browsers and want to aggregate your results, sign in to Browserscope, register your test page and then read the HOWTO to start saving your results in the system. Please send any feedback to me or to our mailing list. We really hope to make this an easy system to use for the tests you're already hosting.

I hope that we see a bunch of tests in there. One advantage of the browser scope approach is that it self-updates. A lot of the other sites are static, and when new browser versions come out the new feature data isn't reflected. It could be cool for example, if the readiness visualization pulled the data from a JSON feed from BrowserScope :)

Thanks for doing this Lindsey and team!

Harmony: Bringing together great libraries to enable awesome JS testing in Ruby

Dion Almaer — Wed, 17 Feb 2010 11:05:04 +0000

Martin Aumont has released Harmony, which "provides a simple DSL to execute JavaScript and DOM code within Ruby."

This enables you to do very cool things such as unit test JavaScript in the same area as your Ruby tests:

PLAIN TEXT

RUBY:

class JavascriptTest def setup
@page = Harmony::Page.new
@page.load('public/javascripts/foo.js')
end

def test_foo
assert_equal "world", @page.execute_js(<<-JS)
foo = new Foo;
foo.hello();
JS
end
end

and you can even use JavaScript libraries.... as script tags are autofetched:

PLAIN TEXT

RUBY:

page = Harmony::Page.new(<<-HTML)

page.execute_js("$('#widget').innerHTML") #=> "ohaie"

This library builds on the shoulders of giants, one of which is Mr. Johnson, John Barnette who I had the pleasure of working with many moons ago. He is the person I think of when I remember that the best engineers that I have worked with haven't been computer scientists, but musicians and biologists. He is also a great fun guy.

Anyway, sorry for the aside.

If you are a Rails chap, you may also be interested in the Rails plugin holygrail.

Helium CSS: JavaScript Library to test your CSS usage

Dion Almaer — Mon, 18 Jan 2010 11:29:22 +0000

Geuis Teses has released an enjoyable library called Helium that has the goal of testing your stylesheets for unused style.

You inject helium into your site (e.g. put it in an included footer) and then when you hit the first page you will have a popup asking for the pages you want to test. Helium will find the style sheets that you use and will store that information and the page information inside of localStorage. Then it will XHR around (so your styles need to be on the same host), test each page, and finally give you results.

I put up a trivial example to give it a ride and ended up with:

I ran into a minor issue with regards to relative stylesheets so I did the Github thing and forked, fixed, and pull requested. I love Github :)

Evidence; that your code will work

Dion Almaer — Wed, 11 Nov 2009 11:53:00 +0000

Tobie Langel of Prototype fame has created another unit testing library for JavaScript. Another one I hear you say? Well, if Tobie did it.... it is worth checking out. Time to get some Evidence.

I found out about it over dinner in Berlin at the great JSConf.EU. At the table were 6 people from 6 different countries. Go JavaScript diversity :) I first asked "BDD?" and Tobie spat out "No, I friggin hate BDD, especially for JS" before I had even finished the question.

Tobie gave a talk on Evidence at the show, and explained the focus of the library:

Library agnostic (the scripty based one in Prototype was based on Prototype which is weird when you test yourself)
Environment agnostic (e.g. not just in the browser)
Self contained (no global pollution)
Built with async in mind (in JS / client server / anything real, you gotta do async)
Easy to automate

PLAIN TEXT

JAVASCRIPT:

Evidence.TestCase.extend('ArrayTest', {
setUp: function() {
this.array = ['foo', 'bar', 'baz'];
},
testFirst: function() {
this.assertEqual('foo', _.first(this.array));
this.assertUndefined(_.first([]));
},
testLast: function() {
this.assertEqual('bar', _.last(this.array),
'Failed to grab the last element of the array.');
this.assertUndefined(_.last([]));
}
});

Riot.js: JavaScript port of the lean fast unit test framework

Dion Almaer — Thu, 05 Nov 2009 11:12:09 +0000

Riot started as a lean Ruby unit test framework with tests that have a style like this:

PLAIN TEXT

RUBY:

Alex Young has now implemented Riot.js which brings you the lean framework in a format that can run stand-along via Rhino, or through the browser itself with tests that look like:

PLAIN TEXT

JAVASCRIPT:

Riot.run(function() {
context('basic riot functionality', function() {
given('some simple equality tests', function() {
asserts('a simple truth test should return true', true).isTrue();
asserts('isNull is null', null).isNull();
});
given('another context', function() {
asserts('equals should compare strings as expected', 'test string').equals('test string');
});
given('a context concerned with functions', function() {
asserts('asserts() should allow functions to be compared', function() {
return 'test string';
}).equals('test string');
});
});
given('yet another context', function() {
asserts('equals should compare strings as expected', 'test string').equals('test string');
});
});

QUnit – A JavaScript Unit Testing Framework

webreflection — Thu, 01 Oct 2009 17:30:34 +0000

There are many options for JavaScript Unit Testing but a "de facto" standard for the jQuery library, related plug-ins, and jQuery UI is QUnit. Nothing new so far? Check this out:

According to tweets from John Resig the state of QUnit is:

QUnit is now completely standalone (it no longer depends upon jQuery)
QUnit now works outside of a browser (in Rhino, Spidermonkey, V8, JavaScriptCore, etc.)
QUnit is now completely moved over to Github: http://github.com/jquery/qunit

The latest news is that QUnit is now a CommonJS/NarwhalJS package that can be installed using tusk 'tusk install qunit'.

Thanks John for bringing some fresh air into JavaScript Unit Testing and Agile projects! I wonder if there is any plan to use QUnit via Fireunit as well...

BrowserScope: UAProfiler++, Crowd source browser tests

Dion Almaer — Mon, 14 Sep 2009 10:07:55 +0000

We are good friends with Steve Souders, but his UA Profiler just got beaten by something much better: BrowserScope. Fortunately for him, he and a new team are the ones who beat it :)

Lindsey Simon says it best:

Browserscope is an open-source project for profiling web browsers and storing and aggregating crowd-sourced data about browser performance.

The goals are to foster innovation by tracking browser functionality and to be a resource for web developers.

Browserscope is based on Steve Souders' UA Profiler, and his original tests have been preserved here as the Network test category. Other test categories include Ian Hickson's Acid3 test (ported by Jacob Moon into Browserscope), Annie Sullivan's Rich Text Edit Mode tests, and John Resig's Selectors API Test Suite (ported by Lindsey Simon into Browserscope).

This one more evolution, and more is to come:

Visualize test result trends over time
Wall of fame, up-and-comers, Billboard top 50
More test categories - cookies, security, reflow
More contributors
Tagged/personalized test results
Normalize time-based results across platforms
User agent parsing library

The guys are on hand to announce BrowserScope at The Ajax Experience this week!

TestSwarm: Crowdsource your continuous tests for JavaScript

Dion Almaer — Wed, 26 Aug 2009 18:01:20 +0000

It has been a pleasure to watch John build TestSwarm and see its alpha release today. It is an ambitious project to help developers get real world testing across browsers. Here you can see it all at work:

John talks about some of the fun challenges:

TestSwarm ended up being a very challenging project to get to an alpha state (and probably will be even more challenging to get to a final release state). Dealing with cross-browser incompatibilities, cross-domain test suite execution, and asynchronous, distributed, client execution has been more than enough to make for a surprisingly difficult project. It's mostly written in PHP and uses MySQL as a back end (allowing it to run in virtually any environment). Patches will absolutely be appreciated.

And, has a detailed look at the project. This is early days (hence alpha) yet beyond stabilization, John has the vision for some cool features:

A pastebin-like service where you can drop in code and see the results come back, from many browsers, in real-time.
IDE integration for sending minor changes out for quick testing.
Manual testing of user interface code. Pushing manual tests, with instructions, to users for them to walk through.
Distributing tests to any number of browsers, rather than a specific sub-set. (You could use this to embed a tiny iframe in your site to collect test results from a small sampling of our users.)
The ability to drive and test browser code or extensions.

What would you like to see?

More resources

Selenium WebDriver Re-introduced

Dion Almaer — Tue, 12 May 2009 11:54:10 +0000

Simon Stewart is a smart man, and nice chap, that I had the fortune to meet quite awhile back at Google London. He is an active Selenium hacker, and has re-introduced Web Driver that lets you write this Java code:

PLAIN TEXT

JAVA:

// Create an instance of WebDriver backed by Firefox
WebDriver driver = new FirefoxDriver();
// Now go to the Google home page
driver.get("http://www.google.com");
// Find the search box, and (ummm...) search for something
WebElement searchBox = driver.findElement(By.name("q"));
searchBox.sendKeys("selenium");
searchBox.submit();
// And now display the title of the page
System.out.println("Title: " + driver.getTitle());

WebDriver has a simple API designed to be easy to work with and can drive both real browsers, for testing javascript heavy applications, and a pure 'in memory' solution for faster testing of simpler applications.

Interested? Check out the getting started guide where you do a Google Suggest test, and watch Simon from waaaay back in 2007:

JsContext to make your JsUnitTest look purty

Dion Almaer — Wed, 01 Apr 2009 09:14:34 +0000

Christian Johansen was sick of testNewUserWithClownShoesShouldSqueak code in his unit tests, so he took some inspiration from the Ruby Shoulda and Context frameworks and wrote JsContext which allows you to write:

PLAIN TEXT

JAVASCRIPT:

new Test.Unit.Runner({
"A new User": JsContext({
"with clown shoes": JsContext({
"should squeak": function() { with(this) {
// ...
}},
// ...more tests and/or contexts
}),
"without clown shoes": JsContext({
"should not squeak": function() { with(this) {
// ...
}},
// ...more tests and/or contexts
}),
// ...more tests and/or contexts
}),
// ...more tests and/or contexts
}, { testLog: "testlog" });

Using this, the test runner will label tests a lot more readable; where you may have been used to "testNewUserWithClownShoesShouldSqueak" you now get "A new user with clown shoes should squeak".

We are looking to standardize on a testing framework for Bespin. What do you like to use?

Improved Readability for JavaScript Unit Tests

Ben Galbraith — Mon, 30 Mar 2009 14:07:35 +0000

Christian Johansen is tired of:

mentally decoding test names like "testNewUserWithClownShoesShouldSqueak", so I've written some code to improve readability for JavaScript unit tests written using JsUnitTest (from prototypejs). It's heavily inspired by Ruby frameworks Shoulda and Context.

Tests with his new framework--called jscontext--look like this:

PLAIN TEXT

JavaScript:

new Test.Unit.Runner({
"A new User": JsContext({
"with clown shoes": JsContext({
"should squeak": function() { with(this) {
// ...
}},
// ...more tests and/or contexts
}),
"without clown shoes": JsContext({
"should not squeak": function() { with(this) {
// ...
}},
// ...more tests and/or contexts
}),
// ...more tests and/or contexts
}),
// ...more tests and/or contexts
}, { testLog: "testlog" });

Christian maintains that:

Using this, the test runner will label tests a lot more readable; where you may have been used to "testNewUserWithClownShoesShouldSqueak" you now get "A new user with clown shoes should squeak".

Christian's introduction/announcement blog post has a lot of additional details. Looks very cool!

TestSwarm: Scaling JavaScript Tests

Dion Almaer — Wed, 25 Mar 2009 12:16:17 +0000

John Resig is working on a new tool that tries to help us scale our JavaScript testing. When building an Ajax application, how nice would it be to have a system running tests across multiple browsers and platforms? This is necessary for anything more than a toy, especially since we have some flexible software that allows us to shoot our feet very easily indeed. We need to fix this:

All of this leads up to a new project that I'm working on: TestSwarm. Its construction is very simple. It's a dumb JavaScript client that continually pings a central server looking for more tests to run. The server collects test suites and sends them out to the respective clients.

All the test suites are collected. For example, 1 "commit" can have 10 test suites associated with it (and be distributed to a selection of browsers).

The nice thing about this construction is that it's able to work in a fault-tolerant manner. Clients can come-and-go. At any given time there might be no Firefox 2s connected, at another time there could be thirty. The jobs are queued and divvied out as the load requires it. Additionally, the client is simple enough to be able to run on mobile devices (while being completely test framework agnostic).

Here's how I envision TestSwarm working out: Open Source JavaScript libraries submit their test suite batches to the central server and users join up to help out. Library users can feel like they're participating and helping the project (which they are!) simply by keeping a couple extra browser windows open during their normal day-to-day activity.

The libraries can also push manual tests out to the users. A user will be notified when new manual tests arrive (maybe via an audible cue?) which they can then quickly run through.

All of this help from the users wouldn't be for nothing, though: There'd be high score boards keeping track of the users who participate the most and libraries could award the top participants with prizes (t-shirts, mugs, books, etc.).

The framework developers get the benefit of near-instantaneous test feedback from a seemingly-unlimited number of machines and the users get prizes, recognition, and a sense of accomplishment.

If this interests you then please sign up for the alpha.

Why Load Testing Ajax is Hard

Dion Almaer — Wed, 31 Dec 2008 14:01:56 +0000

Today we are fortunate to have a guest post by Patrick Lightbody, most recently of BrowserMob fame (and previously Selenium work, OpenQA, WebWork, and more). Let's listen in to him talk to us about load testing, and let him know your thoughts in the comments below:

I've been developing and testing complex web apps for a long time. I was the co-creator of WebWork (now Struts 2.0) and an early champion of DWR, writing one of the first AJAX form validation frameworks for Java web apps. But over the years, I noticed that as our web technologies and techniques got more sophisticated, our testing techniques were not keeping up.

That was why I founded OpenQA and helped grow Selenium to the popular testing tool that it is today. Selenium helps with functional testing of complex AJAX apps, but there isn't an equivalent for load testing, which is why I started BrowserMob, a new type of load testing service.

Traditional load testing

In order to achieve high levels of concurrency, traditional load testing tools (both open source and commercial) work by sending large numbers of HTTP requests as a way to simulate many concurrent users interacting with your web page. These tools work by recording the traffic that comes from a browser session and then requiring that the load tester tweak a generated script so that it worked properly when played back X times concurrently.

Common problems would be that the initial recording would embed in cookie values that were tied to individual sessions. Additional unique state might be encoded in other hidden form elements, all of which required some fine tuning after the fact. If you've ever tried to run a load test, this is probably a very familiar process. It has worked reasonably well up until recent years, but AJAX has made this process even more difficult.

Ajax + load testing = hard

The reason Ajax has complicated things is that it encourages more logic and state to run inside the browser session. This means that just watching the traffic across the wire doesn't necessarily tell the full story. The richer an app gets, the more difficult it gets to simulate the exact effects of hundreds or thousands of users hitting your site.

This is the problem I decided to solve when I started BrowserMob. It's on-demand, low-cost and uses real browsers to completely change the way load testing is recorded and played back.

Do real browsers really matter?

Real browsers absolutely matter. There are two major reasons:

It simplifies the script creation process by letting you avoid all the complexities and hacks you have to do with traditional load testing tools.
It ensures that you'll see 100% of the traffic and load against your site that a real user would cause.

We'll look in-depth at each of these topics separately to see how use of real browsers helps and how a service like BrowserMob compares to existing load testing technologies.

Simplifies script creation

In today's modern web applications, AJAX is just about everywhere. And we're not necessarily talking about super rich applications like Google Maps or Yahoo Mail, but even simple sites like google.com now use advanced AJAX techniques. See Google's auto-complete for a real-world example:

In this case, when typing values in to the search box, the web browser executes JavaScript logic that in turn makes AJAX calls to Google's search engine, asking for search suggestions to display. It does this on every keystroke that the user types in. This is a standard auto-complete control that most Ajaxian readers are very familiar with.

When recording a script with a traditional load testing tool, one of two things may happen here:

The recorder will see the AJAX traffic and capture it for playback in the load test
The record will not see the AJAX traffic and will only capture the request made when the user clicks the "submit" button

Obviously these Ajax requests are causing real load, so we want to make sure they get played back in a load test. Let's assume you're using a tool, such as JMeter, that does capture the AJAX traffic. Here's what that looks like:

The recorded traffic is effectively:



http://clients1.google.com/complete/search?hl=en&gl=us&q=b

http://clients1.google.com/complete/search?hl=en&gl=us&q=ba

http://clients1.google.com/complete/search?hl=en&gl=us&q=ban

http://clients1.google.com/complete/search?hl=en&gl=us&q=bana

http://clients1.google.com/complete/search?hl=en&gl=us&q=banan

http://clients1.google.com/complete/search?hl=en&gl=us&q=banana

Each key stroke by the user is included in each subsequent search term. Let's ignore the requirement of validating the results that come back from the AJAX requests for the moment (they are usually in JSON or XML format and difficult to validate using most tools). Instead, let's just add a twist to the load test requirement for doing searches: the load test must search from 100 different search terms.

Parameterization is very common requirement, since it ensures that the load is realistic and doesn't get cached in any unnatural way. This means that now in addition to searching for the term "banana", we're also searching for "apple", and "orange", among others.

However, this means your script can't just blindly submit requests to those previous URLs either, since those were tied to the "banana" term. Instead, they must search for the sequential characters of the respective search term, such as:



http://clients1.google.com/complete/search?hl=en&gl=us&q=a

http://clients1.google.com/complete/search?hl=en&gl=us&q=ap

http://clients1.google.com/complete/search?hl=en&gl=us&q=app

http://clients1.google.com/complete/search?hl=en&gl=us&q=appl

http://clients1.google.com/complete/search?hl=en&gl=us&q=apple

Unfortunately, this is where even the best traditional load testing tools fall down. They don't provide any help here, so it's up to you to figure out how to, if it's even possible, write complex scripting logic that breaks down the randomly selected search term by characters and then subsequently issue Ajax requests for each character in the term.

At this point, you're basically rewriting the same logic that the web app developer wrote originally. If you're a QA engineer, this may be difficult since you don't know all the internal AJAX logic coded in to the application. If you're the developer, it's still annoying because it's tedious and likely in a language other than the original JavaScript that you wrote your code in.

So how do real browsers help?

Because BrowserMob uses real browsers to both record and playback load, that means you don't have to worry about trying to simulate the logic in a web browser. Instead, all you have to do is record the human interaction with the browser, such as typing in a randomly selected search term. BrowserMob will then pass those instructions on to the hundreds or thousands of browsers participating in the load test, and those browsers will in turn "do the right thing" and issue the proper AJAX requests.

And if the underlying logic, such as the request URL pattern for those AJAX requests, changes? With traditional load testing it's up to you to detect and fix the problem. If your test uses real browsers to play back the traffic, your script won't need to change one bit - the new AJAX logic will be run by the browser in real time.

Ensuring realistic playback

We've seen how use of real browsers helps with script creation, but what about playback? As we just learned, using real browsers simplifies the process of recording and shrinks the behavior coded in to the script itself. This means we're letting the real browser - the same type of program your end users will use - make the decisions about what requests to make.

For example, when visiting http://ebay.com you might see the following page:

But reload the page and now you might see this:

Notice a difference? The upper right section has completely different images displayed. That's because eBay's home page chooses what to display based on complex and multi-variant logic determined at runtime. It's quite likely that it's going to be impossible for a load tester to know which images will be displayed on any given request.

It's true that some load testing tools will try to parse the pages in real time and figure out which images should be displayed, but that's hardly comforting once you've already learned they can't deal with even the most simply Ajax components, as we just saw. And as most AJAX developers know, resources such as images and stylesheets are more and more likely to come from complex JavaScript logic and not due to a simple static reference in an HTML page.

Instead, the only way to guarantee that every single object (image, JavaScript, AJAX request, advertisement from an ad partner, etc) gets requested is to use a real web browser during playback. While it is much more resource intensive, it is also a major time saver on both the front-end, as scripts are much simpler to write, and the back-end, as you can be confident that the most realistic level of load was produced.

So next time you hear of load testing happening on one of your Ajax apps, make sure those doing the testing understand the complexities and difficulties associated with testing a complex web app. Help them be on the lookout for the issues highlighted here.

Thanks to Patrick for writing this. Do you have something important to say? If so, contact us with your idea!