But I disagree that traditional load testing support for Ajax has been good. That’s not because they are bad products. I think it’s just because they are the _wrong_ products for this problem.

At the end of the day, LoadRunner can never provide great Ajax support because it doesn’t run a real browser in real time. Today’s Ajax apps are just too complex to get in to the game of simulating the protocol-level traffic. Sure, it can be done with some level of work, but I just don’t believe it’s worth it.

AJAX + LoadTesting = Change

Rather than testing being hard, it is CHANGE that is hard. This is especially obvious with individuals who have become comfortable with their traditional testing tools. I was one of these engineers who had been testing for so long in 2-tier and 3-tier Web 1.0 architectures. When I first started seeing new client architectures, web services, AJAX and n-tier systems – it was definitely a new challenge, and it meant that I needed to CHANGE. But this is exciting change – because with change comes learning and growth.

I do disagree on the point that “…even the best traditional load testing tools fall down.” because is simply not true. It’s not accurate to suggest at the existing testing tools are falling down because these tools have virtual user or driver technologies for Web 1.0 architectures. We do have those older solutions, of course – but we also have new solutions. For instance, LoadRunner has had support for AJAX testing for nearly 2 years and that’s with a new driver and new scripting capabilities that seriously innovate the proces. The “traditional” Web 1.0 solutions in our tool are maintained because it is still useful solution for older Web architectures. Even the traditional testing tools have new innovations available for customers who are ready for change.

Rather than run an elaborate load test simulation to see if a web app is performing badly, then pick through the code to find out why, I’d rather assume it’s going to be bad (because it always is) and go straight to reading the code…

I’d be interested to see what self-service products you have seen in this realm…

Countless times I have seen projects where the testing team used a variety of self-service tools and either had problems getting the test to execute at all or executed the test incorrectly (problems setting up dynamic data sets, etc.).

For this reason, I think testing is best left to seasoned industry experts that know how to design, build and execute a successful test.

There are lots of self-service tools out there now but they are less expensive for a reason. They don’t come with the expertise and knowledge of how to build and execute a good test.

However, I disagree that it’s “easy” for most people. Based on your blog post, you have very deep technical knowledge, going so far as to patch a C library to achieve your goals. That’s fantastic that you have those skills, but I’m not sure the average developers wants to dedicate that level of effort for someone as mundane as testing :)

While it certainly is possible to skip the browser, doing so introduces additional complexity that could have been avoided if a real browser was used in the first place. What I’ve found is that people who are responsible for performance testing (web developers, QA engineers, etc) often have a very hard time simulating browser session traffic, especially when that traffic has complex state embedded in the browser. This article outlines those cases, which a tool like HTTPPERF would still need complex scripting to achieve.

It becomes especially difficult, as I show in the Google Suggest scenario, when there is a need to provide random/parameterized data in to each simulated virtual user. If you aren’t running a browser (or at least a browser emulator, such as HtmlUnit), then your load test scripts will need to duplicate much of the same AJAX logic that effects state.

For those that don’t have the system resources necessary to run real browsers and aren’t interested in my service, HtmlUnit (http://htmlunit.sourceforge.net/) is a decent alternative. It doesn’t consume as many resources, so it can be run in a larger quantity on a single machine. However, it also isn’t a perfect emulator and often chokes on complex JavaScript such as Dojo and jQuery. The team is always working to improve this though, so check with them for the latest status.

Thanks again for checking out the article!

Patrick

My writeup has some serious work to be done – but it show the core of how easy it is to write a test case script in order to load test your AJAX application. Test case script creation is as simple as firebug cut-and-paste of url requests, headers, and post data into a sequential flat file. That and the scripts are so straight forward and simple to read it’s easy to build a template script and simply edit the post arguments to build another test case.

When it comes to if the browser matters or not, it doesn’t. You can script a HTTP test case “session” to download assets on demand. You can even put randomized delays per request. You can simulate a completely non-caching browser or a browser that might cache certain assets at first load. The browser matters when unit testing, regression testing, etc. but not load testing – except just to aid in building test case scripts.

In the case to simulate a non-serial hightraffic environment, it couldn’t be easier to HTTPERF. I usually enlist several simple unix workstations on several networks to run my tests concurrently. This even works great in a unix/xen “cloud” environment. The last quick load test I simply instantiated several Amazon EC2 instances, loaded HTTPERF and all the test scenarios and let her rip. This is of course biased to a single connection – but took all of 10min to setup 3 virtual machines to begin load testing my application. Much more time was spent building the test cases of course.

I’ll polish my horrible blog post a little more to help make this process easier.