Simon Stewart is a smart man, and nice chap, that I had the fortune to meet quite awhile back at Google London. He is an active Selenium hacker, and has re-introduced Web Driver that lets you write this Java code:
WebDriver has a simple API designed to be easy to work with and can drive both real browsers, for testing javascript heavy applications, and a pure 'in memory' solution for faster testing of simpler applications.
Interested? Check out the getting started guide where you do a Google Suggest test, and watch Simon from waaaay back in 2007:
Christian Johansen was sick of testNewUserWithClownShoesShouldSqueak code in his unit tests, so he took some inspiration from the Ruby Shoulda and Context frameworks and wrote JsContext which allows you to write:
Using this, the test runner will label tests a lot more readable; where you may have been used to "testNewUserWithClownShoesShouldSqueak" you now get "A new user with clown shoes should squeak".
We are looking to standardize on a testing framework for Bespin. What do you like to use?
mentally decoding test names like "testNewUserWithClownShoesShouldSqueak", so I've written some code to improve readability for JavaScript unit tests written using JsUnitTest (from prototypejs). It's heavily inspired by Ruby frameworks Shoulda and Context.
Tests with his new framework--called jscontext--look like this:
Using this, the test runner will label tests a lot more readable; where you may have been used to "testNewUserWithClownShoesShouldSqueak" you now get "A new user with clown shoes should squeak".
John Resig is working on a new tool that tries to help us scale our JavaScript testing. When building an Ajax application, how nice would it be to have a system running tests across multiple browsers and platforms? This is necessary for anything more than a toy, especially since we have some flexible software that allows us to shoot our feet very easily indeed. We need to fix this:
All of this leads up to a new project that I'm working on: TestSwarm. Its construction is very simple. It's a dumb JavaScript client that continually pings a central server looking for more tests to run. The server collects test suites and sends them out to the respective clients.
All the test suites are collected. For example, 1 "commit" can have 10 test suites associated with it (and be distributed to a selection of browsers).
The nice thing about this construction is that it's able to work in a fault-tolerant manner. Clients can come-and-go. At any given time there might be no Firefox 2s connected, at another time there could be thirty. The jobs are queued and divvied out as the load requires it. Additionally, the client is simple enough to be able to run on mobile devices (while being completely test framework agnostic).
Here's how I envision TestSwarm working out: Open Source JavaScript libraries submit their test suite batches to the central server and users join up to help out. Library users can feel like they're participating and helping the project (which they are!) simply by keeping a couple extra browser windows open during their normal day-to-day activity.
The libraries can also push manual tests out to the users. A user will be notified when new manual tests arrive (maybe via an audible cue?) which they can then quickly run through.
All of this help from the users wouldn't be for nothing, though: There'd be high score boards keeping track of the users who participate the most and libraries could award the top participants with prizes (t-shirts, mugs, books, etc.).
The framework developers get the benefit of near-instantaneous test feedback from a seemingly-unlimited number of machines and the users get prizes, recognition, and a sense of accomplishment.
Today we are fortunate to have a guest post by Patrick Lightbody, most recently of BrowserMob fame (and previously Selenium work, OpenQA, WebWork, and more). Let's listen in to him talk to us about load testing, and let him know your thoughts in the comments below:
I've been developing and testing complex web apps for a long time. I was the co-creator of WebWork (now Struts 2.0) and an early champion of DWR, writing one of the first AJAX form validation frameworks for Java web apps. But over the years, I noticed that as our web technologies and techniques got more sophisticated, our testing techniques were not keeping up.
That was why I founded OpenQA and helped grow Selenium to the popular testing tool that it is today. Selenium helps with functional testing of complex AJAX apps, but there isn't an equivalent for load testing, which is why I started BrowserMob, a new type of load testing service.
Traditional load testing
In order to achieve high levels of concurrency, traditional load testing tools (both open source and commercial) work by sending large numbers of HTTP requests as a way to simulate many concurrent users interacting with your web page. These tools work by recording the traffic that comes from a browser session and then requiring that the load tester tweak a generated script so that it worked properly when played back X times concurrently.
Common problems would be that the initial recording would embed in cookie values that were tied to individual sessions. Additional unique state might be encoded in other hidden form elements, all of which required some fine tuning after the fact. If you've ever tried to run a load test, this is probably a very familiar process. It has worked reasonably well up until recent years, but AJAX has made this process even more difficult.
Ajax + load testing = hard
The reason Ajax has complicated things is that it encourages more logic and state to run inside the browser session. This means that just watching the traffic across the wire doesn't necessarily tell the full story. The richer an app gets, the more difficult it gets to simulate the exact effects of hundreds or thousands of users hitting your site.
This is the problem I decided to solve when I started BrowserMob. It's on-demand, low-cost and uses real browsers to completely change the way load testing is recorded and played back.
Do real browsers really matter?
Real browsers absolutely matter. There are two major reasons:
It simplifies the script creation process by letting you avoid all the complexities and hacks you have to do with traditional load testing tools.
It ensures that you'll see 100% of the traffic and load against your site that a real user would cause.
We'll look in-depth at each of these topics separately to see how use of real browsers helps and how a service like BrowserMob compares to existing load testing technologies.
Simplifies script creation
In today's modern web applications, AJAX is just about everywhere. And we're not necessarily talking about super rich applications like Google Maps or Yahoo Mail, but even simple sites like google.com now use advanced AJAX techniques. See Google's auto-complete for a real-world example:
In this case, when typing values in to the search box, the web browser executes JavaScript logic that in turn makes AJAX calls to Google's search engine, asking for search suggestions to display. It does this on every keystroke that the user types in. This is a standard auto-complete control that most Ajaxian readers are very familiar with.
When recording a script with a traditional load testing tool, one of two things may happen here:
The recorder will see the AJAX traffic and capture it for playback in the load test
The record will not see the AJAX traffic and will only capture the request made when the user clicks the "submit" button
Obviously these Ajax requests are causing real load, so we want to make sure they get played back in a load test. Let's assume you're using a tool, such as JMeter, that does capture the AJAX traffic. Here's what that looks like:
Each key stroke by the user is included in each subsequent search term. Let's ignore the requirement of validating the results that come back from the AJAX requests for the moment (they are usually in JSON or XML format and difficult to validate using most tools). Instead, let's just add a twist to the load test requirement for doing searches: the load test must search from 100 different search terms.
Parameterization is very common requirement, since it ensures that the load is realistic and doesn't get cached in any unnatural way. This means that now in addition to searching for the term "banana", we're also searching for "apple", and "orange", among others.
However, this means your script can't just blindly submit requests to those previous URLs either, since those were tied to the "banana" term. Instead, they must search for the sequential characters of the respective search term, such as:
Unfortunately, this is where even the best traditional load testing tools fall down. They don't provide any help here, so it's up to you to figure out how to, if it's even possible, write complex scripting logic that breaks down the randomly selected search term by characters and then subsequently issue Ajax requests for each character in the term.
At this point, you're basically rewriting the same logic that the web app developer wrote originally. If you're a QA engineer, this may be difficult since you don't know all the internal AJAX logic coded in to the application. If you're the developer, it's still annoying because it's tedious and likely in a language other than the original JavaScript that you wrote your code in.
So how do real browsers help?
Because BrowserMob uses real browsers to both record and playback load, that means you don't have to worry about trying to simulate the logic in a web browser. Instead, all you have to do is record the human interaction with the browser, such as typing in a randomly selected search term. BrowserMob will then pass those instructions on to the hundreds or thousands of browsers participating in the load test, and those browsers will in turn "do the right thing" and issue the proper AJAX requests.
And if the underlying logic, such as the request URL pattern for those AJAX requests, changes? With traditional load testing it's up to you to detect and fix the problem. If your test uses real browsers to play back the traffic, your script won't need to change one bit - the new AJAX logic will be run by the browser in real time.
Ensuring realistic playback
We've seen how use of real browsers helps with script creation, but what about playback? As we just learned, using real browsers simplifies the process of recording and shrinks the behavior coded in to the script itself. This means we're letting the real browser - the same type of program your end users will use - make the decisions about what requests to make.
For example, when visiting http://ebay.com you might see the following page:
But reload the page and now you might see this:
Notice a difference? The upper right section has completely different images displayed. That's because eBay's home page chooses what to display based on complex and multi-variant logic determined at runtime. It's quite likely that it's going to be impossible for a load tester to know which images will be displayed on any given request.
It's true that some load testing tools will try to parse the pages in real time and figure out which images should be displayed, but that's hardly comforting once you've already learned they can't deal with even the most simply Ajax components, as we just saw. And as most AJAX developers know, resources such as images and stylesheets are more and more likely to come from complex JavaScript logic and not due to a simple static reference in an HTML page.
Instead, the only way to guarantee that every single object (image, JavaScript, AJAX request, advertisement from an ad partner, etc) gets requested is to use a real web browser during playback. While it is much more resource intensive, it is also a major time saver on both the front-end, as scripts are much simpler to write, and the back-end, as you can be confident that the most realistic level of load was produced.
So next time you hear of load testing happening on one of your Ajax apps, make sure those doing the testing understand the complexities and difficulties associated with testing a complex web app. Help them be on the lookout for the issues highlighted here.
Thanks to Patrick for writing this. Do you have something important to say? If so, contact us with your idea!
John Resig has laid out his thoughts on a Web Developer's Responsibility, and it comes down to working with various up and coming browser versions and filing bugs:
It's safe to say that the biggest tax on a web developer is spending so much time dealing with browser bugs and incompatibilities. Thus it has become the favorite past-time of all web developers to complain about having to deal with them. Browser bugs are annoying, frustrating, and make your job incredibly difficult.
Because browser bugs are so frustrating and such a burden on top of normal development it should be the responsibility of every web developer to make sure that the browsers they develop for are able to find and fix their bugs. By taking responsibility for the bugs that you find - and to not assume that "someone else will find it" - will accelerate the rate at which browsers can improve.
The solution to helping browsers is two-fold: 1) Every time you find a browser bug, file a bug report to the respective browser. 2) Actively test your sites in the latest builds of the major browsers.
The vast majority of web developers have never filed a bug report with a browser vendor - or even used a nightly version of a browser - which is a shame. If you think about it there are few who are more qualified to assess what is going wrong in a browser than those who spend every day developing for them.
I'm especially surprised when I see professional developers not filing bugs with browsers, or testing on nightlies. Since one of the primary tasks of most developers is to paper over cross-browser issues it becomes in their best interest to see the number of bugs reduced (and making their job dramatically simpler).
Now, instead of just saying "go do it", John gives information on how to file a bug, where to go to do it, providing a good simple test case, and arguing for a bug.
He then shows examples of bugs that he has filed across browsers, and asks us all to wear them like badges.
So, got any badges? Been frustrated when filing a bug? What's the weirdest bug you have ever seen (sorry, interview question there).
I have been working on a little personal project of mine for the past couple of months. It’s called CFT and stands for Common Feature Tests. There’s nothing really ground-breaking about it; CFT is simply a set of so-called feature tests for Javascript (or rather for any ECMAScript-compliant flavor - be it Microsoft’s JScript, Mozilla’s JavaScript or WebKit’s JavaScriptCore engines).
These tests aim to detect all kinds of quirks that modern (and not so modern) browsers exhibit and detect those quirks in the most straight-forward way - using feature detection rather than browser sniffing. There are quitefewexcellent resources out there on a subject of “browser sniffing vs. feature detection”. It’s definitely worth checking them out.
CFT started as an initiative to replace as many unnecessary browser sniffing from Prototype.js as possible (now that I’m part of the core, I feel bound to do something useful for the library :)). The suite was then extended with other tests that seemed not quite trivial to implement.
I really hope CFT helps developers eliminate browser sniffing code plaguing their scripts. The project is under an MIT license and is hosted on github (although is still in its infancy).
You can walk through the tests to see results, and show the code, such as "WINDOW_EVAL_EVALUATES_IN_GLOBAL_SCOPE":
The most exciting part of Adobe MAX last week was a service that was announced by Paul Gubbay at the "Sneaks" session that shows cool tech that Adobe folk are working on. His (very early stage) service is called Meer Meer and it is genuinely useful.
You can plugin a URL and the system will render that page on a server farm (many browsers, cross OS) and show you the results. It can visually layer these results so you can see the differences. It was awesome.
I pinged Paul and had a nice conversation with him about the project:
Can you explain what Meer Meer is, and where you got the inspiration?
Meer Meer is the code name for a new hosted service that allows Web Pros to view their content on a variety of different browsers and operating systems without requiring them to install anything other than the standard Flash Player. The inspiration is easy. We spend a lot of time with our customers and cross-browser compatibility consistently comes up as the #1 issue unprompted. It’s a real pain and we wanted to help solve it.
What browsers and OSes are supported? This means I don't have to kick up VMWare to test in IE 6 now right? ;)
IE6 is covered :) We haven’t finalized the complete set yet but we’re committed to supporting the top browsers for Windows and Mac. We expect that mix to change over time as new browsers/versions gain market share.
What are the various views and features?
We’ve really focused on how users tweak/debug their CSS today and engineered the service to fit into that workflow. One of the unique features for Meer Meer is that it can work directly with your local content if you are using Dreamweaver. This is a big benefit as users can tweak CSS and hit the refresh button to get an immediate representation of the page on their target browsers. No need to save, publish to a public location and then view. In addition, we’ve focused on several views (1-up, side by side and onion skin).
Onion skin is the most interesting as it overlays two browser shots over each other and provides the user with complete control over the transparency. This view typically gets the big oohs and aahs :)
Can you talk about the high level implementation? I assume you have a server farm on the back end and you are rendering and sending back images of the rendered image?
That’s correct. In addition we’re using a Flex application as the client. This makes the application feel very snappy and respond like a desktop application without the need to install anything. The team has also done some great work to establish a local connection with Dreamweaver to work with local content. When you make a change to your code and switch back to Meer Meer it knows you’ve made a change and prompts you to refresh. You can choose to refresh from within Dreamweaver and a panel will update and show you the status of the screenshot retrieval in real time.
In Dreamweaver CS4 you showed taking an Ajax app, clicking around to change state, freezing the page and then sending it to Meer Meer. Can you talk more about the freezing feature and other things that you can do?
I’m really proud to be associated with this release of DW CS4. We’ve definitely pushed the envelope. Our new Live View leverages the WebKit rendering engine to provide users with a true browser experience directly within the product. This means that you can see and interact with dynamic content such as Ajax UI widgets. Furthermore, we’ve included the ability to see the Live Code that the browser generates as you interact with the page. Users can freeze JavaScript to get the page into different states and then use the new Code Navigator to jump directly to the CSS that effects the element they select. This is an added bonus for Meer Meer because we can place the page into different states, such as selecting the second pane of an Accordion widget, and then view the page in that state within Meer Meer. Other browser compatibility services simply can’t do this.
Any final thoughts on Meer Meer or other Web Pro services coming from Adobe?
We’re really excited about the future of desktop + services at Adobe. Meer Meer is a great example of what we can accomplish when we take a holistic view to solving a user challenge. We’re also very interested in extending the capabilities of our users by providing them with turn-key hosted solutions that allow them to grow their business. Content maintenance follows very closely on the heels of browser compatibility as a top challenge that our users face today. Our new InContext Editing service that is now in free preview on Adobe labs allows Web Pros to provide content editing capabilities directly within the browser to their end user. No programming required.
We look forward to hearing feedback from the community on these services to help guide our future direction.
This is an exciting server from Adobe, and shows that focusing on the Flash issue isn't the entire story. They have a lot to offer the Open Web if we look to the right places and we can continue to fight for more too :) If we support projects like Meer Meer we can make our voice heard.
Of course, this is just the beginning. I would love to see an API to the service which would allow any developer tools to mashup nicely with the service. You can also imagine actively analyzing the code to not only point out the diff in the image, but the issue in the code. When we get there, we will be in a very good place indeed :)
There were some other cool things from MAX that showed up in Labs:
Alchemy: A research project that allows users to compile C and C++ code into ActionScript libraries (AVM2).
Gareth Rushgrove has published some code for CSS Test, code that lets you get in there and test resulting CSS:
CSS is hard to test automatically. Their appear to be two potential approaches
which might have merit in solving this problem and and this sample code
represents one of them.
test_image.py
The basic idea here resolves around programatic comparison of images, some
based on screen shots and others generated dynamically from URLs. The
capability to analyse only a segment of a page has also been included.
The overhead introduced by the need to take screenshots of a site makes this
technique more useful for spotting regression issues in a live site than for
a test driven approach to writing CSS. Tools will be provided to make this
easier at a later date.
As this is currently a proof of concept in that it only supports testing in
webkit, and even then only on OS X. Support for other browsers relies on
other screen grab tools being integrated into the code.
This is not intended as a tool to check whether a finished website looks
identical to a set of photoshop images. It's intended to spot changes in
unexpected areas of a sites layout or design.
Here we plan on using Selenium to extract rendered DOM values like text-size
from a given web page and compare them against expected values. This could be useful both for regression testing and for assertion based test driven
development.
The sample code is written in Python but other implementations would be
straightforward. Much of the actual work is done via other commands; namely
webkit2png by Paul Hammond and the Imagemagik suite of tools.
Robert Kieffer has announcedJSLitmus a tool "designed specifically to allow you to quickly and easily write a JavaScript test (or test suite), run it on any modern browser, and document and share the results."
To see it in action, Robert writes a test on "++" and plots the results for different browsers, and then draws some conclusions.
JSLitmus.test('Empty function test', function(){});
Once you define your tests you can run them in the page thanks to the popup, and then it will do its thing and give you a Google Chart at the other end of things.
By now, most developers have (or should have) come to realize how important it is to build unobtrusive JavaScript code. Apart from ensuring a better user experience, today's tools and libraries make it extremely easier to embrace this practice.
Continuing down the path of providing developers good tools, Robert Nyman of DOMAssistant fame has updated his Obtrusive JavaScript Checker. The tool comes in a couple of different forms including a GreaseMonkey script, a Firefox add-on and newly included, a command for use with Mozilla's Ubiquity Firefox add-on.
One common and tedious task when improving code is to find inline events in the HTML code, and make sure they are implemented in an unobtrusive fashion instead (more about unobtrusive JavaScript). The web is riddled with inline events, and I strive to make it a better place. :-)
The Obtrusive JavaScript Checker tool traverses through all elements in a web page, and when it finds a HTML element with inline events, it highlights it with a red border. The newest release now offers support for "javascript:" links as well as much greater detail about the element when you hover over it. In addition, a new report summarizes the number of javascript: links and inline events in the current web page as well as the number of occurrences of each type of inline event.
Both Ext JS and Ext GWT applications can benefit from Selenium tests. In fact, with few exceptions, the tests created for one product should be interchangeable as both products produce the same DOM structure.
With GXT applications, GWT provides built in JUnit support. This provides a great way to test your application. However, these tests run only in host mode. Being able to test your compiled application in multiple browsers is important as some issues only appear within your compiled application.
In general, you create Selenium tests and then execute them in a variety of ways. This tutorial will demonstrate creating tests with Selenium IDE, a Firefox plugin, and creating tests within Java. Tests will be loaded and executed within the Selenium IDE, and Java JUnit tests will be executed using Selenium Remote Control.
Darrell walks through Selenium IDE and shows creating a test, and then Selenium Remote Control, which runs tests in the various browsers:
Selenium IDE provides a great way to create your tests and execute them in Firefox. Tests can only be run by manually opening Firefox and executing tests. What if you want to run you want to automate your tests and run them in other browsers? This is where Selenium Remote Control comes into play.
From the Selenium website: “Selenium Remote Control (RC) is a test tool that allows you to write automated web application UI tests in any programming language against any HTTP website using any mainstream JavaScript-enabled browser. Selenium RC comes in two parts. 1. A server which automatically launches and kills browsers, and acts as a HTTP proxy for web requests from them. 2. Client libraries for your favorite computer language.”
There are a lot of monkeys making appearances in the Ajax world. Tamarins, Greasey ones, Action monkeys, and now we have TestMonkey from the Appcelerator crew:
TestMonkey is an open source (Apache License) UI test framework we’re introducing today. It’s going to be fully integrated into Appcelerator (with some additional cool features on top if you’re using in an Appcelerator app). Additionally, it’s completely standalone as well and you can use on any web application - appcelerator-based or not.
Our main goal is to create a better UI framework for building out front-end test cases. We’re initially focused on unit tests. However, we’ll introduce higher level testing like use case testing soon. We’re going to offer some really neat features in the coming months to do much more advanced automated testing and quality control. So, stay tuned!
Features
All test suites run inside their own iframe sandbox. so, if you have any weird issues in one test suite or set of specific tests, you won’t screw up the others…
We’re providing a lot of convenience assertions for common UI testing .. thinks like checking for element attributes, element values, checkbox states, etc. are all as easy as pie.
We’re building a super cool UI on top for driving tests and the reporting of tests. Right now it’s pretty limited but we’re going to blow that out. Our goal is to provide as much information about failures, location of failures, expected results, etc. so it’s easy to figure out the issues.
TestMonkey has a clean API and can easily be extended, for example, to create your own assertion helper functions or test monkey plugin for handling results. In fact, our UI driver is simply an implementation of this plugin. You can easily hook into your own system if you’d like to handle results or do interesting things with them.
TestMonkey itself is very small and you don’t need to include anything in your application related to it. Test monkey can load your HTML files up in the sandbox and then your tests can run against the real source, no crazy includes or manually adding of test framework into the real app. It cleanly separates your tests from the real app.
"Steve, have you done a performance analysis on Google Chrome?"
Steve Souders did one better. Rather than just running some tests, he wrote a UA Profiler that allowed him to point a browser and let it roll through the tests:
UA Profiler looks at browser characteristics that make pages load faster, such as downloading scripts without blocking, max number of open connections, and support for “data:” urls. The tests run automatically - all you have to do is navigate to the test page from any browser with JavaScript support. The results are available to everyone, regardless of whether you’ve run the tests. See the FAQ for a description of each test and my list of caveats (additional tests that are needed, limited browser detection, etc.). Here’s the overall scores for some popular browsers.
Browser
UA Profiler
Score
Chrome
8/11
Firefox 2
7/11
Firefox 3
8/11
IE 6,7
4/11
IE 8
7/11
Opera 9.5
5/11
Safari 526
8/11
Getting back to where I started, let’s look at Chrome’s performance profile. It’s nice to see that Chrome is at the top right out of the shoot, having 8 of the 11 performance attributes tested, tied with Firefox 3 and Safari 526. The three performance features Chrome is missing are loading scripts and other resources in parallel, not blocking downloads when a stylesheet is followed by an inline script, and support for prefetch links. Looking at the Chrome user agent string we can see it’s based on Safari 525. If Chrome incorporates Safari 526 improvements, it’ll fix the most important performance trait - loading scripts in parallel. The detailed results show that Chrome has one improvement over Safari: caching redirects that have an expiration date in the future.
Steve is keen to add more tests, so what would you like to see? I think that there is room to tests for non-performance related features too. For example, tests such as "does the browser support postMessage" would be nice to see. There are a bunch of "is there window.Foo" at the shallow level, and then going deeper. CSS and selector tests could be added too.
Chad Myers has a simple look at GUnit, the jQuery based unit test framework. His article explains how to get going, and walks through a test like this:
