Tuesday, March 31st, 2009

TaskSpeed: More benchmarks for the libraries and browsers

Category: Library, Performance

>I didn’t want the TaskSpeed library task test suite to be lost in the Dojo 1.3 announcement. Alex called it out:

Pete Higgins has been working on a new set of benchmarks with the help of other toolkit vendors (to ensure fairness) called “TaskSpeed“. Dojo 1.3 wins by a wide margin. Across all the reported browsers so far, Dojo is at least 2 times faster than other toolkits on common DOM operations. We’ve worked very hard over the years to make sure that Dojo’s APIs don’t encourage you to do things that will hurt you later, and TaskSpeed finally shows how much this philosophy pays off:

Given that DOM is the primary bottleneck in most apps, these tests demonstrate how Dojo’s approach to keeping things fast pays off not just on micro benchmarks like CSS selector speed, performance improvements to single toolkit functions, or even file size – but on aggregate performance where it really matters. Dojo’s modern, compact syntax for these common operations doesn’t slow it down, either. For instance, if you go check out the TaskSpeed reporting page, you’ll see that where browsers are slowest (IE6/7/8, etc.), Dojo’s focus on performance pays off most. Why use a toolkit that’s going to hurt you when it really counts, particularly when Dojo so easy to get started with? Dojo’s Core has been designed from the ground up with APIs that encourage you to do things that are fast and keep you from doing things that are slow unless you really know what you’re doing. In some cases, we’ve made hard size-on-the-wire tradeoffs in order to keep actual app performance speedy. That hard engineering doesn’t show up in micro-benchmarks or single test release-over-release improvements or the “my toolkit is smaller” comparisons that some would prefer that web developers focus on. It’s easy to win rigged games, after all. It’s only when you see APIs composed together in real-world ways, across browsers, that you can start to see the real impact of a toolkit’s design philosophy. Dojo is designed to help you make things that are awesome for users, and that means they need to be FAST.

Other toolkits have released performance numbers of late, and most of them have been either reported badly or run without much rigor, so it’s exciting to see everyone finally pitching in to build end-to-end tests that show how library design decisions interact with real-world realities of browsers. The TaskSpeed tests have been designed to be both even-handed and reliable (no times below timer resolution, etc.). The reporting page is also designed to make the results understandable and put them in context. A lot of care has been taken to keep this benchmark honest. JavaScript developers have suffered at the hand of chart junk for far too long.

It is interesting indeed to see the browsers on the graph. I will let you guess which browsers are which, but the visual difference is astounding:

Go and run it yourself and let us know what you see!

Posted by Dion Almaer at 12:07 pm
29 Comments

+++--
3.9 rating from 54 votes

29 Comments »

Comments feed TrackBack URI

Ext results?

Comment by sroussey — March 31, 2009

Where’s YUI??

Comment by bshenanigan — March 31, 2009

Unfortunately, YUI and ExtJS were both unable to supply tests in time for this initial set (each library author or rep wrote the “tests” themselves). Both plan to be included in future versions – we’ve been talking off list.

Regards,
Peter

Comment by phiggins — March 31, 2009

Why can’t anyone use “faster” and “as fast” properly? Is it really that hard? Sorry, but it’s a big pet peeve of mine.

Comment by MattCoz — March 31, 2009

“I didn’t want the TaskSpeed library task test suite to be lost in the Dojo 1.3″
What I want to see is a tool without bias.

“Given that DOM is the primary bottleneck in most apps…”
The DOM is NOT the primary bottleneck. The database along with outbound http calls and IO is the bottleneck, not the DOM.

Comment by jaysmith — March 31, 2009

@jaysmith – I took great pains to prevent any bias from being being apart of the suite. The framework is a stock slickspeed modified slightly to accept a hash of tests by name to run rather than a selector to run.
.
Pragmatically, I inject Dojo into the page after the tests are run, as I didn’t feel like writing a XHR implementation to post the results. Any bias or unfairness potentially in these tests is unintentional and should be addressed.
.
http://github.com/phiggins42/taskspeed/tree/master
.
Alex ‘jumped the gun’ on announcing this. I had not yet had a chance to thank John Resig, Andrew Dupont, Thomas Aylott, or Valerio Prioetti for their participation in creating the tests for jQuery, Prototype, and MooTools, as well as their participation in updating the framework itself.
.
Regards,
Peter

Comment by phiggins — March 31, 2009

jaysmith:

You’re right that network I/O gates most client-side performance since most devs don’t know how to optimize for it. But amongst toolkits which are single files (which is the apples-to-apples comparison here), the network I/O winds up not dominating perceived user latency. Everyone’s paying about the same price for their toolkit, which puts the bottleneck someplace else. In this case, it’s now DOM. On super-complex pages, it might be CSS, but that’s less often the case.

Regards

Comment by slightlyoff — March 31, 2009

Another interesting observation is the test file size…

jq-tests.js 0.63 kb
dojo123-tests.js 0.80 kb
moo-tests.js 0.84 kb
dojo-tests.js 0.86 kb
proto-tests.js 1.15 kb

Comment by Les — March 31, 2009

@Les – absolutely. There have been many suggestions since the creation of the charts (I literally did them two days ago) about measuring the byte count of each test somehow as a weight in the results, measuring thresholds within each test block (most libs are within ms of each other in some tests), graphing the individual tests (or even collecting individual test scores) …

I’ll be ‘formally’ announcing TaskSpeed shortly, and will hopefully address any questions or concerns brought up here now and off-list with the other leads. We’ll be adding more tests (hard to do because everyone has to ‘agree’ to their fairness relative to the test), reducing iterations (and rather measuring single metrics repeatedly), yadda yadda … No point in blogging in here :)

Regards,
Peter

Comment by phiggins — March 31, 2009

jaysmith:

Before I forget to mention it, I’ve updated the original post with a clarification. Thanks for keeping us honest.

Regards

Comment by slightlyoff — March 31, 2009

As a side effect the tests can also serve as a great way to quickly get insight on how stuff is done across the libraries. Neato!

Comment by rasmusfl0e — March 31, 2009

@slightlyoff (Alex?) – I have a few questions.
.
Why do some of the Dojo tests get the benefit of using native appendChild, createElement, and innerHTML while other framework tests don’t ? (dojo123-test.js and dojo-tests.js)
.
Example:
mootools-sethtml-alt: div.set(‘html’, “alt content”)
.
jquery-sethtml: $(“div”).html(“new content”).length;
.
dojo-sethtml: n.innerHTML = “new content”
.
dojo-sethtml-alt: n.innerHTML = “alt content”
.
Why are the number of tests different for each framework ?
.
Why are some of the tests overly heavy ?
Example there are some Prototype tests using $w() and .each(), when a simple array, or ._each call could be used ?
.
Very cool stuff, I dig the concept of method execution benchmarks, though I think taskspeed has some tweaking to go through.

Comment by jdalton — March 31, 2009

@Peter

These benchmarks are a great tool for the framework developers, for tuning their performance. They aren’t so great for end-users.

I think, like myself, the majority of end-users don’t make their decisions about frameworks based on DOM operation performance.

How do you benchmark community, code style, feature focus, documentation, etc?

I’ll take a slower framework any day of the month if it is a better fit for my work. I really hope framework comparison doesn’t de-evolve into a ‘mine is faster’ discussion. That’s just silly. Just as silly as the ‘mine is smaller’ discussion, which thankfully, has faded a bit.

Comment by MaratDenenberg — March 31, 2009

@jdalton – I hope to address all of these questions in my initial announcement (which was founded around “this is cool, but is somewhat flawed, here’s why …” concept you mention), but wasn’t quite ready.

To answer the .html question quickly – the suite started because I wanted to test the new Dojo 1.3 api’s for the “basic dom stuff”, and when making the 1.2.3 equivalents the api’s simply did not exist. The [new in 1.3] api for you above examples are:

dojo.query(“div”).addContent(“new content”, “only”).length

.addContent does the same dom-normalization (eg tables, tr, tbody, option, etc) as any lib … I suppose 1.2.3 gains from being able to be forced into using .appendChild / .innerHTML as well as that particular test not requiring any special normalization in the dom.

Regarding $w() and .each(): I cannot speak for the Prototype samples, as I didn’t write them, only compared against the other tests to ensure they were all under the same “heavy” constraints as the others (doing the operations in the loops, order of operations, etc). If I’ve missed something to make it “unfair” in any direct way, it needs to be resolved.

Your comments are entirely valid, especially regarding the tweaking needed to make taskspeed more useful, I encourage you to help out. I really did just throw this all together for my own benefit, and has been admittedly released too soon (though the exposure will do nothing but make the process faster)

Regards,
Peter

Comment by phiggins — March 31, 2009

jdalton:

Because that’s how the other frameworks author’s chose to write their tests. This isn’t some nefarious plot, it’s how the people who wrote each library chose to show off their tool. If framework authors want to write their test differently, I’m sure Pete will be happy to take the patches (as he has so far).

I repeat: each framework’s tests were written *by the authors of those frameworks*. You can judge the code for each sample and decide for yourself whether or not the Dojo syntax is significantly worse than the other frameworks, but what you’re seeing is how each toolkit author would (presumably) recommend you write your code too. You might not like Dojo’s idioms (I think they’re just fine), but it’s pretty indisputable that idiomatic Dojo code is just flat out faster.

Regards

Comment by slightlyoff — March 31, 2009

Hi, I create another test (no use SlickSpeed) and the results are similar than TaskSpeed.
http://www.anieto2k.com/demo/fra/

Good job Alex.

Comment by aNieto2k — March 31, 2009

@MaratDeneberg – agreed. Being able to profile as well as unit test (bonus points for charts being generated from json!) is/would be great for us internally, especially if/when we can measure each test individually.

… and you are correct – you absolutely cannot benchmark any of those other aspects. (I suppose you could measure lines of code to pages/lines of documentation, but ? really …), though the tooltips do give a little bit of insight into “code style” on top of the speeds reported.

Comment by phiggins — March 31, 2009

@phiggins – I think you are handling the early release splendidly.
@slightlyoff – (Alex?) As the creator of Dojo I would expect a bit more thought from your replies. Many of your ajaxian comments seem confrontational and petty. For example take how phiggins responded to my questions and how you did. He avoided pushing negative assumptions and re-iterated a positive message. While you seem to suggest because I ask questions I a) am a conspiracy theorist, or b) dislike Dojo’s idioms.

Comment by jdalton — March 31, 2009

jdalton:

*phew*. That’s a lot of aspersions packed in there. Clearly I’m bad at dealing with them.

Good thing we had the sense to put phiggins in charge of Dojo ;-)

Regards

Comment by slightlyoff — March 31, 2009

slightlyoff: I think that throwing aspersions is far from what jdalton was doing there.

The fact is, you can’t use unrelated tests that don’t do explicitly similar things for benchmarking. Those tests have no relevancy to eachother. Maybe I’m missing something, but that’s what it sounds like you’re doing and dojo’s tests are the most performance-minded.

I think Prototype and jQuery’s first intent isn’t to make super fast tests for users; rather, I think it’s to show off the power of the framework.

Take that as you will, but I assure you those are facts and not aspersions.

Comment by mcaudill — March 31, 2009

@mcaudill – the point is that they should be doing explicitly similar things, at least as defined by the sample-tests.js plain-english, to the best of the ability of the api’s provided by each framework for a given version. Again, please let me explain/justify these experiments before making any assertions as to the validity of the tests. They’ve not even been `released`.

Regards,
Peter

Comment by phiggins — March 31, 2009

I would really have loved to see YUI in these tests.

I must say that I currently use Prototype for creating widgets and YUI 2.x for CSS normalization and History management, and am currently interested in YUI 3.x, even though it’s not a straight-forward progression from YUI 2.x at this point.

I also love Google Chrome and find it to be a game changer for Web applications and it’s good to give Firefox some tough competition at last.

YUI 3.x has some impressive features which also make YUI 3.x a little bit harder to be developed and a little scary given it needs build files and stuff… Even though to most end-users they just won’t need that and will rather just easily load the YUI files and make good use of them. My thinking is more of a library writer that could use YUI 3.x for more widgets, and find a way to convert my Prototype files to it…

I think YUI 3.x is speedy enough to give Dojo some tough competition in these benchmarks, which seem to be simple enough to not pose a great deal of problem when writing the tests. Meanwhile, the YUI guys have been working on its source code verbosity and it’s less scary on that front, so congratulations to them.

These transitions can be tough, see for instance the Dojo transition to the 0.9.x version to stay in the JavaScript realm. But at the end of it, good things can be expected… So. Give YUI 3.x a chance. ;-) Sooner or later it will be its turn to flourish.

Comment by jpedrosa — March 31, 2009

That said, I figure I better give Dojo a try. :-) It has come a long way since I last checked it out and Dojo hopefully won’t try to bastardize JavaScript in order to make it more similar to static typed languages.

Comment by jpedrosa — April 1, 2009

I don’t quite get the point of benchmarking these type of things. Much like my feelings on the CSS selectors stuff mentioned here the other day, I just don’t get it.

If I ever have a need to perform 16 heavy DOM operations in a row then I know to go with Dojo. Gotcha.

But the performance is heavily dictated by the browser. According to my runs; in Firefox3 dojo is 1100ms faster than jQuery, in Chrome2 dojo is 900ms faster than jQuery and in Safari4 dojo is 400ms faster than jQuery.

Therefore the major problem is in the browsers themselves and not the libraries we choose to use. If everyone would use Safari4 and the latest version of the libraries then things would be fine.

But I can understand the usefulness in this benchmark for the developers of the libraries to see where improvements can be had. I just wish the browser developers could get their respective JS engines to perform to much the same level.

Comment by travisalmand — April 1, 2009

@WebReflection – thank you! This is one of the primary reasons why I don’t use JavaScript libraries, the layer of abstraction they provide us with far to often is a burden on performance for nothing more than usability and syntastic sugar.

Sure, $(‘#myElement’).addClass(‘someClass’).show() may be cool and easy to use, but the overhead is way too much for my liking.

Comment by RyanMorr — April 1, 2009

@ryanmorr

I suppose that depends on code size of your project and your expectations of the library. I for one do not work on projects that require a huge dose of DOM manipulations that a library is too slow to handle as compared to the speed of raw code. Most of my end users won’t care that one code method responds in 20ms while another responds in 35ms.

That’s why I don’t get this kind of benchmark since it does not show what I would think to be real-world results. Do people commonly perform 16 heavy DOM manipulations in a row that slow things down? This benchmark is made to cause the library to brute force a large number of computations to give some indication of time spent completing the task since normal tasks are done so quickly you can’t tell they were done other than the outcome.

Because, I tell you, I can’t remember how many times I wanted jQuery to rip through a for loop 250 times altering the DOM as it goes and became so upset that it took a whole 170ms to do it.

DOM versus Library is just like the old debate of low-level versus high-level programming.

Comment by travisalmand — April 1, 2009

@travisalmand, it is not a religious war, it is just about the fact that Internet Explorer is the most penalized browser in this Web 2.0 scenario.

You should look with your eyes how bad it performs compared to other browsers in a WebOS like application I am working on here in my company and via Ext JS (and those guys perfectly know what I am talking about)

The problem here is not 20ms against 30, the problem is 100 seconds with internet explorer 6 against 3 seconds … can you spot the gap?

My post would like to bring attention about how bad IE is performing with all these libraries and common tasks … it does not matter if Chrome is 200ms rather than 100ms if IE performs 200x slower.

Dojo team simply said thanks for the test case and I am sure they’ll consider it for next releases. I hope other libraries authors will do the same but it is quite ovious than no library could perform as native DOM but at least these libraries could try to perform better in the slowest and most common browser :-)

Comment by WebReflection — April 2, 2009

@webreflection

IE is penalized no matter what because as a modern browser it pretty much sucks. But how bad it performs with Javascript backs up my thought that the browsers are the problem and not the libraries we use.

Is straight JS manipulating the DOM faster than a library? Of course it is and it will most certainly always be faster regardless of the browser. A library can get faster in two ways, an increase in efficiency or removing features. But features are the reason to use a library in the first place. If a library removes enough features you might as well dump it for old-fashioned JS.

But I think the point I’m trying to get across is that since the browser is the issue when it comes to compile speed there’s only so much the library creators can do to speed things up. No matter what tinkering you do to a mini I doubt it’ll ever compete with a stock Ferrari right out of the factory.

But I asked what’s the point of a benchmark like this? What’s the point of your own numbers? They in no way represent a real-world situation. Most JS applications rely on user interaction and I find it doubtful this much DOM manipulation happens when a user clicks a button. Maybe there is and I would like to see it out of curiosity.

How about a benchmark based on things people actually do? How about user clicks on button and it manipulates maybe five things on the page? Would that be so slow in IE6 to be noticeable?

Comment by travisalmand — April 2, 2009

obviously your web vision is limited to buttons and links, I am working on a sort of Web operating System and everything is created runtime.

Every jQuery.append does something similar, John tests about document.createFragment were focused on these problems.

That bench shows how different is the scenario if we use libraries in IE rather than manual tasks, which means that if we, as libraries developers, do not focus more on IE performances, all this cool stuff could become not usable anymore as soon as our applications become more dynamic. This is the point, 3 seconds against 44 with jQuery or others do not make sense at all, it does not matter if Chrome spends 100ms rather than 300ms if we are still talking about milliseconds. It concerns me that IE, with all these libraries overloads, is even slower, if it is possible.

I am not complaining neither blaming anybody, it is just a fact that I hope developers will consider. Dojo team apparently did without problems.

Comment by WebReflection — April 2, 2009

Leave a comment

You must be logged in to post a comment.