Benchmarks can be computations/time or time/computation, telling that one of these patterns would be totally useless blow it out of all proportions.

It’s not that one method is useless, it’s that it’s impossible to measure accurately in any browser but Chrome, and even then only to a resolution of 1 ms. In contrast, measuring operations per second gives you a great deal more accuracy, and also a great deal more distinction between sub-ms timing. Something that runs 0.01 ms takes 1/90 the time of something running 0.9 ms (using slightlyoff’s example), but are reported identically in a ms/op scenario. Obviously the distinction in this case is minute… until you run thousands of like operations in one go.

If the information gleaned from benchmarks is important, then the information gleaned from the more accurate benchmark is more important than otherwise.

@slightlyoff/eyelidlessness: Benchmarks can be computations/time or time/computation, telling that one of these patterns would be totally useless blow it out of all proportions. But henceforward I will also check the numbers with computations/ms. (gotta say thanks for sharing your benchmarking wisdom, as I asked for it)

I also did not add the latest acme (but dojo) because it returned several 0-results (for simple id selectors) and made the results useless (so much I learned about benchmarks in school, but I’m still trying to find my old notes). Maybe you can point me to a working version. Sizzle was not that different, in comparison to jQuery, since my jq tests already used jQuery.find (alias for Sizzle).

@sixtyseconds: Using a selector more than 10 times (or more) in a short period of time would have various common reasons. First of all is event delegation, which is more and more used (selectors are matched against a bubbling event). Think also about ideas like LiveQuery from jq, which queries the dom in a short interval for new elements. The same would happen when you use a simple query like “.title” in a loop on a list of elements. Today, many developers (usually programmer background) also overuse selectors, without thinking of the bottle neck situations.

most browsers (save Chrome, as eyelidlessness said) don’t have fine-resolution timers. That means that if your function runs in 0.9 ms and mine runs in 0.01ms, they both report as “0ms”. The point of doing enough iterations is to show what heavy usage of the selector engine will *really* look like in relative terms. Cumulative numbers (like slickspeed generates at the bottom) aren’t good for anything else anyway.

The objection is about methodological accuracy. Slickspeed is b0rked in this regard, but pumping up the # of revs gives you a simple way to address the timer resolution issue. NOT doing it simply obsfucates actual comparative performance.

Regards

I can’t speak for slightlyoff, but the point isn’t that running the same query N times is “real world”, but that because of limitations of the ECMA 262 implementation of most browsers (save Chrome), measuring the number of executions in a defined time gives you a more accurate measurement of the performance of the function itself. This translates indirectly to real world use in the sense that you won’t run the same query N times, but that you’ll run dozens of queries in a given setting, and that the real world speed of each of those queries affects cumulative speed of the application generally.

I’m really shocked that this needs any comment at all. Of course benchmarks are somewhat removed from “real world” use; the purpose is to test real world methods discretely and accurately, to identify general and specific performance boons and bottlenecks.

I think you didn’t understand the point at all: your tests aren’t measuring what you think they are. Running tests 10 times tells you *nothing* if the total accumulated test time is below the resolution of the system timer you’re using to measure with. And with Slickspeed on most browsers, they are. The version of Slickspeed I used to develop Acme against runs each test for at least 1s, not a fixed number of times, and reports # of queries/ms, not time to run a fixed number of operations. Your numbers aren’t measuring what you think they are.

Regards

Not much difference between the two in most other browsers. Sure, Sly did better in most but the differences tended to be around 10ms. I don’t know about you but I can’t tell the difference between 20ms and 30ms all that well.

Now, perform several similar operations in a matter of a single function, and watch 20ms vs 30ms become 200ms vs 300ms or 1000ms vs 1500ms.

Users do notice, and care about, speed.

Everything sucked in the IE variants except for IE8 which shows marked improvement.

Not much difference between the two in most other browsers. Sure, Sly did better in most but the differences tended to be around 10ms. I don’t know about you but I can’t tell the difference between 20ms and 30ms all that well.

Everything absolutely screamed in Chrome and Safari4 and jQuery beat Sly by a small margin each time. It was actually fun doing the benchmark in Chrome just to watch how fast it ripped through the test.

Therefore, I am left to conclude that if the selector engine is efficient in its task then its performance is heavily influenced by the browser. And if all browsers will eventually be as fast at Javascript as Chrome then the selector engine you choose won’t matter a whole lot.

Results are also different among libraries for p:contains(selectors), p:only-child div[class!=made_up]

The value of a selector engine probably depends on the application. For something relatively light weight in the first place, you[r users] probably won’t notice the difference in performance, and you probably won’t see your code get out of control. If you’re doing anything particularly heavy, you’ll start to notice the performance hit right away. And it’s clear from your comment that you aren’t using particularly complex selectors (which is only the tip of the iceberg for selector engines, but I digress), but it’s not safe to assume others aren’t.

A project I’m working on right now, for what it’s worth, relies heavily not just on selectors that would be a pain in standard DOM selection, but on caching and well-engineered performance that I’d be hard-pressed to build for myself in the same timeframe without the support of a library (in this case, I’m supporting both jQuery and Prototype).

* Dojo has good results, what’s the arguing about? I tested with ACME and the results were very similar.
* List of frameworks is long enough, this is not a jQuery comparison. If you need one, you can find on in their 1.3 release announcement.
* jQ test uses jQuery.find, which is an alias for Sizzle.
* Selectors run 10 times, so the results are totally super honest, I swear! If you want the responsible persons for the small bars in the graphs, ask the awesome browser developers.
* No, I love them! Those colourful graphs with stacked times, very informative and you can still see browser differences. Maybe you are confused because they are cut off, you have to open the image to see them fully. I’m just hoping that google graphs soon adds a glas or 3d effect, maybe gradients.

Numbers ran on a slickspeed setup that was also used to measure for other framework releases, with most used selectors on “real-life” template. To contribute, fork the speed test suite and share your benchmarking wisdom, I’m looking forward to it.

* the performance tests should be using Dojo 1.3rc2, not 1.2.3
* similarly, both JQuery 1.2.6 and 1.3.x should both be included
* for fairness, the tests should test Acme and Sizzle’s *stand alone* versions, not the versions that wrap returns in different types (dojo.NodeList and the JQuery object). Otherwise the tests are apples-to-oranges
* the times of most of the tests are below browser resolution on the test harness that’s reporting these numbers. That’s just flat-out dishonest benchmarking
* stacking results like this to show “aggregate times” isn’t useful, and is indeed deeply misleading

Some of these things are the fault of the source benchmark suite, some of them are the author’s own lack of experience with benchmarking. They should all be fixed, though. The numbers, as presented, are meaningless.

Regards

You can get the slickspeed suite I used from the github repo, linked in the article, so you can check against DOM Fragments and XML Documents (which are imo too often ignored in tests but yet important to support).

@Spock: “browser consistently and stability is more important”. That is why I worked first on the specs and added the slickspeed tests later. The full feature/quirks detection is still missing and therefore the next coming task. I’ll also exclude more pseudo selectors into additional files, so the core file keeps slim.

Regardless, I enjoyed the headline. :)

I’m a bit suspicious about the marketing of CSS selector engines, and the accompanying nth% faster that X library claims — it’s more hype than sizzle half the time.

I personally never had a problem with the overall combined size of TinyMCE – the only thing I dislike about it (and it really doesn’t matter most of the time) is the insane amount of time it take to upload it over FTP (especially within WordPress). That’s do more to the sheer number of files, combined with the inefficiency of FTP for such a large number of tiny files, and not necessarily the total file size.

I agree with most of that you posted btw. :-)