Thursday, November 13th, 2008

Timing in JavaScript and browsers can’t be trusted

Category: JavaScript, Performance

This is officially the week of John. If he delivers top notch posts for the rest of the week he wins an Ajaxian award or something. Maybe we need to bring back the “pack of cards” where each card is an Ajax personality and John gets to be Ace of Hearts or something.

I remember talking with some of the V8 team about how poor the world of timing is. Chrome is a lot more accurate in its timing, which can do it a disservice in browser performance tests. Some browsers would respond with “0” when Chrome would return “0.001” and it would hence suffer.

Add that to the flawed “just add up the total time for all tests” mentality of some tests and you end up with very skewed results (you could do amazingly bad on one test that in practice never matters and really well on the others, but it all evens out).

Here comes John with a post on the accuracy of JavaScript timing which came out of a bad situation:

I was running some performance tests, on Internet Explorer, in the SlickSpeed selector test suite and noticed the result times drastically fluctuating. When trying to figure out if changes that you’ve made are beneficial, or not, it’s incredibly difficult to have the times constantly shifting by 15 – 60ms every page reload.

This lead him to tests life on various browsers and operating systems and he put up the raw data for you to check out.

He concludes:

Testing JavaScript performance on Windows XP and Vista is a crapshoot, at best. With the system times constantly being rounded down to the last queried time (each about 15ms apart) the quality of performance results is seriously compromised. Dramatically improved performance test suites are going to be needed in order to filter out these impurities, going forward.

Posted by Dion Almaer at 12:54 am
3 Comments

+++--
3.6 rating from 20 votes

3 Comments »

Comments feed TrackBack URI

While in general the timer can’t be trusted due to quantization problems (noted by many, including John) and the 0ms problem (e.g., see my blog post http://lazutkin.com/blog/2008/mar/23/javascript-edp-and-0ms-timeouts/ — just scroll down to the table), it is still possible to combat such issues with the good old statistics. Yes, make sure that your measured values are at least 2-3 times more the quantization barrier, run stuff many times, and collect stats.

Unfortunately the garbage collection penalty still distorts the picture — you will have outliers, and macro level wobbling on different runs.

Without reliable profiling it is impossible to develop software seriously. Personally I found that the averaging doesn’t work in most cases, and the best way to judge the performance is the median value. From experience it has far better correspondence to the real life.

Of course using a more complex properly weighted statistics will give even better results, the median value is simple to calculate (http://en.wikipedia.org/wiki/Median — just take a middle value from your sorted data array), and it works very well.

PS: While John singled out Windows XP, JavaScript on all existing OSes experiences quantization, the 0ms problem, and garbage collector-related random delays — all factors that skew the picture considerably.

Comment by elazutkin — November 13, 2008

Is there a reason Speedslick can’t just be rewritten to do everything 100 times inside the timing loop? I get zeros all over when I run it on the newer browsers.

Comment by Nosredna — November 14, 2008

I posted on this back in April (http://timepedia.blogspot.com/2008/04/work-on-gquery-progresses-comment-on.html) I thought the timer resolution issues in JS were pretty well known by now. The GWT benchmarking facility for example has long been designed around these limitations.

Comment by cromwellian — November 16, 2008

Leave a comment

You must be logged in to post a comment.