Thursday, May 3rd, 2007

Census: RIA Data Loading Benchmarks

Category: Adobe, Ajax, Dojo, Performance

James Ward of Adobe has created a Flex application that is able to run benchmarks on various RIA technologies called Census.

The benchmark renders a large data table using:

  • Ajax: HTML, SOAP, XML, JSON
  • Dojo: JSON to Dojo filtering table
  • Flex: SOAP to AS and E4X, XML to AS and E4X, AMF3, Paged

The benchmark results will tell you the time taken for server execution, transfer, parsing, and rendering.

The output itself is a large table that you can play with to feel the speed of filtering on the client. For the HTML versions James uses an iframe that he hides to a 1×1 version to the top right.

Run some tests on your own browser and play with the results. To show you that it is cool, it is black. All of the cool interfaces are black these days.

Census

Posted by Dion Almaer at 8:00 am
27 Comments

+++--
3.9 rating from 54 votes

27 Comments »

Comments feed TrackBack URI

I thought orange was the new black.

Comment by Andy — May 3, 2007

Spiffy UI, for sure! The Flex AMF3 transport looks to be the clear winner for performance (data size and parse time), as it passes binary data back and forth (raw AS objects I believe?) – plugins “ftw” here. :)

Comment by Scott Schiller — May 3, 2007

A guy from Adobe publishes a benchmarking tool written in flex to show that flex is the best performer….

I remember getting upset in the old /. days when M$ used to pull this.

Is the source available for both the client and server side (sorry Dion ;-) ) available, so that this can be replicated and checked by anyone?

Think of it as peer review, in attempting to get the real best performer.

I will admit I may wrong and James Ward’s test may be accurate. But the perception is one that leads to believe otherwise. If the code for both sides of the test are available, we can know for certain.

Comment by Mike — May 3, 2007

Thanks Dion for the post! There are more benchmarks and options I want to add for the next version including Gzip & Laszlo. More details on my blog:
http://www.jamesward.org/wordpress/2007/04/30/ajax-and-flex-data-loading-benchmarks/

-James

Comment by James Ward — May 3, 2007

Hi Mike,

All the code (client & server) is on SourceForge:
http://flexapps.cvs.sourceforge.net/flexapps/census/

Please do check it and make sure it is accurate. I’ve gone through quite a few revisions to make sure the benchmarks are accurate but there are probably still other things that can be improved. For instance an older version used DOM manipulation to create the HTML tables. While presenting this a while back someone informed me that it is a lot faster to create a big string and do an innerHTML. So I switched the Ajax benchmarks to do it that way instead.

I really created this app to be as fair and accurate as possible. Still, I encourage you to use these results only as an initial guide. You should always do your own benchmarks that more closely resemble your actual use case. And if you find things that should be changed to make the benchmarks more accurate, please send me a patch or let me know so that I can fix it. Thanks!

-James

Comment by James Ward — May 3, 2007

Do you use gzipping text and data for ajax calls, because in real world use they would be gzipped.

Comment by whatabout — May 3, 2007

does not work here (Konqueror 3.5.6 / Flash 9.0 r31)

Comment by Jan Mentzel — May 3, 2007

JSON is being sent text/plain @770k over the wire. Gzip (–fast) will reduce this to 60k, less than what Flex/AMF is sending over the wire. The test is bascially comparing a bytecode/jitted, optimized runtime versus an unoptimized, uncompressed ajax implementation.

Comment by Sur — May 3, 2007

I figured as much, also I use a little stronger than -fast but still, I figured they didnt optimize it and where comparing apples to oranges.

Comment by ahha — May 3, 2007

I do want to add a Gzip option for the next version. Using Gzip does reduce bandwidth, but that is not the only thing being measured. Gzip would also add server and client latency. Depending on the speed of a users connection this trade off may or may not be worth it. The transfer time measurement is only one of six different measurements being shown in this benchmark app. Also AMF3 is not being compressed either, so adding GZip on top of AMF3 would also further reduce the bandwidth. Please read my blog to get more details on all this.

Also I’m sorry this doesn’t work in Konqueror. As soon as Gentoo finishes compiling Konqueror I’ll test this and figure out what the problem is.

-James

Comment by James Ward — May 3, 2007

Who cares about the actual benchmarks? That UI is sexy as hell. ;)

Comment by Jesse Kuhnert — May 3, 2007

I’ll agree with Jesse!!!

Comment by Kahn — May 3, 2007

Thanks Jesse! :)

But I would like to see more conversation about the actual benchmarks. The most interesting benchmark to me (which actually isn’t directly measured in this version) is sort time. Compare sorting 20,000 rows in the Flex app with 500 rows in the Dojo Filtering Table. Sorting even 500 rows client-side in JavaScript is slow enough that it is typically faster to just go back to the server for the sorted data (I think this is what Open Rico does). But that has higher latency and server resource usage than just doing this kind of thing on the client. Filtering data is similar. Why go back to the server for something that can easily be done on the client?

-James

Comment by James Ward — May 3, 2007

There’s no question that JS datatables like Dojo’s and the excellent one in Ext become performance nightmares fairly quickly, when you move away from the 10-15 row demos and into 400, 500, and on up row tables. I made the mistake of implementing a 200-300 row table using a JS toolkit, developing the whole thing in Firefox (which wasn’t exactly snappy but seemed reasonable), then testing on IE and discovering that render and sort times were nowhere close to good enough. I’ve since moved over to Flex and won’t be looking back anytime soon. The thing absolutely flies, just like it does in James’ app.

I don’t have benchmarks, and have not deployed the new Flex version, but here is an example of what I tried moving over to a JS-fueled datatable: http://www.s-rod.com/tools/playerRanker.xhtml

(Note that currently, it is a JSF-powered table–i.e. server handles sorts, paging, updates via Ajax–which I’m fairly happy with, except for the fact that you’re maintaining state for a 400 row datatable between Ajax requests, which ain’t gonna scale if/when the site ever gets a lot of simulatenous users. And, obviously, it’s nowhere near as sexy/responsive/skinnable as a Flex datagrid.)

Not posting this as criticism of anyone … just wanted to share my experience, if anybody else is evaluating client-side datatable solutions. Maybe you can make it work acceptably with paging (rendering 20 or 30 rows at time), but if you wanna shove everything to the client at once, don’t expect to do it with Javascript. It’s just too much processing.

Comment by Rogers Reilly — May 3, 2007

The raw text with no annotations is ~ 340k so AMF3 @ ~ 80k is doing some sort of compression. It’s also a binary format so gzip/deflate may not have a great effect on it. The cost of mod_deflate @ low-compression (—fast) is minimal on any modern cpu. Transfer time was the largest factor in both the AMF case and the Ajax/JSON case.

It’s more than just compression, You could send lookup tables for for sex, classOfWorker, race, education, and maritalStatus. I have a hard time believing the web will move to binary formats (AMF, RPC etc.) over text (HTML, XML, etc.). Ever try to debug binary data over the wire, open or closed?

Comment by Sur — May 3, 2007

with Silverlight and ApolloFlexFlash both clearly having Faster JS implementations , isnt it just a matter of time before this makes it into browsers, what with MS likely beefing up the JS in IE using silverlight tech, and the AS3 engine now open sourced and going into mozilla? at that point, speed won’t be a selling point for silverlight/apolloflexflash

+1 to the sexy UI tho, almost makes me want to figure out how to use XUL widgets in normal webpages without serving up the whole document as XUL..

Comment by carmen — May 3, 2007

carmen: Yeah I took a look at the flex stuff with my friends the other weekend and think I’ll pass on all of that for actual programming, but the ui is still sexy either way. ;)

Comment by Jesse Kuhnert — May 3, 2007

Rogers Reilly – your flex grid only shows 30 rows at a time and does not run as quickly as many DHTML grids I’ve seen.

Comment by Jim Hilbert — May 3, 2007

as I mentioned in my post, that isn’t the flex grid- i haven’t deployed the flex version yet. it’s a JSF server-backed grid. (since every operation has to go back to the server, i’m not surprised you’ve seen faster DHTML grids.) was just referencing it as the sort of use case that doesn’t work with JS grids.

Comment by Rogers Reilly — May 3, 2007

“Sexy”? Not so much for me. I strongly prefer black on white as I have a hard time reading white on black. And this is why I dislike Flash and root for HTML/CSS. With a press of a button, I can instruct the browser to override the colors on a web site with a black on white color scheme. No way to do that in Flash.

I also would like to copy some of the census result data. Even if the tooltip would not go away, no copy & paste. Census is testing data tables and the conclusion is that Flex is fastest, but you can’t actually copy anything out of a Flex-based data table? WTF?!?

Comment by Martin — May 3, 2007

martin: Yes I know…..even after feverishing downloading all “build tools / ide extension / anal insertions” I was still left feeling a little gyped as far as a strong trustworthy programming model goes. .. Need an ide tool to get it done? Insta – f-you . Double f-you even.

Stop trying to constrain us google(gwt)/adobe. We like what we have – improve that and we’ll love you. :)

Comment by Jesse Kuhnert — May 3, 2007

Sorry guyz…..
I was just chking out tht weather script executes or not, in above comment.

No XSS vuln. Gud

cheeres

Comment by s — May 4, 2007

Hi Martin, You wrote:

““Sexy”? Not so much for me. I strongly prefer black on white as I have a hard time reading white on black. And this is why I dislike Flash and root for HTML/CSS. ”

You may not be aware that Flex also uses CSS. This article gives some examples: http://www.onflex.org/ted/2007/01/flex-201-understand-runtime-css.php

“I also would like to copy some of the census result data. Even if the tooltip would not go away, no copy & paste. Census is testing data tables and the conclusion is that Flex is fastest, but you can’t actually copy anything out of a Flex-based data table? WTF?!?”

You can copy and paste out text or data of Flex (and Flash). The developer can control this. This is a design choice, not an issue with Flex or Flash.

Regards,
David

Comment by David — May 4, 2007

Thanks James, that is all I wanted to know. I will check it out.

Comment by Mike — May 4, 2007

Hello David,

the CSS stuff does not seem to work to well. I open your RuntimeCSS example with Opera. Then I select the “High Contrast (Black on White)” user stylesheet. On every HTML webpage (like Ajaxian), a get a white background with black font for every elemetn of the page. If I try this with your RuntimeCSS example, nothing changes. Same thing for the census example.

If you take Firefox, I can force usage of my own page colors through the preferences or use the web developer toolbar to disable all CSS or load a user stylesheet. None of this affects the display of your RuntimeCSS example or the census example.

Flex Runtime CSS seems to allow the developer to use CSS for styling, but it does not allow the user to override the display through the browser when he needs to.

Thanks for the heads up about copy and paste. I have yet to come across one flash app that allows it. How come? Is it set to no as default and noone recognises that it is useful and often critical? Is the feature unknown to flash developers? Or is it too hidden in the menus?

It is extremely frustrating to come across a flash website and you can’t copy the product specs for reference. Or to open the contact page and you can’t copy the address data :-(

Comment by Martin Bialasinski — May 4, 2007

Martin,
What is this “High Contrast (Black on White)” thing? Sounds like something your browser has that applies to HTML content, not Flash content.

Most Flex & Flash apps are more like desktop applications, so copy and paste is something that developers can implement when it makes sense. But by default the Text and TextArea components in Flex support copy and paste. One company doing a Flex app that needs to have great c&p support is Virtual Ubiquity. Their app is a Word processor. It hasn’t been release yet, but you can find out more info at:
http://www.virtub.com/

-James

Comment by James Ward — May 4, 2007

This is correct, it is a user stylesheet. This is what my dislike about flash is: with HTML, I can make any black on white page (which is have trouble to read) into a black on white page with one click. I can make the font-size bigger on any page. With flash, I do not have this power. I am restrained to the micro-sized font and color combination the developer thinks looks cool and which I have trouble to read. And this control is critical for me, they are so many badly designed HTML pages and Flash pages. Flash is a huge step back usability-wise compared to HTML in that area.

> copy and paste is something that developers can implement
> when it makes sense.

Regulary users want to use the information in a way the developer has not forseen. Happens all the time. No problem there with HTML, but no luck with flash, unfortunately. If only flash would not default to the control-freak restriction side and simply allow copying any text by default. That would help so much. E.g. just yesterday, I wanted to copy a song name from one of the recommendations you get at the end of a youtube video. Does not work :-(

Comment by Martin — May 5, 2007

Leave a comment

You must be logged in to post a comment.