Wednesday, March 18th, 2009

Super fast client side searches – the Flickr way

Category: Examples, JSON, Performance, XmlHttpRequest, Yahoo!

Over at the Flickr development blog, Ross Harmes, one of those lesser sung JavaScript heroes explains in detail how Flickr creates really fast client side searches and one of the implementations of these findings is the newly released find people faster feature:

find people faster feature on flickr

The main findings of the team were that eval() is not only evil but also very slow whereas dynamic script nodes are fast but insecure. The solution was to do a custom evaluation of string data rather than using JSON:

Having set the performance bar pretty high with the last approach, we dove into custom data formats. The challenge would be to create a format that we could parse ourselves, using JavaScript’s String and RegExp methods, that would also match the speed of JSON executed natively. This would allow us to use Ajax again, but keep the data restricted to our domain.

Since we had already discovered that some methods of string manipulation didn’t perform well on large strings, we restricted ourselves to a method that we knew to be fast: split(). We used control characters to delimit each contact, and a different control character to delimit the fields within each contact. This allowed us to parse the string into contact objects with one split, then loop through that array and split again on each string.


  1. that.contacts = o.responseText.split("\\c");
  3. for (var n = 0, len = that.contacts.length, contactSplit; n < len; n++) {
  5.     contactSplit = that.contacts[n].split("\\a");
  7.     that.contacts[n] = {};
  8.     that.contacts[n].n = contactSplit[0];
  9.     that.contacts[n].e = contactSplit[1];
  10.     that.contacts[n].u = contactSplit[2];
  11.     that.contacts[n].r = contactSplit[3];
  12.     that.contacts[n].s = contactSplit[4];
  13.     that.contacts[n].f = contactSplit[5];
  14.     that.contacts[n].a = contactSplit[6];
  15.     that.contacts[n].d = contactSplit[7];
  16.     that.contacts[n].y = contactSplit[8];
  17. }

Once this had been speeded up, all they needed to use was the YUI AutoComplete control and voilà – fast client side searches even with massive datasets.

Posted by Chris Heilmann at 2:38 pm

3.6 rating from 23 votes


Comments feed TrackBack URI

Ross Harmes is the man. He is co-author of a FANTASTIC JavaScript book, “Pro JavaScript Design Patterns”, you all should buy it now.

Comment by cancelbubble — March 18, 2009

Hello.An easy question.How can i split using control characters?
I mean after hex editing a text file and setting 0x039(ctrl+c) in the middle of a text,how can i use split? \\c doesn’t work.
Thanks,very interesting article!

Comment by tasos — March 18, 2009

ok it was as easy as foostring.split(/\cM/)…
thanks again for posting!

Comment by tasos — March 18, 2009

ok a few mistakes.forgive me.
i meant 0x03(ctrl+c) and \/cC/.
ok i think i have to sleep.thanks again and sorry for the posts.
ps:maybe editing would be a good idea :)

Comment by tasos — March 18, 2009

all this performances troubles and they ended up with a non-sense multiple array access? … that.contacts[n] access for each property? I think this one did not require a post like that …

that.contacts = o.responseText.split("\\c");
for (var n = 0, len = that.contacts.length, contactSplit; n < len; ++n){
contactSplit = that.contacts[n].split("\\a");
that.contacts[n] = {

are they posting another bench now? :P

Comment by WebReflection — March 19, 2009

P.S. obviously if they have dynamic pairs they could simply cache the object which thanks gosh in JS is always by reference:

for (var n = 0, len = that.contacts.length, contactSplit, contact; n < len; n++) {
contactSplit = that.contacts[n].split("\\a");
contact = (that.contacts[n] = {}); // whataaaa!!!
contact.n = contactSplit[0];
contact.e = contactSplit[1];
contact.u = contactSplit[2];
... etc ...

Comment by WebReflection — March 19, 2009

This approach looked perfect, except for one thing: in order for this JSON to be executed, we had to wrap it in a callback method. Since it’s executable code, any website in the world could use the same approach to download a Flickr member’s contact list. This was a deal breaker.

I don’t understand this. It seems to me that the way you load your JSON does not change your ability to do auth for the URL that is requested. The insecurity in JSONP lies in it’s inability to prevent malicious code of the server being executed on the client.

Comment by fforw — March 19, 2009

Leave a comment

You must be logged in to post a comment.