Wednesday, March 18th, 2009
Super fast client side searches – the Flickr way
Category: Examples, JSON
, Performance
, XmlHttpRequest
, Yahoo!
Over at the Flickr development blog, Ross Harmes, one of those lesser sung JavaScript heroes explains in detail how Flickr creates really fast client side searches and one of the implementations of these findings is the newly released find people faster feature:
The main findings of the team were that eval() is not only evil but also very slow whereas dynamic script nodes are fast but insecure. The solution was to do a custom evaluation of string data rather than using JSON:
Having set the performance bar pretty high with the last approach, we dove into custom data formats. The challenge would be to create a format that we could parse ourselves, using JavaScript’s String and RegExp methods, that would also match the speed of JSON executed natively. This would allow us to use Ajax again, but keep the data restricted to our domain.
Since we had already discovered that some methods of string manipulation didn’t perform well on large strings, we restricted ourselves to a method that we knew to be fast: split(). We used control characters to delimit each contact, and a different control character to delimit the fields within each contact. This allowed us to parse the string into contact objects with one split, then loop through that array and split again on each string.
- that.contacts = o.responseText.split("\\c");
- for (var n = 0, len = that.contacts.length, contactSplit; n < len; n++) {
- contactSplit = that.contacts[n].split("\\a");
- that.contacts[n] = {};
- that.contacts[n].n = contactSplit[0];
- that.contacts[n].e = contactSplit[1];
- that.contacts[n].u = contactSplit[2];
- that.contacts[n].r = contactSplit[3];
- that.contacts[n].s = contactSplit[4];
- that.contacts[n].f = contactSplit[5];
- that.contacts[n].a = contactSplit[6];
- that.contacts[n].d = contactSplit[7];
- that.contacts[n].y = contactSplit[8];
- }
Once this had been speeded up, all they needed to use was the YUI AutoComplete control and voilà – fast client side searches even with massive datasets.
Ross Harmes is the man. He is co-author of a FANTASTIC JavaScript book, “Pro JavaScript Design Patterns”, you all should buy it now.
Hello.An easy question.How can i split using control characters?
I mean after hex editing a text file and setting 0x039(ctrl+c) in the middle of a text,how can i use split? \\c doesn’t work.
Thanks,very interesting article!
ok it was as easy as foostring.split(/\cM/)…
thanks again for posting!
ok a few mistakes.forgive me.
i meant 0x03(ctrl+c) and \/cC/.
ok i think i have to sleep.thanks again and sorry for the posts.
ps:maybe editing would be a good idea :)
all this performances troubles and they ended up with a non-sense multiple array access? … that.contacts[n] access for each property? I think this one did not require a post like that …
that.contacts = o.responseText.split("\\c");
for (var n = 0, len = that.contacts.length, contactSplit; n < len; ++n){
contactSplit = that.contacts[n].split("\\a");
that.contacts[n] = {
n:contactSplit[0],
e:contactSplit[1],
u:contactSplit[2],
r:contactSplit[3],
s:contactSplit[4],
f:contactSplit[5],
a:contactSplit[6],
d:contactSplit[7],
y:contactSplit[8]
};
}
are they posting another bench now? :P
P.S. obviously if they have dynamic pairs they could simply cache the object which thanks gosh in JS is always by reference:
for (var n = 0, len = that.contacts.length, contactSplit, contact; n < len; n++) {
contactSplit = that.contacts[n].split("\\a");
contact = (that.contacts[n] = {}); // whataaaa!!!
contact.n = contactSplit[0];
contact.e = contactSplit[1];
contact.u = contactSplit[2];
... etc ...
}
I don’t understand this. It seems to me that the way you load your JSON does not change your ability to do auth for the URL that is requested. The insecurity in JSONP lies in it’s inability to prevent malicious code of the server being executed on the client.