Monday, October 22nd, 2007

JSLoader: On Demand JavaScript Libraries

Category: JavaScript, Library

<>p>Dov B. Katz has released JSLoader, his open source on-demand JavaScript library. He explained to us:

It provides a methodology for organizing JS libraries, and programmatically loading them by simply “asking” for them. Example code: JSLoader.load(“ria”,”ext”,”2.0-beta1”);

I developed it as a mechanism to provide hosted Ajax libraries within a large enterprise (zero install, we maintain the latest releases) and it has been very successful. Furthermore, because no install is needed, I have leveraged TWiki to create a rapid prototyping environment, which has led to widespread adoption in the enterprise.

Ultimately, it’s just dynamically writing script and link tags onto the page… Not rocket science, but it works well, and it’s proven its value in an enterprise environment.

Why JSLoader?

  • First of all, it’s a zero-install solution. The goal is to eventually deploy this style of loading and file organization on a “hosted toolkit” system and allow websites to leverage the distribution of new toolkits without having to figure out how to install them.
  • Second, because multiple sites will share this code the browser and proxy caches will help make things more efficent
  • Finally, the maintainability of toolkits is minimized as content needs to be distributed only once, instead of having each user download a private copy

You can see some demos such as PlotKit.

Dov is going to be at The Ajax Experience this week in Boston. He is keen to meet you to discuss it!

Related Content:

Posted by Dion Almaer at 9:25 am
31 Comments

++++-
4.2 rating from 53 votes

31 Comments »

Comments feed TrackBack URI

I checked out the demos, and I just don’t get it.

How is it easier or better to include this javascript file, which then dynamically includes the javscript files you actually want, as opposed to just including them yourself?

I mean in both instances you have to know the name of the library you want anyways. And most likely you have this JSLoader in the same folder as your real libraries (like prototype) so, if you know the folder and the name of the library you want (which will basically be the filename) why not just include it yourself, saving the user of having to download the extra unneeded library?

Hopefully I’m just not getting it.
Rather then keeping all your javascript files in one spot, and including whatever

Comment by Nick — October 22, 2007

Its nice to see that some people have similar ideas… but I have to add that there is already a solution to on-demand-everything (JS / CSS / HTML ), utilizing the same techniques but taking it a step further – look at this:

http://www.two-birds.de
(programming in progress, you should use firebug so check the loading sequence)

Other than that, thats the way to go :-)

Comment by Frank Thuerigen — October 22, 2007

Whoops, obviously that last half-sentance wasn’t supposed to be there…

Comment by Nick — October 22, 2007

@Nick:
If you have a common codebase that is always a plus. On-demand loading also decreases the site loading times – as felt by the user – dramatically.

Comment by Frank Thuerigen — October 22, 2007

I’ve been doing similar work although I haven’t tackled remote loading for security issues:

http://www.openmv.com/OpenAjax/tests/all-compressed.html

I think there definitely needs to be a standard approach if this is going to be adopted. It’s mainly for bigger AJAX project where you might want to use several widgets or components from different frameworks. As Nick said, just the JS loader by itself doesn’t make much sense. It’s better to just include the script.

Comment by Jonathan Bond-Caron — October 22, 2007

look a lot like jsan
http://openjsan.org/

though I must admit that I like the general idea of jsan better, since I’m a perl dev :)

Comment by /dev/urandom — October 22, 2007

I guess this warrants a “Why JSLoader” topic on the wiki… I posted one http://vps.jsloader.com/twiki/bin/view/JSLoader/WhyJSLoader

Comment by Dov Katz — October 22, 2007

JSAN did seem to get this mostly right a long time ago:
http://openjsan.org/doc/c/cw/cwest/JSAN/0.10/lib/JSAN.html
Some take issue with it being sync instead of async, thus there is:
http://www.jspax.org/

I have participated in a few threads on the topic:
YUI: http://tech.groups.yahoo.com/group/ydn-javascript/message/10686
jQuery: http://groups.google.com/group/jquery-en/browse_frm/thread/258363a3aaf2a916/

The first question is always “why”. Answer, big footprint libraries or ajax heavy apps that do not use/need all methods at run time. If you never use a calendar widget why load the js for it? Furthermore, why load it on the first screen when only some users will need it on the 10th. The idea is, load the js when you need it.

Once someone is signed on to the idea then the discussion begins of sync or async, namespace integration or not, and eval /same domain (XMLHTTP) or DOM append /cross domain.

JSAN is sync with namespaces and eval /same domain
jsPax is async with namespaces and DOM append/cross domain

JSLoader looks to be closer to JSAN but lacking the namespace aspect. (I would have to dig further to be sure)

The inclusion of namespaces is really great and encourages good coding practices, but when using libs developed by a third party it is not always going to work (that YUI thread is about just this point) thus as JSLoader’s wiki indicates their goal is immediate adoption, they dropped it for now.

Ideally the namespaces would always be part of the package system while sync/async and XHR/DOM are dually supported depending on use case. (i.e. sync by default unless a callback is specified, XHR by default unless a URL is specified)

JSLoader seems to be targeted as a stop gap “lets try this out” solution until there is a wider adoption of this code style.

Word of warning: this approach needs to be used wisely as there is a tipping point where too many HTTP requests will bog your app down instead of giving you the performance boost you were hoping to achieve by not using a single 100k+ js file. In addition things like packer work much better with a single large file instead of many small ones.

Comment by Wade Harrell — October 22, 2007

oh yeah, forgot about this thread:
http://tech.groups.yahoo.com/group/ydn-javascript/message/15052
Douglas Crockford thinks the idea is a “nightmare”, lol

Comment by Wade Harrell — October 22, 2007

Isn’t this a huge security hole? You’re putting a lot of trust in the outside company hosting, basically, the brains of your client-side code. Hack them, and you’ve hacked all their clients. Maybe I’m missing something?

Comment by Brian Dillard — October 22, 2007

At the most basic level, this is nothing more than writing the same script and style tags on the page without having to know which ones to put and in which order. That alone has made it useful for my stakeholders. It’s not intended to solve the namespace problem, and for the majority of cases in an enterprise setting things are all eventually in your browser cache since multiple intranet sites share the same loader and hosting of assets.

100% agree. It’s important to know (1) what this is, and more importantly, (2) what it’s not.

Comment by Dov Katz — October 22, 2007

@brian – In an proper enterprise setting, this is intended to run locally (see jsloader.js source, it doesn’t even allow remote url’s unless it was loaded from one). As long as there’s a responsible maintainer of the infrastructure, it’s secure. Sure, if it is used maliciously, and hosted maliciously, its only as secure as the person you entrust with the hosting.

Comment by Dov Katz — October 22, 2007

it looks like there’s no Dojo fans inhere…well, I am! And it hurts me to see so many ppl talk about this without knowing how Dojo does it.

Comment by morriz — October 22, 2007

Morriz: I was thinking the same thing when I saw this. Dojo has namespacing and auto/lazy loading of everything from square one.

Actually, I think that someone has used the dojo loader to load other things as well (maybe mentioned above).

Anyway, it _is_ a great idea. For instance my web IDE would load twice as slow if I had to load all the stuff at once, instead of deferring, say, the code editor until the first time someone open a function.

Good idea, good competition. keep’em coming :)

PS

Comment by peter svensson — October 22, 2007

To the best of my knowledge, dojo’s package loader requires calling dojo.loaded() for remote modules, and local ones are loaded via XHR. It also makes assumptions about namespaces which would not make it easy to, say, load FCKEditor via the dojo loading system, though maybe I’m wrong.

Going back to “what it’s not” from my previous response. It’s not a dependency management system, nor is it a synchronous package loader or namespace management system. It’s just a way to abstract knowledge of asset location to make it easier for developers to use libraries. I’m looking at something along the lines of the efforts in OpenAjax Alliance’s space as a long term strategic solution to the problem this tool tactically solves.

Comment by Dov Katz — October 22, 2007

We’ve been able to load JS files on demand in Ajax Callbacks for more than 6 months now ;)
Oh yeah, we’re also taking care of the issue with types and methods from the JS file being referred to before file is finished loading…
One liner… ;)

Comment by Thomas Hansen — October 22, 2007

@Thomas http://use.perl.org/~schwern/journal/24112 JSAN started in April 2005
-
Not about who did it first, but who does it best. Dov seems to have a good approach realizing that not everything out there is going to play nicely yet. So instead build something that works in as wide a range as possible.
-
Next step, propose conventions and start talking about namespaces. Would be nice if JSLoader used dot notation so namespace migration would be easier in the future, but directory names with periods make that difficult. Perhaps a JSLoader.import method that uses the ideal implementation?
-
Also thinking it would help adoption immensely to do as many “ports” of JSLoader as possible, jQuery, YUI, etc. That way all the users of those libs can become familiar with the concept. Increases maintenance but if the result is wider adoption then perhaps the cost is worth the result.

Comment by Wade Harrell — October 22, 2007

@Dov, Wade: I really suggest you have a look at twoBirds principles http://blog.phpbuero.de/?page_id=10 and implementation http://blog.phpbuero.de/?page_id=9 articles.
At least it unifies site structure and you can easily import foreign libs using the same structure. Also it allows for asynchronous on-demand loading and handles requirements loading by using standardized coding patterns.

Comment by Frank Thuerigen — October 22, 2007

BTW one more thing: while I generally like the idea of a centralized lib repository – if all sites in an enterprise environment get their code from there, you have a single point of failure that will bring the enterprise to a grinding halt if the repository site fails.
IMHO it is better to have a proxy function that gets required JS files from the central repository and stores them locally on the application server. And this would be a no-brainer – using twoBirds – as well.

Comment by Frank Thuerigen — October 22, 2007

A bit more about my approach. We have a large scale replicated filesystem. While I talk about it being hosted, I have the identical filesystem mapped to all webservers’ docroots here.

The hosted solution works sometimes (its on load balanced regional pairs of highly available servers) the local one is also an option.

This eliminates any single point of failure concerns, as well engineered infrastructures do, in general.

If the globally replicated filesystem is down in general, I’d say you have more problems to worry about than your ajax libs!

Comment by Dov Katz — October 22, 2007

Some time ago I set up a similar system… it’s built a little different but it gets the job done as well… basically, you add a ns.tw.csi.registerInclude(function(){});
call to all your libraries and that’s pretty much it. It works perfectly with recursive dependencies and has very little overhead (code-wise). Of course, loading seperate files is considerably slower than one big server-assembled file but for me that’s not crucial and I love the fact that I get the correct filenames and line numbers in the erorr console.

Sample:
http://tapper-ware.net/devel/js/csi/csi.htm

Project Page:
http://csi.origo.ethz.ch/

Comment by Hans Schmucker — October 22, 2007

This is great if you want to do heaps more requests than needed.
On sites I work on, keeping the number of requests down is a significant challenge. Moreover, centralising the javascript store opens up your entire network of sites to significant downtime if the Javascript server goes down.

Comment by Dave — October 22, 2007

“We’ve been able to load JS files on demand in Ajax Callbacks for more than 6 months now ;)”

That stuff was done way back when webos.com was still alive and kicking, so it’s not exactly groundbreaking..

Comment by mikael bergkvist — October 22, 2007

I’m not saying it’s ground breaking, or revolutionary. I am saying it tactically solves a problem in a way that gets it working with any third party library effortlessly, which is something I unfortunately did not find elsewhere when I put it together. It does so in a way developers would *want* to use it, which is key for managing an enterprise infrastructure.

It would be easy to modify it to hit a servlet with all the “modules” to load and have it merge the JS on the fly and send it back in a single packed, Expiry-controlled response (for caching). At it’s core, it’s nothing more than writing script tags for you.

As I’ve pointed out, if you’re an enterprise (or Google, Yahoo, or some other large distributed content network) single points of failure are not a concern. You’ve engineered around them. The same reason I see sites using YUI hosted from Yahoo without worrying about yahoo going down.

I would *not* recommend relying on jsloader.com for the same thing :), but if, for example Google or Yahoo promised to host things resiliantly, and they’re able to pull it off, and they offered free lib hosting for libraries, developers would be able track use of their libraries, and maintain them much more easily than having people freeze them in time and use them locally. (imagine a google code, hosted libraries, google groups, and google analytics power pack offerring with the ability for google to get a first-hand view of the emerging AJAX libraries)

If you took a clone of the “repository” filesystem and replicated it on every web server hardware in your enterprise, you might benefit from not having to worry about individual sites/webapps installing the libraries on their own, and benefit from an always up-to-date repository of libraries. (Value for developers in ease-of-library use, and Value of infrastructure teams in knowing what’s deployed in their environment).

Comment by Dov Katz — October 22, 2007

Point taken. I wish that exact text of yours was in the post here from the start though.

Comment by mikael bergkvist — October 22, 2007

I actually liked the Import feature of Dojo Kit. Too good and organized.

Comment by Adnan Siddiqi — October 23, 2007

I work on the Dojo loader, both the synchronous, XHR-based one, and the asychronous, script-tag one (the xdomain loader). It is great to see more people trying to tackle this problem. Some points of clarification on Dojo, and some feedback on the general topic of JS loading:

The xdomain Dojo loader is a script tag loader. It requires a build process to prep the JS files for xdomain loading, but the benefit to doing so means that you can load the files after page load, and you get dependency resolution.

From what I can tell, Dov Katz’s JS Loader uses document.write only, so this will not work after page load. I also believe it will result in the same sort of “freezing” of the page that you can get with the regular XHR-based dojo loader if you try to load lots of modules as the page is loading.

The Dojo xdomain loader makes it possible to load all of the JS libraries from another domain, even a CDN, like the JS Loader tries to do. This means the xdomain loader delivers on the benefits in the “Why JS Loader” wiki page: No local installation of JS library files needed: add a script tag and go. You can use all of Dojo from AOL’s CDN using this approach. Not everyone will want to rely on a 3rd party site for their live site needs, but it sure makes prototyping and exploration easier. And you can always install an xdomain build on your own network, to at least get the async/more domain connection benefits.

I generally prefer requiring as little configuration to specify what needs to be loaded for a library. I think Dojo does good in this respect, requiring just a base URL to give the root of a library, then relying on a convention to load the rest (dojo.require(“foo.bar”) maps to http://base/url/foo/bar.js). Having good dependency resolution code also helps in this case. I know that is not a goal for the JS Loader, but I think it would help get away from needing the extra incr files.

For more info on some of these loader topics, I suggest checking out my Ajax Experience presentation on the xdomain loader, and my blog post talking about the YUI loader.

The latest jQuery 1.2 can load scripts via dynamic script tags (via jQuery.getScript). It does not have dependency resolution, but does the basic script loading, and it works after page load. It relies on script.onreadystatechange/onload to know when the script is loaded. I thought onreadystatechange was unreliable in a cached situation for IE 6. Although, I heard that a few years ago, maybe it is not true anymore. Still something I would like to investigate.

This is an interesting subject, I hope to see more discussion in this area.

Comment by James Burke — October 23, 2007

I prefer Amir Salihefendic at http://amix.dk/blog/viewEntry/174 loading method, the remote script is called only when you call the function that need the script library.

Ex:

function needsScripts(arg1) {
var div = AJS.DIV();
alert(div);
}
needsScripts = onDemand(needsScripts, ['AJS.js', 'greybox.js']);
needsScripts("Cow is mad");

Comment by Wendel — October 23, 2007

it is great to see such an interest in this topic. Dov you have done the good work!

Related to the topic I was emailed a link to http://en.wikipedia.org/wiki/AJILE which is yet another entry into the fray
and to recap the previous entries:
http://www.jsloader.com/
http://openjsan.org/doc/c/cw/cwest/JSAN/0.10/lib/JSAN.html
http://www.jspax.org/
http://csi.origo.ethz.ch/
http://amix.dk/blog/viewEntry/174
http://dojotoolkit.org/support/faq/how-does-dojo-require-work

So there are a few choices. What directory structure are you using? Personally I use /jslib/com/.. following the reversed domain approach. Is everyone using closures?

(function(){
if(typeof(window.com)===”undefined”){window.com = {};}
if(typeof(com.mydom)===”undefined”){com.mydom = {};}
if(typeof(com.mydom.myClass)===”undefined”){
com.mydom.myClass = function(){
var myPrivVar;
return {
myMethod:function(){
var foo = myPrivVar;
}
}
}();
}
})();

Comment by Wade Harrell — October 23, 2007

ajaxflakes – Read all about the latest developments on web design 2.0 and ajax + lots of tips. TOP 100+ best Free Opensource Software for windows XP and Vista. Thought i should add it might be helpful to others… http://ajaxflakes.com

Comment by startoy83 — October 26, 2007

I know about the xdomain loading of dojo, and a member of my team was at the Ajax Experience in San Francisco (I went to Boston) where I believe xdomain was presented.
-
The problem is, to date, I haven’t seen a (1) CDN host non-dojo files like EXT, JQuery, Prototype, and everything else on top of it, and (2) a loader that works with these other libs out of the box.
-
What JSLoader tries to do is (1) eliminate the need for developers to know where to put the assets within their infrastructure, and (2) eliminate the need for developers to know how to include/use the libraries.
-
A nice by-product is that there’s a twiki where all of these new libraries can be played with, without having to run a webserver or manage files anywher. I have not seen anyone else to date leveraging wiki for its amazing potential to be a playground for ajax developers. I’d love to see people hoist simple how-to examples up on a wiki somewhere (Not necessarily jsloader.com, but I tried to get a decent hosting plan so it could handle it if needed). The twiki I set up allows HTML and JS for that very reason.
-
Yes, I only do document.writes(), but I’m solving a much more simple problem (I do not aim to solve the complex problems): I want to use this for every library that currently exists, without making any major changes to the library, and essentially, I want the end result to generate the LINK(css) and SCRIPT tags developers would have needed to know to insert at the top of their documents, based on what they want.
-
In my production environment, in the firm where I deploy this, the end result are very empowered developers who have global access to a highly available filesystem hosting all AJAX libraries we bring in. The learning curve is minimized, which is key to adoption in an enterprise setting. Examples are deployed within twiki topics, and developers are encouraged to add their own examples.. and they do..
-
If I saw examples of dojo’s loader loading YUI, and dojo’s loader loading prototype, scriptaculous, jquery, etc, I’d think it would have solved my problem and there would be no need for JSLoader.
-
I hope that things within the realm of OpenAjax Alliance (I’ve spoken to Jon F. at the Ajax Experience about this) will bring the common loading strategy that I think this widely diverse collection of libraries need. In the meantime, my goal (and thus the purpose of jsloader in my workplace, and to a lesser extent, jsloader.com) is to make developers as productive as possible given the current landscape, and stop adding requirements the moment I stop adding value.
-
I think JSLoader fits in the short-term, tactical space, as I mention earlier in this board, and gets people thinking about organizing their personal use of the wide assortment of libraries out there in way that allows them to reuse their download efforts across their development projects. I’ve deployed jsloader on a personal host, and symlinked (IIS reader read “Virtual Directory”) it to the webroot of all of my sites. A great way to isolate code I wrote, from OSS code, in my revisiion control system, etc…
-
Either way, I’m not here to defend JSLoader. I don’t think it’s the best thing since sliced bread. I just aim to explain the problem it solved for me, and share it to the extent anyone else finds it useful. This discussion thread has been tremendously helpful in providing me with excellent feedback from industry experts and I appreciate everyone taking the time to participate.

Comment by Dov Katz — November 4, 2007

Leave a comment

You must be logged in to post a comment.