Tuesday, November 17th, 2009

Resource Packages; Making a faster Web via packaging

Category: Browsers, Performance

  1. <link rel="resource-package" href="site-resources.zip" />

What if we could agree on a simple backwards compatible approach to packaging Web resources so the browser could suck down the puppies in one go?

Alexander Limi (of Plone and Firefox UX fame) has been thinking about this for awhile, and has gotten his latest proposal out there which defines a simple way to package things.

It builds on existing tech: zip, manifests, and the like…. and degrades nicely.

This is so much nicer than spriting hacks, but does have its own issues:

  • In theory the browser could sometimes render faster by grabbing items in parallel
  • Mobile caches may be messed up (as larger than a cache size, like say iPhone’s 256k)
  • What if the zip gets out of date with the other resources?
  • Ideally tools (and in fact even web server modules) would auto generate this stuff

Although not perfect, for the majority case this could be a real boon to performance. What do you say Steve?

Oh, and it sounds like Firefox will implement this in 3.7!

Posted by Dion Almaer at 6:23 am

3.1 rating from 31 votes


Comments feed TrackBack URI

Would it be possible to specify multiple zips for one page?
Then you can split it up in 2 zips:
1 for the long-lasting content (like logo’s and css)
1 for the short-lived content (like news html)

Comment by Jaaap — November 17, 2009

It looks really nice anyways!!!

Comment by stoimen — November 17, 2009

I have always been thinking about it. It’s great to see it so well thought. Go for it!
Does anyone mentioned the possibility of having browsers to parallel download files in multiple parts now? Maybe that’s another point to consider.
Also… better compression options? GZip, 7Zip?

Comment by fredck2 — November 17, 2009

Screw hacking the planet. Hack the web!

Finally we are about to come to a place where all the major browser vendors (except MS ofcourse) grab this kind of stuff from the community and run with it in their next beta’s. AS IT SHOULD! The web should evolve on simple good ideas.

Comment by SchizoDuckie — November 17, 2009

Hmm, why is manifest.txt file needed? The zip would itself contain a list of the files?

Comment by Afraithe — November 17, 2009

I don’t really think it’s a good idea.
This looks more like a monkeypatch, it do not solve the underlying problem which is – fixing http pipelining – .

Also it introduce new problems :
– break cache
– zip compression may not / is not the best choice for every ressource in the package (images, binary) it may even generate overhead !
– overlap with and break http content negociation
– all ressources are either downloaded immediatly even if not necessary (in the case of a single package) or downloaded multiple times in the case of multiple packages with common parts (which is the case for 99% of the web)

Comment by ywg — November 17, 2009

The proposal looks really good. I was actually just thinking about this exact problem the other day, since I cringe every time I have to hack a CSS background position to make a sprite work! Looking forward to seeing this in Safari/WebKit.

Comment by sentientholon — November 17, 2009

I don’t see why this can’t be done with multiple link tags with a rel of prefetch. Only advantage this seems to have is that the files are zipped. Still good, but seems like an awful lot of work for yet one more distribution mechanism.

Also, I thought that the next version of Firefox was 3.6. Did that change again?

Comment by blepore — November 17, 2009

@Afraithe, the manifest.txt file is there to ensure that the browser has access to the file list as soon as possible (which is why it must be first). This is important as the browser can then decide which assets from the page must be fetched separately, without having to wait for the full .zip file to load.
I quite like the sound of this; it would work well for site-wide assets.

Comment by kissmyawesome — November 17, 2009

If all files are sent in one zip I guess you’d need the manifest to specify file order? As the order of CSS/JS matters, right?

Comment by mankz — November 17, 2009

I’m not wild about this idea, because I think it promotes an anti-pattern which is burdening the web right now. I believe we’ve gotten so paranoid about the overhead of an HTTP request that we’ve forgotten a few things:
1. It’s important to download things in parallel when possible. single big files are often loaded only serially, and this slows down initial load user-experience.
2. It’s important to not completely forget about or negate the cache of the browser. Sure, it’s not fully reliable, but that doesn’t mean it’s completely moot. It still has a very valuable role in improving user experience. But if we serve everything in one file (see, JS file concatenation), and even one character of that file has to change, all users have to redownload everything. This is bad.
3. Not all resources that a page uses need to be there before the page is rendered the first time. If you try and push everything down the wire all at once, you may reduce load on the server, but it comes at the price of forcing the user, especially a first-time user, to sit and wait unnecessarily. I would think people would remember that “you only get one chance to make a good first impression”. Don’t you want the loading experience of your site to be “friendly” to first-time visitors? Or have we returned to the idea of full-flash sites from 8 years ago that it’s ok to just stick a big pre-loader animation on the site and assume people want our content so much that they’ll just wait. This is bad UX. Plain and simple.
What we need is a flexible enough control so intelligent devs can choose among all these dimensions based on the specifics of your site’s resources. But I don’t think I’ll ever be convinced that sticking it all in one file is ever going to be the right answer. Let’s rethink this problem, it seems like it’s time to do so.

Comment by shadedecho — November 17, 2009

Idea sounds cool reducing the number of http request compressing the text, but how can we ensure the sites are compatible with old and other browsers ?

Comment by mridul — November 17, 2009

This is already mostly possible (with some accounting for backwards compatibility, which you’d need to do with this proposal for years to come anyway).

Any browser which supports Javascript will download a single Javascript file. Any browser which supports gzip compression will fetch it compressed (assuming proper configuration). Any Javascript file can embed styles and add them to the DOM in early processing. Any browser which supports base-64 embedded resources (like images) can deal with these from the same Javascript file. The only real issues are that some people turn off Javascript, and that some popular browsers don’t support base-64 embedded resources.

I’m not saying this is a solution. I’m thinking this is a good example of why this is not a solution. In fact, I’d say that if people started getting to sites which abused this, browsers or extensions will start to come around to block “resource packages” greater than N size (or using my example, simply disable Javascript, or simply any “resource packages”).

I think a better solution might be, to expand a little on one of shadedecho’s points (and take it off in a tangent), to have finer-grain control over cache headers and to serve files in smaller parts, only re-serving those parts which have changed, streaming on one HTTP connection with Keep-Alive. I don’t know, but I suspect the breaking up of larger files into smaller parts with different headers may not yet be supported. But with Keep-Alive, this becomes a practical solution and would be a much better extension to the existing system than the proposal, in my opinion. It produces the same effect (one HTTP connection to serve all content), while addressing most if not all of the concerns raised above.

Comment by eyelidlessness — November 17, 2009

I actually have experimented with this sort of thing. In the end.. It comes down the MS not adopting it :( that will hold back the web and ideas like this one. One day we will get together and decide on a way to kill that stoodgy bitch… force MS to start suing thus generating a massive media buzz and getting the word out that the coupdegras has been delivered.. muahaha

Comment by sourceRoot — November 17, 2009

Didn’t Firefox support JAR urls that work similar to this? Add to that a little caching and you can get the same effect.

[img src="jar:site-resources.jar!pics/pic1.jpg"]
[img src="jar:site-resources.jar!pics/pic2.jpg"]
[script src="jar:site-resources.jar!pics/runme.js"][/script]

Comment by Jordan1 — November 17, 2009

Lets first see how Chrome SpDy project is coming alone before we even consider extending or doing more with HTTP: :)?

Comment by V1 — November 18, 2009

I summarized my thoughts in a blog post last night. I also did a survey across the Alexa U.S. top 10 sites that shows the number of HTTP requests would drop from 42 to 17 if resource packages were used. This is a huge savings.

@Jaap: Yes, you can have multiple zips in one HTML page.

@Afraithe: The manifest file lets browsers know the entire contents of the zip without having to download the entire file. This allows the browser to continue parsing the page and know which resources will arrive in the resource package, and which ones it should request separately.

@ywg: The file sizes are actually smaller (see my blog post). Developers have control over the granularity of package contents to avoid the issue of overlap.

@blepore: LINK PREFETCH helps the *next* page. This makes the current page faster.

@shadedecho: There are likely other files to still achieve parallel downloads. Across the Alexa U.S. top 10, 37% of the resources would *not* be in resource packages (see my blog post). Developers don’t have to change the resource package if a single resource changes, although they probably will which is fine because the resources that would be packaged are likely to rev in unison (checkout the number of sites that have a common version # across all assets). You don’t have to nor want to put *everything* in the resource package, so there’s no issue with downloading unnecessary stuff too early.

@eyelidlessness: See my blog post for an explanation of why this is better than the current alternatives of concatenating scripts, concatenating stylesheets, and creating image sprites.

Comment by souders — November 18, 2009

@souders — OK, I can agree that *maybe* devs will be smart (and unlazy) enough to create their resource packages in a smart enough way that other non-packaged resources will be able to download in parallel at the same time, and still make good use of the browser’s bandwidth. And also, yes it’s possible they’ll be smart enough not to force down more content before domready than is really necessary.
But notice that these are some huge assumptions about everyone understanding *and* adopting possibly complex best-practices. It’s being awfully optimistic about the general UI dev community, isn’t it?
In reality, the potential for this approach to lead to anti-patterns is a big concern for me. Rather than bog down this comment thread, I’ve put up this blog post detailing some of my thoughts/responses:
Resource Packaging? Silver Bullet?

Comment by shadedecho — November 18, 2009

souders, I wasn’t talking about concatenating scripts, concatenating stylesheets, and creating image sprites. I was talking about delivering a single gzip-compressed Javascript file, with stylesheets and images embedded within. That’s possible today, and it’s effectively identical to the proposed zip solution, with the exception that it fails if Javascript is off.

Comment by eyelidlessness — November 18, 2009

As far as I know HTTP1.1 supports GZIP natively for any request:

So the server can transparently reduce the size of files without even asking for it. Of course only for the browsers that support it (which all I know do). Aside of that: I understand that multiple requests do in fact end up in http connections to be established which simply takes long. But having a solution where every developer has to take care with is no fun. I would be more another solution: How about adding a webserver featuere (evaluatable through the first HTTPResponse) that tells the browser: “Hey! I offer merged download.”: Meaning having apis in a shape like:
http:///merge/rawText?fileA=style.css&fileB=funny.css&fileC=moody.css (of course sent by post :-p)
or/and a image services that return one big jpeg file and a file area descriptor for the various slices the browser has to cut.

This way the browser writers could start off as many requests as they want/are necessary: You could still keep things like Caching per file seperate with a special-combined HTTPResponse Header.

The difference would be: We could write in as clean as we need: No certain work has to be done while development. This also means that knowledge about compressing is not required for creating simple html web content (if its just a apache module).

What do you think?

Comment by martinheidegger — November 19, 2009

Leave a comment

You must be logged in to post a comment.