[CC-A by Anomalous4]

Michael Mahemoff has posted an extremely in-depth tutorial on HTML5Rocks on the subject of offline web apps:

• Introduction: The Meaning of “Offline”
• Application Cache and Offline Storage
• Older Offline Storage Techniques
  • Cookies
  • Plugin Based Storage
  • Browser-specific features
• Offline Storage in the Era of HTML5
  • Web Storage
  • Web SQL Database
  • IndexedDB
  • File API
• How to Use the Offline Technologies
  • Storing miscellaneous data
  • Storing binary data
  • Ensuring the Application Runs Smoothly
  • Handling simultaneous data opertations
  • Storing large amounts of data
  • Detecting if you’re online
  • Structuring file data
  • Ensuring cross-browser compatibility
  • Protecting against lost data
  • Securing offline data
• Summary
• Further Reading

(BTW, don’t hate me ‘cuse of the LOLCat pic — you know you secretly like it)

This is a big deal. It uses Gears of course, and many people are always saying “it’s about bloody time.”

This is easy to say from the comfort of the arm chair at home. In reality, an architectural change like this is huge. Moving to a sync model that works reliably is tough in the best of times, and when you try to do it to a product that is popular, it gets that much harder to do right.

I got to use this when I was working at Google and saw numerous ups and downs (one version works well, then it gets buggy, then it gets better, etc). It was fantastic to be able to use it on the plane (or would have been if I wasn’t scared that someone would look over my shoulder). It resets expectations. You get used to opening up the browser when you KNOW you are offline. Having this in a product that you use as much as email will change user behaviour and will lead us down the path of more easily being able to do browser like things when connectivity is rough. Hell, knowing that WiFi jumping up and down won’t affect me is a big deal.

Congrats to Andy Palay and the team. I hope that we get to see an interview him as he discusses the architecture and the various tweaks to that architecture as they nailed this all down. We have much that we can learn from the process.

If you don’t have it in your “Labs” settings (I don’t right now, and the witing game is annoying I know!) check out the high level video:

Kris Zyp has created OfflineRest, a new module in Dojo 1.2 that allows for a simple offline pattern to building your application.

Dojo 1.2’s new dojox.rpc.OfflineRest module automates the local storage of data and synchronization by leveraging the Dojo Data and REST abstractions. The OfflineRest module augments the JsonRest service in Dojo such that requests are cached in local storage for offline access, and modification requests (put, post, and delete) modify the cache and are recorded for delivery to the server; immediately if online, otherwise when connectivity is restored. Furthermore, JsonRest is the core engine used by JsonRestStore. Consequently, you can simply use the standard Dojo Data API with the JsonRestStore and effortlessly add offline capability with no modifications to your data interaction code. The underlying Rest service automates the handling of caching, storing data locally, and syncing changes. In addition the new OfflineRest module has no dependency on plugins, but rather progressively utilizes offline features that are available, while still operating properly on legacy browsers without offline support.

He has a demo that involves a CRUD system for hiking trails which interestingly asked me for Gears permission, even though it appears that OfflineRest doesn't abstract on top of both Gears and the emerging storage standard, which is something I would like to see, to extend the reach.

This was just the first step in their offline support, and now they have announced the next step. This new version allows you to edit on the road with a synchronization system that keeps you sane:

Working on the Gears team we also run across applications that we would love to take offline. A lot of these applications aren't Google's so we thought it would be nice to be able to take third party apps offline. This also makes sense since Aaron Boodman (Mr. Greasemonkey) is co-tech lead on Gears itself!

Ben Lisbakken has written up his work taking Gears and Greasemonkey to make this happen. He details the real example of taking Wikipedia offline which has once piece of narly code to do with iframe injection to be able to store data on third party sites (e.g. media site vs. wikipedia main site):

1. Initialize Gears on page
   • Check if site has been allowed, if not, trigger allow dialog
2. Insert iFrame
3. Initialize Gears on iFrame
   • Check if site has been allowed, if not, trigger allow dialog
4. If Gears is initialized on both, insert Cache Page link (unless page is cached)
5. When user clicks Cache Page:
   • Capture the HTML and CSS of the Main Page
   • Store the URLs of all links to HTML, CSS, and media files in the Gears database (so we can remove them from the ResourceStore later, if needed)
   • Create an iFrame whose src is in the domain of upload.wikimedia.org. Pass all media file URLs to the iFrame in the src URL after the hash, e.g. src="http://upload.wikimedia.org/#thisimgloc.jpg||anotherimgloc.jpg||lastimgloc.jpg"
   • Initialize Gears in iFrame
   • Capture all URLs from the iFrame's href hash.
6. When user clicks [x] to remove an article from cache:
   • Grab all URLs from the Gears database that correspond to that article
   • Remove all URLs from the ResourceStore of the Main Page that contain the string "en.wikipedia.org"
   • Remove all URLs from the Gears database that correspond to that article
   • Create an iFrame whose src is in the domain of upload.wikimedia.org. Pass all media file URLs to the iFrame in the src URL after the hash, e.g. src="http://upload.wikimedia.org/#thisimgloc.jpg||anotherimgloc.jpg||lastimgloc.jpg||remove||"
   • Initialize Gears in iFrame
   • Remove all URLs from the ResourceStore that are listed in the iFrame's href hash.

Here's the script in action as I save pages away. This is just the beginning. Ideally we would have the code automatically save content that you have been too, and do smart spidering to get more on the subject too. We will also work on making GearsMonkey scripts even easier to write.

Mindmeister has taken their mind map tool and now allow you to map offline.

The tool itself is a nice Ruby on Rails application (includes pink fade effects! yellow is so 2006) that gives you a visual canvas to play with your mind.

Their approach to offline is similar to Google Reader in that the user has to say "hey, take me offline". The interface to that is a nice little slider widget. At the point your maps are sync'd down to the local store.

I would love to see it auto sync, and I noticed a couple of issues when I actually went offline but didn't tell the tool first (would be nice to have the tool grok that) but the mind map tool in general is a nice app to use. It feels like Geni.

One new feature we added to WebRunner is web application styling or theming. You can drop a “webapp.css” file in the bundle and it will be applied to the hosted web application. You can also use platform-specific folders in the bundle to apply specific CSS based on the current platform (Windows, Mac or Linux). We’ll put more details up on the wiki when the feature is released. In the meantime, here is a screenshot of Google Reader hosted in WebRunner, running on Mac, using Jon Hicks’ excellent CSS theme.

To get started you first pick a locale, then a category, then do a search. You can also add multiple regions and categories too.

Here is a quick demonstration of the app in action:

The product has also graduated to public beta, thanks in no small part to the enthusiasm and support of all of you. Whether or not you installed San Dimas, you can now download eBay Desktop, for free, from http://desktop.ebay.com

eBay Desktop works with eBay US, and in the coming weeks and months we will be extending support to other countries. You can continue to use San Dimas to shop on any eBay site in the meantime. The response to San Dimas worldwide was incredible, and we are committed to bringing this desktop eBay experience to everyone.

The application team made substantial changes for this release to ensure a more personalized feel. Changes included:

• Revamping the home screen to dedicate more area to highlighting your eBay activity -- your watch list, bidding list, recent items, recent categories, and favorite searches.
• A new browse feature
• Favorite search feeds allow you to automatically pull down items that match a favorite search
• Ability to run in the background even when the application itself is closed, so that you can get reminders or outbid alerts at any time.
• Performance improvements
• Numerous bug fixes

eBay Desktop is a free download and all you need to use it is a valid eBay account. You can download eBay Desktop from http://desktop.ebay.com

He ended up with a new library called Vortex, a 10k JavaScript file that features:

• Ability to automatically detect referenced resources such as images, scripts, and stylesheets (including nested @imported stylesheets)
• Automatic detection of network state with corresponding UI/feature changes (also implements .isOnline() and .isOffline() methods)
• Auto syncing of events. it essentially records any user-defined actions and automatically plays them back when the network comes back online
• A generic storage provider to quickly and easily save javascript objects (JSON-encoded)
• Graceful fallback if no offline support is available (google gears not installed or not allowed to run)
• Library-independent (does not require dojo, prototype, Ext, YUI, etc) and cross-browser compatible

He has a simple RSS demo that shows the library at work.

I took it for a spin and went online and offline to see how it automatically detects. Also in the video below I take a peek at the code and we see what the definition of "online" means (able to XHR a particular file).

WebRunner is a simple XULRunner based browser that hosts web applications without the normal web browser user interface.

WebRunner 0.7 was just released with improved configuration capabilities and file system organization. New webapp bundles have also been added for Facebook and Twitter to compliment the existing profiles for:

• Gmail
• Goggle Calendar
• Google Docs
• Google Groups
• Google Analytics

WebRunner and the webapp bundles are available for Windows, Linux and Mac.

He has seen that there is a common case, which is "When you build a true web application, many people do so via the single page model". If you take this assumption, wouldn't it be nice to tag the top page as an application, and have the browser subsystem take care of caching files and using them correctly when offline. His assumption is based on the logic that: "If you have complicated server logic you use a multiple page application, which isn't going to do much in an offline mode anyway, as it has complicated server logic!".

Some people do not subscribe to this view, and think that there is a lot of multiple page applications that can be made to be web enabled, and so his latest thoughts take that into consideration.

The idea is that you can then <html application="manifest-of-urls.txt"> and use a different path. He starts with:

Ok, new proposal:

There's a concept of an application cache. An application cache is a group
of resources, the group being identified by a URI (which typically happens
to resolve to a manifest). Resources in a cache are either top-level or
not; top-level resources are those that are HTML or XML and when parsed
with scripting disabled have with the value of
the attribute pointing to the same URI as identifies the cache.

When you visit a page you first check to see if you have that page in a
cache as a known top-level page.

If you do, skip the next two paragraphs; the 'new cache' flag is set to
false.

.... and the logic keeps going ....

Seeing simpler and simpler offline support is great.

At Sybase iAnywhere we provide a data sync infrastructure for mobile and remote apps. The Background Sync architecture mentioned on the Gears site is a lot like mobile application architectures we see all the time. As a result, it seems as though the same mobile sync technology can be used to solve the offline sync problem for Google Gears.

This prototype uses a local lightweight database (based on the iAnywhere UltraLite database) with built-in synchronization
capabilities, meaning:

• Built in change tracking, so that changes (including deletes and updates) can be sent to the server,
• Built in state tracking, so that the local database knows when synchronizations succeed and can take appropriate clean-up operations automatically,
• Built in synchronization over TCP/IP or HTTP.
• Optional encryption of the data store and of the sync stream
• Somewhat stricter data management model than SQLite.

The other piece is the sync server that manages synchronization to Oracle, SQL Server, IBM DB2 and Sybase databases. The sync technology is fully transactional, meaning data does not get partially changed at either end. It also has automatic mechanisms to detect conflicts and hooks that allow users to programmatically handle the conflicts using SQL, .NET, or Java.

As far as scalability, these technologies are currently being used in projects that involve nearly half a million mobile devices.

Eric also recorded some videos show this all in action:

Google Gears Prototype with Auto Change Tracking and Syncing (3 min 13 sec)

Prototype of Google Gears Database Syncing with Oracle (3 min 16 sec):

Omar Kilani of Remember The Milk took the time to write up his teams experience in Taking web applications offline with Google Gears.

The article moves past an introduction to delve into the design decisions around an offline-capable architecture, and user messaging and presentation of state. We learn why Omar decided to go with the explicit offline mode, and then the five steps to offline conversion:

• Ensuring resources are available offline
• Decoupling the application from the network
• Persisting data on the client
• Re-creating application state from persisted data
• Developing a synchronization strategy.

There is a lot to learn here.

From their architecture considerations:

RTM was designed as a client-side application from inception. The server-side portion of RTM is mainly used as a "dumb" data store, and the application periodically synchronizes with the server. In this case, using Gears to provide offline access was a natural fit, and was relatively quick to implement as we had some prior experience with data synchronization protocols.

There were some features of the online experience that could not be carried over to the offline mode. One of these was the Google Maps integration, in which users can geolocate their tasks and quickly visualize where their tasks are occurring in the real world. As Google Maps requires access to Google servers to fetch map tiles and data, and such a data set is quite large and thus hard to cache, this functionality is disabled once the user enters offline mode.

The undo feature of RTM is also unavailable in the offline version as this is a complex server-side operation (due to the multi-user nature of RTM and the ability to share tasks and lists). Instead, the user is presented with a dialog box asking for confirmation if they execute a destructive action such as delete. Undo functionality in offline mode is on the RTM roadmap, however.

To the decisions they made on showing the user information on whether they are online or offline:



A fundamental design decision is whether to implement offline support as "modal" or "modeless." Choosing which style to implement will, in most cases, be dictated by the type of data the application works with and how much of that data will be available offline. One style is not necessarily superior to the other, and, for example, it's much easier to implement a modeless style for tasks (in RTM) than it is for feed items (in Google Reader) based on the size of data items and the total data set alone.

Finally, we learn some tips and caveats from the RTM Gears implementation, including dealing with the LocalServer, the different types of offline, defensive coding, debugging, and coding with upgrades in mind.

Omar Kilani wraps it up in his conclusion:

By now, you should be itching to add offline support to your web application (we hope!). If you should take anything away from this article, it's that taking your application offline isn't as hard or complex as it may first seem, and that Gears is a joy to work with (and it'll become even easier and more fun as the project matures and is used by more applications).

As for us at RTM, we couldn't be happier with Gears. The speed at which we were able to provide offline functionality (four days from reading the documentation to a launchable implementation) is a testament to the quality, ease of use, and production-readiness of Gears. Many thanks to the Google Gears engineers for their foresight and for making this an open source project to which members of the Internet community can contribute.

Thanks to the Remember The Milk team for taking their application offline in record speed, and for taking the time to share their experience.

Features

An offline widget that you can easily embed in your web page with just a few lines of code, automatically providing the user with network feedback, sync messages, offline instructions, and more.

A sync framework to help you store actions done while offline and sync them with a server once back on the network.

Automatic network and application-availability detection to determine when your application is on- or off-line so that you
can take appropriate action.

A slurp() method that automatically scans the page and figures out all the resources that you need offline, including images, stylesheets, scripts, etc.; this is much easier than having to manually maintain which resources should be available offline, especially during development.

Dojo Storage, an easy to use hashtable abstraction for storing offline data for when you don't need the heaviness of Google Gear's SQL abstraction; under the covers Dojo Storage saves its data into Google Gears.

Dojo SQL, an easy to use SQL layer that executes SQL statements and returns them as ordinary JavaScript objects.

New ENCRYPT() and DECRYPT() SQL keywords that you can mix in when using Dojo SQL, to get transparent cryptography for columns of data. Cryptography is done on a Google Worker Pool thread, so that the browser UI is responsive.

Integration with the rest of Dojo, such as the Dojo Event system.

To get started

Visit the Dojo Offline home page; read the new tutorial titled "Creating Offline Web Applications With Dojo Offline"; download
the new Dojo Offline 0.9 beta SDK; and play with the demos.

As part of the experience Jack has created Ext.air.MainWindow which makes your AIR window look like a Ext.Window.

Jack actually created two versions, the online one actually uses Google Gears, and he abstracted the database access via Ext.data.SqlDB.

They decided to take an approach similar to Google Reader's offline model:

What can I do offline?

Just about everything that's possible online with Remember The Milk now works offline too -- not only can you access your lists, but you can add new tasks and notes, edit existing tasks (complete, postpone, prioritise, tag, and change due dates to your heart's content), use your own personal tasks search engine, create new Smart Lists, and more.

We asked Omar Kilani of the team a couple of questions about the offline release / experience:

Why did you take the step of asking the user to go offline (as reader does) versus just taking the todo list data offline in the background?

The app state is always in sync, and everything is updated in the
background (data is stored as in-memory JS objects, and sqlite is used
as a backing store in a write-through fashion). So you can pull the
plug, lose your Internet connection, etc, and the app will notice and
switch you to offline mode.

Right now, we take a sort of hybrid modal/modeless approach --
regardless of whether the app syncs in the background, we also thought
it was important to allow the user to control the mode they're in and
request a "sync".

I think an analogy to this is the 'Get Mail' button in email clients --
most mail clients check for new mail in the background, and if you're
using IMAP, you get notified immediately of new mail, but there's some
security in seeing that button there, and knowing you can press it. :)

I think we need to improve the UI aspects of this, and we're going to
look into doing that very soon. :)

How was the experience using Gears for offline?

Gears is awesome, and I believe it is the future of offline web
applications. :)

I personally feel it gives you all the right building blocks to do
whatever needs to be done to take an app offline.

What would you like to see the framework give you?

1. Progress indication from LocalServer! (This would be very helpful in
   communicating with the user for the situation described above with
   initial manifest download
2. A more stable WorkerPool. (It's currently pretty easy to cause a
   browser crash with it.
3. Off the top of my head, how about bundle support in LocalServer?
   Perhaps it can go download a Zip/JAR with all your files in it, and have
   the manifest entries point into that. That would be pretty sweet.

Brad Neuberg and the SitePen team has been working tirelessly to get this beta out of the door. Now we have the Dojo Offline Toolkit in beta, released to the world:

Dojo Offline is a free, open source toolkit that makes it easy for web applications to work offline. It consists of two pieces: a JavaScript library bundled with your web page and a small (~300K) cross-platform, cross-browser download that helps to cache your web application's user-interface for use offline.

You can try out the Moxie demo, view source on the Hello World example, or go ahead and download the SDK.

Building offline apps isn't simple. It isn't a case of magically waving a "make my app work offline" wand. Developers need to re-think their architecture most of the time, and think about what it means to be offline. What can work. What can't. How do I sync when I go back online. All tough questions for many applications. DOT gives us a nice playground to try out some of these ideas.

I hope that offline matters less and less over time, but for me connectivity can be very spotty indeed. It isn't just when you are on a plane. EVDO and friends often suck. Wireless often sucks. Hell, if they can't fix it so my calls don't drop all the time then how can I expect decent net access from anywhere? And all of this is from someone living in the bay area, let alone Lonely, Wyoming, or somewhere in the third world (yes they have internet too!)

The folks at Cerulean Studios (makers of Trillian IM) have blogged about their offline IM client that is Flash based, but can jump offline.

The current focus of the technology is to rip application-quality web software *out* of the browser and *on* to the desktop. It is (currently) technically impossible (for security reasons, at the very least) to magically invoke a real application on your desktop without a download of any sorts; the current demo you’re watching is using a small Firefox plugin to do its dirty work. The actual IM product is still the same web-based Flash that you’re invoking from your web browser. In a way, you can just think of this as a mini web browser (very mini :) ).