Friday, May 29th, 2009
Category: Database
, Google
Robert Kroeger has released a nice library for local database access. The Web Storage Portability Layer nicely abstracts on top of HTML5 and Gears for database access.
The WSPL consists of a collection of classes that provide asynchronous transactional access to both Gears and HTML5 databases and can be found on Project Hosting on Google Code.
There are five basic classes:
google.wspl.Statement - A parametrizable SQL statement class
google.wspl.Transaction - Used to execute one or more Statements with ACID properties
google.wspl.ResultSet - Arrays of JavaScript hash objects, where the hash key is the table column name
google.wspl.Database - A connection to the backing database, also provides transaction support
google.wspl.DatabaseFactory - Creates the appropriate HTML5 or Gears database implementation
Also included in the distribution is a simple note-taking application with a persistent database cache built using the WSPL library. This application (along with Gmail mobile for iPhone and Android-powered devices) is an example of the cache pattern for building offline web applications. In the cache pattern, we insert a browser-local cache into the web application to break the synchronous link between user actions in the browser and server-generated responses. Instead, as shown below, we have two data flows. First, entirely local to the device, contents flow from the cache to the UI while changes made by the user update the cache. In the second flow, the cache asynchronously forwards user changes to the web server and receives updates in response.
By using this architectural pattern, a web application can made tolerant of a flaky (or even absent) network connection!
You can of course take a peak at the code to see how it works, for example:
JAVASCRIPT:
-
-
google.wspl.DatabaseFactory.createDatabase = function(dbName, dbworkerUrl) {
-
var dbms;
-
if (window.openDatabase) {
-
// We have HTML5 functionality.
-
dbms = new google.wspl.html5.Database(dbName);
-
} else {
-
// Try to use Google Gears.
-
var gearsDb = goog.gears.getFactory().create('beta.database');
-
var wp = goog.gears.getFactory().create('beta.workerpool');
-
-
// Note that Gears will not allow file based URLs when creating a worker.
-
dbms = new wireless.db.gears.Database();
-
dbms.openDatabase('', dbName, gearsDb);
-
wp.onmessage = google.bind(dbms.onMessage_, dbms);
-
-
// Comment this line out to use the synchronous database.
-
dbms.startWorker(wp, dbworkerUrl, 0);
-
}
-
return dbms;
-
};
-
Nicely done. It would be great to see a version that acts as a shim and when in Gears mode manually implements the HTML5 standard API so you can write your user code to that and not a new google package.
Tuesday, April 21st, 2009
Category: Database, JSON, JavaScript
Kris Zyp recently posted about an intriguing new chapter in the application persistence space:
The latest beta of Persevere features a new native object storage engine called JavaScriptDB that provides high-end scalability and performance. Persevere now outperforms the common PHP and MySQL combination for accessing data via HTTP by about 40% and outperforms CouchDB by 249%. The new storage engine is designed and optimized specifically for persisting JavaScript and JSON data with dynamic object structures. It is also built for extreme scalability, with support for up to 9,000 petabytes of JSON/JS data in addition to any binary data.
This comparison isn't exactly apples-to-apples as it turns out–for the web app use case, Perservere has a bunch of value-adds on top of data storage:
Persevere/JavaScriptDB goes further [than relational DBs] with the flexibility to evolve schemas and handle partial schemas. Persevere also provides integrated server side JavaScript (SSJS) with persistence, Comet-driven data change notifications, JSONQuery, standards based HTTP interface with content negotiation, JSON-RPC interface to SSJS, cross-domain handling, CSRF protection, and more. All of these things are additional features that one would have to add to the stack for other storage systems, making them even slower. Persevere includes this functionality out of the box, while still maintaining extremely fast performance.
Kris spends a bit of time in his post explaining his test setup, but then gets to the good stuff:
So how does Persevere achieve this level of performance with the JavaScriptDB storage? The dynamic object-oriented nature of the data that is stored in JavaScriptDB is much different than that of a traditional relational database, so a number of innovative approaches were employed.
He goes into quite a bit of detail explaining the implementation details behind JavaScriptDB. The summary (with lightly edited quotes) is:
- Direct Data-Bound Object Representation: "In a traditional application stack, a record must have separate in-memory representations for [the database] and [the application] result set which then might be mapped to an object representation. With JavaScriptDB, the single in-memory object is reused for all result sets and data caching."
- Shared Cache of Objects with Copy-on-Write
- Append-based Database Storage: "Many traditional database commit data to a transaction log before committing data to the table, requiring multiple writes. JavaScriptDB appends transactional data directly to the main storage file; writes can be committed with a single IO operation."
- Adaptive On-Demand Concurrent Indexing
- Batched writes in integrity mode
- Pluggable Storage
Check out the full post all the details.
Category: Database, JSON, JavaScript, Library
Atul Varma, a fantastic colleague in Building "S" at Mozilla, has been playing with a JavaScript implementation of CouchDB called BrowserCouch:
BrowserCouch is an attempt at an in-browser MapReduce implementation. It's written entirely in JavaScript and intended to work on all browsers, gracefully upgrading when support for better efficiency or feature set is detected.
Not coincidentally, this library is intended to mimic the functionality of CouchDB on the client-side, and may even support integration with CouchDB in the future.
Why?
This prototype is intended as a response to Vladimir Vuki?evi?'s blog post entitled HTML5 Web Storage and SQL. A CouchDB-like API seems like a nice solution to persistent storage on the Web because so many of its semantics are delegated out to the JavaScript language, which makes it potentially easy to standardize. Furthermore, the MapReduce paradigm also naturally takes advantage of multiple processor cores—something that is increasingly common in today's computing devices.
There is also an interactive CouchDB project that "is an emulator written in 100% JavaScript with tons of jQuery thrown in. It also implements the collation schemes as well as the map/reduce algorithms. While it doesn’t demonstrate replication, conflict management and a host of other capabilities in CouchDB, it does strive to illustrate concepts like schema-less JSON documents, map/reduce and how these things fit together."
You can check out the emulator here.
Tuesday, February 3rd, 2009
Category: Database, HTML, Mobile, Standards, iPhone
Brad Neuberg told me about two cool additions to the iPhone that now use HTML5 features:
Safari JavaScript Database Programming
The HTML 5 specification provides a new mechanism for client-side data storage: JavaScript database support. HTML 5 is currently in development by the Web Hypertext Application Technology Working Group (WHATWG).
JavaScript database support is available in Safari 3.1 and later, and in iPhone OS 2.0 and later.
You should read this documentation if you are a web developer who wants to store data locally on a user’s computer in amounts beyond what can reasonably be stored in an HTTP cookie.
HTML5 Application Cache
Michael Nordman of Google asked "Is this stuff built into shipping Safari or iPhone browsers yet?"
David Kilzer replied "This feature shipped with iPhone OS 2.1. When you use "Add to Home Screen" from the "+" button on Safari for iPhone, a web application with a manifest defined (per the HTML5 spec) will be saved with any cached resources. Note that the manifest file *must* be served with the correct MIME type for this to work.
There is no shipping version of Safari for Mac OS X or Windows that supports this feature yet."
Fantastic to see!
Wednesday, October 1st, 2008
Category: Database, JavaScript
Uriel Katz wrote in to tell us that he's burned his GearsORM framework to the ground to create the entirely new JStORM framework, announced in his blog. JStORM currently supports Google Gears, Aptana Jaxer, and Adobe AIR, but it's still bleeding edge: no documentation yet, just code. But the feature-set is interesting:
* define your tables as models.
* full CRUD support.
* events on(Before/After)Delete/Save/Update.
* SQL LIMIT/OFFSET support.
* order by support.
* automatic creation of tables.
* selecting and deleting over relations.
* easy iterating with each syntax like in Ruby,support for query chaining and more.
* basic introspection support.
* transaction support.
* self relations.
* support multiple backends and multiple dialects,currently MySQL (in Jaxer only) and Sqlite (Jaxer, Gears and AIR).
Here's an example of loading "Person" entities from a database using JStORM:
JAVASCRIPT:
-
var Person = new JStORM.Model({
-
name:"Person",
-
fields: {
-
firstName:new JStORM.Field({type:"String",maxLength:25}),
-
lastName:new JStORM.Field({type:"String",maxLength:25}),
-
},
-
connection:"default"
-
});
-
-
Person.all().each(function(person) {
-
console.log(person.firstName);
-
});
Looking forward to seeing some docs, Uriel ;-)
Tuesday, March 11th, 2008
Category: Ajax, Database
Working client-side with data can be challenging and projects have tried to address this in different fashions. Ian Smith, creator of Joe's Goals, has his own approach. He's created Taffy DB, a lightweight Javascript database
that allows you to insert, update, delete, order, loop, and query against a client-side collection of data. The great thing about it is that it's library agnostic and can work with all of the popular frameworks.
The idea came to me as I was working on Joe’s Goals 2.0 and I realized that one of the hardest parts about building any Web 2.0 application is working with data. There is no good way to use JavaScript by itself to gather, search, and maintain a collection of data. There are lots of great ways to great interfaces now days, but what about the data behind them? Taffy DB is the result of the research and testing I did for Joe’s Goals and is now avaliable for everyone to use.
Some of the key features include:
- Under 10K!
- Simple, JavaScript Centric Syntax
- Fast
- Easy to include in any web application
- Compatible with major Ajax libraries: YUI, JQuery, Dojo, Prototype, EXT, etc
- CRUD Interface (Create, Read, Update, Delete)
- Sorting
- Looping
- Advanced Queries
The syntax to use the libary is very straightforward:
Updating:
JAVASCRIPT:
-
-
friends.update(
-
{
-
state:"CA",
-
married:"Yes"
-
},
-
{
-
name:"Joyce"
-
}
-
);
Inserting:
JAVASCRIPT:
-
-
friends.insert(
-
{name:"Brian",
-
gender:"M",
-
married:"No",
-
age:52,
-
state:"FL",
-
favorite_foods:["fruit","steak"]
-
});
In addition, Taffy DB provides 12 different conditional statements (e.g.: "greaterthan", "equal", "like") to help build advanced queries that can further filter your results.
Ian has created a Getting Started Guide which will help in getting up to speed on the full capabilities of Taffy DB.
Wednesday, March 5th, 2008
Category: Database
Christopher Lenz has been spending time on the CouchDB project. He got lulled over when CouchDB went to JSON.
He talks about how "CouchDB is pretty well positioned for storing and querying XML data in addition to JSON" via the E4X support that it gets out of the box as it uses SpiderMonkey.
JAVASCRIPT:
-
-
by_lang: function(doc) {
-
var html = new XML(doc.content);
-
map(html.@lang, {title: html.head.title.text(), …});
-
}
-
To be fair, this is already possible if you use other view servers (such as the Ruby or Python ones), where you also have access to the XML support provided by the respective standard libraries. Given CouchDB’s incremental view update model, you usually don’t care so much about the performance of view functions as you care about the data it produces. So if your view function can somehow parse the XML and put some data into the view index, that's usually all you need. Actually querying the view is going to be really fast.
But E4X is an exceptionally convenient API for XML. I think using E4X is going to be a pretty good approach for those who want to to CouchDB to store and query XML content.
Thursday, January 3rd, 2008
Category: Database
CouchDB has been getting people re-energized about DB stuff recently. Some got gung-ho about the OODBMS and that fizzled and people went back to the "ug, I guess we just do the SQL thing and be done with it... and maybe use an ORM if we really hate it".
Then the Couch came along and had us all thinking about JSON being the next "turtles all the way down".
Well, IBM was excited enough to hire Damien Katz to work full time on it:
All the code will be Apache licensed and donated to the Apache Software Foundation, with the plan CouchDB will eventually become an official Apache project. A big plus here is the Apache license allows anyone to do pretty much anything with the code, so everything remains truly open source. I wouldn't have done this without IBM's commitment to keeping CouchDB open.
It is going to be to see what us JavaScript folk do with a DB that is RESTful and JSONified.
Congratulations to IBM for a great score (and to Damien too).
NOTE: Kris Zyp joining SitePen
Also, congrats to Kris Zyp and SitePen for tying the knot. Both Kris and SitePen are top notch outfits, and a great fit.
Monday, October 22nd, 2007
Category: Browsers, Database
The WebKit team has implemented an implementation of the HTML 5 client side storage API which gives you an asynchronous API:
JAVASCRIPT:
-
-
var database = openDatabase("Database Name", "Database Version");
-
-
database.executeSql("SELECT * FROM test", function(result1) {
-
// do something with the results
-
database.executeSql("DROP TABLE test", function(result2) {
-
// do some more stuff
-
alert("My second database query finished executing!");
-
});
-
});
-
Once you have the latest nightly, you can check out their database example and even see the tool that is available in the Web inspector:
Friday, August 24th, 2007
Category: Adobe, Database
Justin Palmer has an interesting post about the dealing with the asynchronous nature of Adobe AIR’s SQLConnection class and the challenges it poses when dealing with the DBMS' response. Justin shows some creative ways of working around this through the use of Responder objects and event listeners:
var connection = new air.SQLConnection();
connection.open(air.File.applicationResourceDirectory.resolve('development.sqlite'));
var statement = new air.SQLStatement();
statement.sqlConnection = connection;
statement.text = "SELECT * FROM contacts";
statement.execute();
The code above, albeit a little long winded, is the bare necessities for executing a query. While it looks fairly straightforward on the surface, we’ve already run into our first problem. If the execute operation is asynchronous, how do we know when we can begin to retrieve and manipulate the data returned? We can do it one of two ways, register an event listener or pass a Responder object to the execute method.
From the looks of this thread it appears this is on the minds of several other folks and Adobe's working on developing a method for synchronous SQL calls is in the works.
Wednesday, February 7th, 2007
Category: Database, Framework, Java
OpenToro 4.0, an open source Ajax Web Database Publishing tool, has been released.
OpenToro is a CMS written in Java that allows the development of database driven web Applications in a quick and agile way.
If you need to build simple CRUD applications that are faces onto your database, this could be a tool for you.
Download the set of demo screencasts to learn more.
Friday, July 7th, 2006
Category: Database, Remoting
A new PHP component by Adnan Siddiqi accepts a MySQL result set and converts it into a JSON message. MySQL To JSON:
This class can be used to convert data from MySQL query results into a JavaScript expression in JavaScript Object Notation.
It takes a MySQL query result handle and retrieves the query result column names and the query result data.
The class generates the definition of a JavaScript object in JSON that contains an array of a rows of query result data.
Each array element represents an object with the properties set to the query result column names. The property values are the query results for the respective row and column.
Thursday, March 2nd, 2006
Category: Database, Editorial, Office
After four "When" companies got a chance to impress us with their Web 2.0 goodness, four "Make It Easy" concerns are now up to the plate: Dabble DB, Rallypoint, The Form Assembly, and Zoho.
Each company has eight minutes to present their wares. A panel (Michael Arrington, Rael Dornfest, and Krishna Akella in this session) and the audience give feedback for eight minutes.
Dabble DB
The co-founders are up talking to us about their Dabble DB service. It's a hosted database for the rest of us. Business users know that spreadsheets aren't the best way to store data, but they feel that databases are "scary."
They showed a demo of taking a spreadsheet of information from an O'Reilly OSCON from a few years ago. The UI of Dabble DB lets you either import data to create a spreadsheet, or manually enter the data. They cut-and-pasted and CSV export from Excel, and they're demoing a really neat ajaxian interface for the data -- spreadsheet on the web.
They're also showing ad-hoc queries against the spreadsheet interface, using a simple filter type interface both across all text or specific fields, and you can save the results of queries for later viewing.
They also have a calendar view of database data, where they parse out date-like values and plot them on a calendar -- and you can apply the same searches against the calendar view, too. It's a really fast, compelling interface -- better than some of the When calendaring apps we saw in the last session.
They are also demoing schema changes. For example, they make it super easy to take a text field and change that to a lookup table relationship -- something that traditional databases make painful, and they also make it easy to migrate other fields to the new lookup table.
They also export data to RSS, PDF, CSV, iCal, HTML, OPML, and text.
The audience was very complementary, and both Michael and Rael on the panel seemed impressed. A great product.
Rallypoint
The Rallypoint dude is talking about how hard it is to do effective collaboration in a team environment: you start passing documents around, you lose track of ersions, and before you know it, things are a big soupy mess.
Goal for Rallypoint: combine the features of a word processor with the collaboration features of a Wiki, tied together with an ajaxian user interface.
The demo of adding a new page shows off their word processor interface. It's pretty much the kind of rich text editing we've come to expect from ajax apps without any dramatic differences.
The security features allow you to protect a page, allowing only certain users and groups to view/edit/subscribe to the page.
The versioning features weren't shown, and there's no way to export data at present, though they all working on an export feature.
The general consensus from the panel was: "This is a crowded space, good luck."
The Form Assembly
The creator of The Form Assembly is up, talking about web forms, saying that many websites need web forms for various reasons (registrations, etc.) and up until now there hasn't been an easy way to get them (riiiiiight).
The product lets you build forms and show reports based on the form data. The UI was pretty nice. I didn't see anything revolutionary.
The product is free up until you get more than five responses to your form. After that you either pay as you go (12 cents per response) or you buy a $25/month subscription.
Michael Arrington: "Can I massage the data that users enter?"
Answer: There's a junk filter for removing spam.
The panel thought the pricing was just way off but otherwise seemed to appreciate the product, though not enthusiastically.
Zoho
Wow, Zoho is a suite of business applications on the web: Writer, Virtual Office, CRM, and, err, more. They're showing us Zoho Creator, which is for "creating web applications with no lines of code."
(By the way, thanks to a crash, we're able to see it was written using Java Struts. Neat.)
You can create applications either by basing it off a template, or creating a new one from scratch. So far, the application is composed of creating a web form.
The form creation is less visually impressive than the other form builders we just saw. In fact, once you actually start using the form you've created, the quality of the UI is quite primitive compared to the other Web 2.0 apps we've seen. There are some ajax features, but they look tacked on in a really cheesy fashion.
So, they're done showing features, so it looks like "applications" are just web forms and tables showing the data entered in the web form. Neat. The table viewer of the entered data isn't as impressive as Dabble DB but does have a few ajax features, like the ability to dynamically change the set of columns displayed, etc.
They showed us their on-ine spreadsheet for just a few seconds, and it looked really cool. I was tempted to think they were showing us an embedded Excel instance as the demo was in IE Windows, but I think it was a pure Ajax implementation. Geez, they also have an Outlook clone, too. So, some of the products look pretty cool but they didn't go very deep.
Their vision is to be a complete, fully functional MS Office clone on-line. They let you import Excel and Word and other formats into their products.
Michael Arrington: "You guys have a reputation of copying others and being overly aggressive in PR. You need to change that."
Answer: Some sort of muffled argument refuting the accusation.
Conclusion
There are a lot of really cool emerging database/form applications that make web-based form applications really easy to create. Dabble DB rivaled desktop database front-ends in terms of making it really easy to deal with data. Very cool!
Wednesday, January 25th, 2006
Category: Database, Showcase
We've seen in-browser SQL, a database administrator, and even a database designer. Now there's a database monitor...
AjaxMyTop shows active MySQL connections. You get a data grid showing connections, each showing ID, user, duration, and so on. Periodic Refresh ensures the connections are kept fresh, and you can easily set the refresh period in an input field. Being a true data grid, the connection list can be sorted and filtered on one or more fields. In a nice touch, the column heading morphs to include a text input when you want to type in a filter for that column. It's nice to see keyboard shortcuts here, certainly a feature every DBA will welcome.
Right now it looks to be read-only ... hopefully the team will add a "destroy connection" button real soon now!
