Comments on: Browser storage: What is the correct API? SQL? JSON?

By: aljames

aljames — Fri, 17 Apr 2009 07:16:34 +0000

@kriszyp Agreed that indexes are important (although remember that we are talking about *tiny* databases here), but pre-indexed O(log n) access is not the sole domain of SQL!
For instance, whats the difference between “SELECT * FROM rows WHERE id>5″ and “function(row){return (row.id>5) ? row : null; }”?

Both could be computed with a index on id, but instead of the query optimizer optimizing SQL, it would optimize javascript instead. So here we have the same as SQL, i.e. a small subset of possible queries are solvable in O(log n) using indexes, others require a table scan.

In an ideal world, the indexing would be carried out behind the scenes, like on google app engine developer kit. I.e. if you execute a query that could be solved on an index on field A so many times, it generates it so only the first access is slower.

By: mnx2012

mnx2012 — Thu, 16 Apr 2009 03:27:57 +0000

I used JSON and implement query syntax like SQL:

create data: http://vnjs.net/?id=1000000069
insert: http://vnjs.net/?id=1000000077
update: http://vnjs.net/?id=1000000079
delete: http://vnjs.net/?id=1000000075
select: http://vnjs.net/?id=1000000078
……
and select HTML element from DOM:
http://vnjs.net/?id=1000000082

end link to download my framework: http://vnjs.net/www/src/kombai.rar

By: kriszyp

kriszyp — Wed, 15 Apr 2009 13:15:30 +0000

@aljames: Map/reduce is intuitive, but it doesn’t help with the general use cases of a database, it does nothing to provide the workhorse parameterized queries with O(log n) access to data, that is the cornerstone of database querying. It is easy and does a great job of distributed processing for tasks that require analyzing large sets of data, but it works in O(n), so it will never compare to pre-indexed binary tree access for the majority of database query operations.

By: aljames

aljames — Wed, 15 Apr 2009 07:59:19 +0000

@kriszyp I disagree entirely. Map/reduce at its core is just a function that returns a value (or not) for a given object in the set. This is actually a very intuitive way of working with persistent storage in object based languages (particularly loosely typed ones). It has many advantages (the mot significant of which is that you dont need to learn SQL and whatever language you are using). I am not talking about the complexity of parallel searching that is necessary on a large scale cluster.

By: BH23

BH23 — Wed, 15 Apr 2009 06:22:00 +0000

I have been working on the nextdb.net project, which is a hosted relational database with a javascript API that allows you to update, delete, insert and execute queries that return pure JSON. The server returns pure JSON that is structured within the very definition of the queries that are defined and securely compiled on the server. SQL is a great query language but it was not designed for the web. Nextdb.net has a query language that addresses the security issues making it safe and easy for javascript programming by allowing the user to sculpt the object structure of the JSON response within the query language itself, the query language is called NextQuery. No ORM mismatch, no synchronization issues. The project is still in alpha but getting ready for beta soon.

By: GingerNinja

GingerNinja — Tue, 14 Apr 2009 22:59:05 +0000

@genericallyloud I agree with you mate, CouchDB could certainly be the way to go. The query interface is already JSON, and it’s document-centric nature means that it essentially avoids the Object Oriented-Relational mismatch.

It’s kinda frustrating that after so many years of work and technological advances, the best thing we’ve really come up with to address this mismatch is ORM. Part of me always resents having to fiddle with my domain model, even in small ways, just to satisfy the persistence strategy I’ve chosen.

By: thnkfstr

thnkfstr — Tue, 14 Apr 2009 22:21:27 +0000

@mpcm

nice pratchett ref!

By: dimitryz

dimitryz — Tue, 14 Apr 2009 22:13:53 +0000

Dion, thanks for the post.

I must say that having SQL on the client is a good idea. Once we have JS on the server become a norm, the only concern will become identifying what part of the logic is private (server) and what is public (server/client). As for Relational to Object models, they both have their strengths and weaknesses and there are tools to bridge the two. I wouldn’t want to lose the performance and legacy of SQL until a mature alternative is presented.

By: kriszyp

kriszyp — Tue, 14 Apr 2009 20:00:49 +0000

@WebReflection: I don’t have any issue with a low-level API, but SQL isn’t low-level, it only becomes so when you are forced to rebuild the stack on top of it. A low-level API that provided direct access to tables and indexes through iterators and binary search would be fine. Libraries could then implement query languages on top of it (SQL or JSONQuery). However, this isn’t entirely about SQL, it’s about using a relational DBs to store data from an environment that is very object-oriented. If you look at all the functionality that ORMs provide, all of that is basically doing translation where data paradigms are mismatched. Why build a mismatched storage system when the data structure of the language is known (the JS object model)?

By: WebReflection

WebReflection — Tue, 14 Apr 2009 18:08:41 +0000

@kriszyp, my point is that I prefer “low level” layers rather than high levels with inevitable lacks of this or that feature. What you would like to have is possible via intermediates layers and thanks to recent JSON.parse while having only JSON means our “fantasy” will be blocked because of a missed layer, the direct db manipulation via SQL language.
Why are you guys that scared from a client db engine? Possibilities are numerous and hacks problem will be the same with whaever JSON layer we put in the middle. I do not even get the discussion above … SQLite is SQLite, it is not new, it is fast, it is reliable, it is transactional, full-text, etc etc … I wonder why you do not use JSON in the server side as well, since I cannot figure out any difference between these two environment, only difficulties to sync them because of people whose do not want “to learn” SQL … again, where is the problem? It is like getter/setter discussion, useless, imo.

By: ilazarte

ilazarte — Tue, 14 Apr 2009 17:53:57 +0000

what amazes me is xsl didn’t get the mass adoption it deserves. it’s xml oriented and transformation oriented which is that 90%.

By: mpcm

mpcm — Tue, 14 Apr 2009 16:37:18 +0000

JSON, like turtles, all the way down. Doesn’t have to be that under the covers, but that is what the interface should be.

By: kriszyp

kriszyp — Tue, 14 Apr 2009 15:37:39 +0000

@WebReflection: The point is that querying, transaction, and full-text search is not dependent on SQL, extensive DB capabilities do not require SQL to be the query language. Relational DBs/SQL are a poor fit for storage within the realm of persisting JS data. Relational DBs within JS would almost always end up at ORM, which basically means you have lost, the DB didn’t fit the data, it requires translation. On other hand, we have the opportunity to use a query language and storage engines that are actually designed for JavaScript, not just carried over from archaic traditional design.

By: WebReflection

WebReflection — Tue, 14 Apr 2009 15:29:29 +0000

@kriszyp, is it better to have the possibility to implement a JSON layer over a database engine with great SQL support, transactions, and full-text searches, or have only the JSON layer without natural database development possibility?
I think we should have the possibility to use SQLite, the rest will be part of this or that library. What I mean, who cares about libraries users? Most of us do not even deeply know JavaScript or DOM, just a lib API … SQLite is the right way, imo

By: kriszyp

kriszyp — Tue, 14 Apr 2009 14:54:44 +0000

@Joeri: Believe it or not it is possible to build indices and not use SQL (I know it is difficult to imagine efficient filtering can be done without a specific query syntax :P ). We’ll have a post coming out soon showing how Persevere (which uses JSON-style storage with JSONQuery query language) can actually outperform the typical MySQL setup.

@aljames: Map/Reduce is a cool technology for a specific set of problems, but it is a very poor fit for the broad general use cases of the web.

By: Joeri

Joeri — Tue, 14 Apr 2009 14:16:31 +0000

I don’t see how you get decent filtering performance on anything but a SQL engine. I want live filtering, with the same abilities as where clauses (filter on dates, count(), …). Otherwise a client-side database just is not useful for the sort of work I would use it for.

By: DomDerrien

DomDerrien — Tue, 14 Apr 2009 13:57:13 +0000

With JSON in the browser and JSON on the wire, JSON in the database is an obvious requirement! Up to now, my REST APIs mostly converting DTOs from Java/Python/etc. to JSON and vice-versa, after the translation between SQL/GQL/etc. and programming languages. So much time and CPU wasted :( Will JSON be the final format? I hope not because I would like to start carrying enhanced messages, a-la RDF with triples for example...

By: genericallyloud

genericallyloud — Tue, 14 Apr 2009 13:00:55 +0000

I think something like CouchDB would be interesting to look at as a model. It already does synchronization well and uses JavaScript as a query/view language. Its document oriented as opposed to relational or OO, but its an interesting thought. However, the practical side of me says that this kind of thing is way too new and that despite difficulties, people would rather use what they know - sql.

By: WebReflection

WebReflection — Tue, 14 Apr 2009 12:43:16 +0000

SQLite and its features “forever”. I use this little monster since version 2 as portable desktop applications engine and website database as well (faster than MySQL in many SELECT cases).
With JSON in a couple of months we all will reinvent the wheel creating libraries to search words in saved strings, order data, etc etc.
If you need JSON, use JSON, if you need a database, SQLite is the best choice ever, IMO. Both things are completely different and I hope nobody will consider seriously to remove SQLite as storage engine from browsers.

By: andysky

andysky — Tue, 14 Apr 2009 12:11:54 +0000

JSON is clearly the way to go.