Monday, June 14th, 2010

JSonduit: Turn the Web into a JSON feed

Category: JSON

Chris Winberry recently built a node-htmlparser library that we posted on. Now we know why he built that library. He has released

Any data, anywhere.

JSonduit is a service that can turn practically anything on the web into a JSON feed that any website may consume. A JSON conduit, if you will.

Feeds are created by specifying one or more source URLs and a custom transform, written in JavaScript, that can manipulate the data before the feed is served.

JSonduit also provides a hosting service for web widgets so that any site can easily display JSonduit feeds on their pages. In fact, those recent/popular lists of you see below are actual widgets served by the JSonduit service; all done in a couple of lines of JavaScript (go ahead, view the page source).

To see what it is like to query the world in this manner, check out the most popular feed…. a view on hacker news:


  1. var result = [];
  3. var items = getElements(
  4.     {
  5.         class: "title"
  6.     },
  7.     data[0]
  8. );
  10. items.forEach(function(item){
  11.     var links = getElementsByTagName('a', item);
  12.     links.forEach(function(link) {
  13.         // Remove the "more" link.
  14.         if(link['attribs']['rel'] == 'nofollow') return;
  16.         result.push({'title': link['children'][0]['data'], 'link': link['attribs']['href']});
  17.     });
  18. });

This returns something like:


  1. {
  2.   "error": null,
  3.   "result": [
  4.     {
  5.       "title": "U.S. Discovers Est. $1 Trillion of Minerals in Afghanistan",
  6.       "link": ""
  7.     },
  8.     // ..

If this looks a touch familiar though… remember that the awesome YQL gives you acccess to the Web from a simple query language too.

Posted by Dion Almaer at 5:28 am

4.3 rating from 3 votes


Comments feed TrackBack URI

Sweeeeeeet. I really like the look of this. Maybe SQL-style (a la YQL) is better for querying data, but writing custom transforms in JavaScript feels a lot more flexible, and much, much more fun :)

Comment by Skilldrick — June 14, 2010

Doubt this will scale. Also, there must be a huge amount of IP law being violated here. I would not use this for any real business.

Comment by leptons — June 14, 2010

sounds a lot like what YQL does.

Comment by ChrisEsler — June 14, 2010

@leptons – It has already been exercised to 1000s req/sec per instance and the fetch queue is keyed so that 10,000 requests for an uncached feed still only result in one source URL request.

Comment by Tautologistics — June 14, 2010

Leave a comment

You must be logged in to post a comment.