Comments on: YQL execute now allows you to convert scraped data with server side JavaScript

By: sh1mmer

sh1mmer — Fri, 01 May 2009 06:30:55 +0000

I looked at getting Sizzle running to do the CSS selectors before we launched YQL Execute and in order to use it you need a DOM. In order to get a DOM in Rhino you need env.js which currently runs to about 8k lines of code.

This means in order to get Sizzle working you need about 9k lines of JS. CSS2XPath currently weighs in at under 100 lines of code. XPath is natively implemented in Rhino and doesn’t require any additional code.

So, from the perspective of speed it’s 9k lines of interpretation vs 100 lines, and from the perspective of the execution cycle limits YQL has you can spend them on processing data, not creating a DOM.

By: postream

postream — Fri, 01 May 2009 00:38:25 +0000

unfortunatly you can’t safely scrape everything on the web, because there are some conversion quirks

the main problem is that YQL return well-formed XML, but the web is often a mess of both HTML and XHTML (also notice you can only scrape what’s inside the body tag)

look at this sample page I made (valid HTML 4): http://www.playquery.it/sandbox/yql/test3.html

this is how YQL parse it

some convertion errors:

- some HTML entities are converted to the corresponding character code (nbsp and reg), and some other not (amp,lt,gt)

- an anchor with a name=”top” now has also an id=”top”

- the textarea has some whitespaces inside, but in the YQL result is empty

- the table really freaks out (some p tags added, the form on the bottom of the page is put inside a td tag, the table is moved under the main paragraph)

and, as I noted on James blog some days ago (http://james.padolsey.com/javascript/using-yql-with-jsonp/), if you are forced to use the JSONP format instead of the XML is even worse

but, anyway, if you know very well the source of your query, and it’s XHTML well-formed, I think YQL could be really awesome

By: infosage

infosage — Thu, 30 Apr 2009 20:47:07 +0000

@Nosredna

Three words:
YQL honors robots.txt

By: WebReflection

WebReflection — Thu, 30 Apr 2009 18:34:45 +0000

JonathanT, I partially agree about a better version but I do not get the “should be part of datatables.org” part … I mean what’s wrong with my or James website/project? I better see a specific one out of whatever box … what do you think about?

By: JonathanT

JonathanT — Thu, 30 Apr 2009 18:20:00 +0000

All,

One of the main reasons we made use of James’ CSS/xpath converter to show how easy it was to plug in useful JS functions and libraries into a table, to get new functionality that people want in YQL.

Why not create a better CSS selector open data table and submit it to github for others to use and share? The sample ones aren’t part of the community respository (datatables.org) so that seems a good place for a better version to go.

Jonathan

By: JimmyP22

JimmyP22 — Thu, 30 Apr 2009 16:49:56 +0000

@Chris, Awesome work! I recommend using WebReflection’s converter though; as mentioned it’s more complete (and less buggy) than mine.

By: WebReflection

WebReflection — Thu, 30 Apr 2009 15:48:58 +0000

yep, tested right now and James Padolsey function is both incomplete and buggy (with results as well) … James, give me a shout if you read me.

By: Nosredna

Nosredna — Thu, 30 Apr 2009 15:42:31 +0000

Thanks for the answer Chris. Agreed that it’s always been possible to scrape. It’s the ease of doing it and the indirection through Yahoo! servers that I was thinking of.

The caching is nice.

By: WebReflection

WebReflection — Thu, 30 Apr 2009 15:37:29 +0000

I did not know James Padolsey function but it seems quite incomplete compared with the one I created for vice-versa. Here the specific function via experiments and document.query.css2xpath function Maybe me and James could collaborate to create a complete and stable function (mine at least pass every CSS selector used in SlickSpeed test ;) )

By: Chris Heilmann

Chris Heilmann — Thu, 30 Apr 2009 15:30:45 +0000

@Nosredna YQL access is limited to a cap that would prevent that:

YQL has the following API usage restrictions:
Per application limit (identified by your Access Key):
* 100,000 calls per day.
Per IP limits:
* /v1/public/* 1000 calls per hour
* /v1/yql/* 10000 calls per hour
All rates are subject to change. In addition, you may also be subject to the underlying rate limits of other Yahoo and 3rd party web services.

However, what prevents me to curl his page every second? I don’t need YQL to scrape people’s pages. What YQL does though is cache the results which actually means less hits for the scraped page.

By: Nosredna

Nosredna — Thu, 30 Apr 2009 15:16:46 +0000

So I can use Yahoo!’s servers to screen scrape anything I want?

That’s terrific. But what prevents abuse, such as huge attacks on some poor guy’s $5 a month data-limited hosted account?