Wednesday, September 10th, 2008

Using Canvas in IE via Flash

Category: Canvas

Grant Jones stepped up and has written a gues t post on implementing Canvas using Flash.

Grant starts out with the timeline of shims:

  1. Manish Jethani mentions something about the canvas tag being used for a 3D game, John Dowdell from adobe responds in Nov. 2005
  2. Manish Jethani suggests a way to do a SWF-based Canvas on Dec 1st, 2005
  3. iecanvas from Emil A Eklund released December, 2005 (uses VML)
  4. excanvas from Google released March, 2006 (uses VML)
  5. AFLAX (uses ExternalInterface) adds canvas emulation in March 2006

Grant wasn’t happy with the performance of excanvas for a simple game he was writing, so he wanted to see if Flash would be better, creating FlashCanvas:

FlashCanvas is a canvas tag to flash bridge for browsers which do not support the canvas tag like Internet Explorer. FlashCanvas is an experiment to see if any performance improvement could be realized by using flash technology. The javascript code is a fork of the ExplorerCanvas project

Unfortunately, the performance issue is tough. Is it faster?

The short version: not really.

The longer version: All drawing commands are being forwarded from Javascript -> ExternalInterface (down into the browser) -> ActionScript -> Flash Drawing commands. This is a lot of overhead for drawing code. Doing something simple like a moveTo and lineTo would result in at a very minimum 8 separate levels of indirection.

He then goes into detail on the implementation, and things that he tried to get performance up a notch. It appears that Vlad’s work with putting the Mozilla implementation into IE via ActiveX may be the best hope yet. Or, what if the IE team actually implemented the darn thing!

Posted by Dion Almaer at 7:09 am
18 Comments

+++--
3.8 rating from 19 votes

18 Comments »

Comments feed TrackBack URI

I wish you guys could stay objective. Food for thought.

Comment by ibolmo — September 10, 2008

it would take a lot of work, but you could try moving the drawing commands to the other side of the fence by building a translator or something. It could read in your canvas js code and move entire functions over the wall instead of individual commands. Then use the bridge for the rest.

Comment by genericallyloud — September 10, 2008

i think genericallyloud has the right idea. a javascript interpreter written in actionscript. hmmmm….

Comment by shadedecho — September 10, 2008

Cromwellian might post more on this (I think he implements this in his library), but what about marshalling drawing commands?

Comment by elfpoet — September 10, 2008

was going to post the exact same thing as generically loud.

Comment by ilazarte — September 10, 2008

@ibolmo:
No one is objective. Fact of life, you may as well face it at some point. Is there some undue bias that you object to in specific?

Comment by eyelidlessness — September 10, 2008

Well, if someone feels froggy, you could learn some Base64 math and directly manipulate an image data URL for IE8 (obviously IE7 and IE6 are out of the picture in this case and god knows what memeory leaks it will probably cause)

Comment by TNO — September 10, 2008

@eyelidlessness. I’m just tired of the same bickering (“… what if the IE team actually implemented the darn thing!”). Aside from that, an interesting article.

Comment by ibolmo — September 10, 2008

Wow, people remember my implementation? :) Yes, I’ve been shipping this in my charting code for IE since January. See my blog post here: http://timepedia.blogspot.com/2008/01/chronoscope-demo-in-flash-whatwg-canvas.html

I make use of several things to help speed up performance:
1) buffering up commands between beginFrame()/endFrame() and flushing them all at once to the Flash Vm

2) Multithreaded rendering. As soon as the Flash VM receives the drawing commands, it yields the CPU so the Browser’s Javascipt is not hanging around waiting, and can go on to calculate the next frame.

3) Layers. Within a Canvas, you can create independent layers which do not need to be redrawn every frame if they don’t change, and can be drawn into one another.

4) Display Lists. A given set of drawing commands make be recorded like a macro and played back. These are transmitted in the drawing buffer as just as ID, so arbitrarily long sequences (like 1000 lineTo calls) can be transmitted as a few characters.

Implementing the Canvas API in Flash is non-trivial and goes beyond just gluing CanvasRenderingContext2D Javascript calls to Flash equivalents. In many cases, there are no equivalents, or a severe mismatch in rendering semantics.

For example, the Canvas API includes Porter-Duff blending modes, Flash only includes a subset of these. Flash maintains a scenegraph, so compositing order, the effect of “clearRect”, etc have to be emulated carefully for both semantics, and to avoid memory leaks. (e.g. clearRect()/fillRect() is often used by Canvas programmers to erase the Canvas, but this won’t release all the scenegraph memory used by the Sprite/MovieClip/Graphics object)

Comment by cromwellian — September 10, 2008

“Or, what if the IE team actually implemented the darn thing!”, may not have been phrased well, but the idea behind it is rather rational.
.
Consider how little we like using simple CSS hacks for IE6 like *html. These different attempts at getting canvas to work in IE8 are much more complex hacks and often require the user to jump through hoops to get it to work, e.g., install this plugin or allow this ActiveX.
.
We already had a semi-functional hack in IE6 and IE7 with excanvas that didn’t require any additional action from the user, but that seems to be fairly broken in IE8. In IE8 compatibility mode plus quirks mode I can get some of my canvas elements to display, but not others, while at the same time breaking other aspects of my application due to being in quirks mode. The Javascript library I use for instance requires a doctype.
.
In any case, any hack we use for IE8 we will be stuck with for another 10 years, the life-cycle of a Microsoft browser. So I will have one conditional in my headers for IE6 and IE7 and another one for IE8, plus all the other nonsense to tell the Microsoft browser what modes to render the page in. I fooled around with the modes in IE8 beta 2 a little last night. I would actually need to be able to set different combinations of browser modes and document modes for different parts of my webpage to get most of the page working (which is not possible), and some parts of the page won’t display properly with any of the combinations of modes.
.
There are like 8 different combinations of modes that you can have. Talk about making things complicated, and yet, with all this complication, what worked in IE7 still won’t in IE8.
.
– Browser Mode: IE7, Document Mode: Quirks Mode
– Browser Mode: IE7, Document Mode: IE7 Standards Mode
– Browser Mode: IE8, Document Mode: Quirks Mode
– Browser Mode: IE8, Document Mode: IE7 Standards Mode
– Browser Mode: IE8, Document Mode: IE8 Standards Mode
– Browser Mode: IE8 Compatibility View, Document Mode: Quirks Mode
– Browser Mode: IE8 Compatibility View, Document Mode: IE7 Standards Mode
– Browser Mode: IE8 Compatibility View, Document Mode: IE8 Standards Mode
.
This is only going to get crazier with each new Internet Explorer Release.

Comment by GregHouston — September 10, 2008

The problem with the VML hack is that its performance is not good, and certainly operations are either prohibitively expensive or impractical to emulate.

Honestly, I think the solution is an ActiveX Canvas implementation, stashed away in something like Google Gears.

Comment by cromwellian — September 10, 2008

I love the idea of Google Gears, (and much more so the HTML5 version that will eventually not require a plugin), but you mention performance as being a reason for going with ActiveX in Gears rather than using VML. A couple weeks ago I spent a weekend experimenting with Gears in MochaUI, and found that in the offline Mode, my app was almost completely unfunctional in Firefox for the first 60 seconds. For the first 60 seconds of going offline my CPU stays at about 20%. After 60 seconds, almost on the dot, it drops to fluctuating between 0-2% and my app becomes useful. Though my CPU is only at 20% it feels like it is at 100%. Dragging an element becomes just about impossible.
.
This problem didn’t exist in IE, but IE has its own Gears issues. Granted gears is in beta, but I scrapped the idea of playing with it for at least the time being. Not to mention it is not supported by Opera yet, though I think there is a beta version for Safari.
.
Also, if an ActiveX control is bundled with Gears, does that mean the end user has to both allow the Gears installation and then allow the ActiveX control on top of that? Thats a lot of steps for the end user to go through just to see a graph.

Comment by GregHouston — September 10, 2008

@ibolmo: LOL you’re the only one bickering. Everyone else wants them to do the damn thing.

Comment by eyelidlessness — September 11, 2008

Before creating an ActiveX Canvas Control based on the Firefox Canvas implementation, I would suggest to wrap the whole Gecko or Webkit engine into an ActiveX Control and replace the whole rendering engine of IE for specially marked pages… I still don’t know why no one ever tried to do this.

Comment by AndiSkater — September 11, 2008

The Silverlight version of ExCanvas is a lot faster. It is still under development (I haven’t touched it for months) but last time I tested it passed most of the tests that the VML based version of ExCanvas passed.

http://excanvas.svn.sourceforge.net/viewvc/excanvas/silverlight/

Eventually, we want to finish this and pick the fastest method available. That being ActiveX, Silverlight or VML.

Comment by ErikArvidsson — September 11, 2008

For anyone still following this thread, I just tried out Ibolmo’s Moocanvas in IE8 beta 2 and was able to get it to work in a lot more IE8 mode combinations than excanvas:
.
Excanvas works in:
– Browser Mode: IE7, Document Mode: Quirks Mode

Moocanvas works in:
– Browser Mode: IE7, Document Mode: Quirks Mode
– Browser Mode: IE7, Document Mode: IE7 Standards Mode
– Browser Mode: IE8, Document Mode: Quirks Mode
– Browser Mode: IE8, Document Mode: IE7 Standards Mode
– Browser Mode: IE8 Compatibility View, Document Mode: Quirks Mode
– Browser Mode: IE8 Compatibility View, Document Mode: IE7 Standards Mode
.
What does not work with Moocanvas is IE8 standards mode. I am not sure what loss of CSS features new to IE8 this may or may not entail. If you are in IE7 standards mode does that mean you are locked out of any new CSS features after IE7? I’ll have to test to see. I hope not.
.
The last time I tried it out, Moocanvas didn’t have support for rotate. That seemed to be the only drawback at the time, but I haven’t tested it in the version I just downloaded.
.
http://ibolmo.com/projects/moocanvas/

Comment by GregHouston — September 11, 2008

I tested :after in IE8’s IE7 Standards mode and it does not work, though it does in IE8 standards mode. So if you are using either Excanvas or Moocanvas, neither of which currently work with IE8 Standards mode you are most likely locked out of using any new CSS features that come after IE7 … unless Microsoft changes something between now and their next beta release.

Comment by GregHouston — September 11, 2008

As a follow-up, using Moocanvas I created a quick proof of concept for a workaround that allows working with canvas/vml in a page viewed with IE8 beta 2 in IE8 standards mode:
http://greghoustondesign.com/examples/canvas-ie8/

Comment by GregHouston — September 13, 2008

Leave a comment

You must be logged in to post a comment.