Monday, March 2nd, 2009

Manipulating video in real time with Canvas

Category: Canvas

Now that we are getting browsers that natively grok video as well as images and the like, we can do interesting things.

Paul Rouget has demonstrated “chroma-keying” (green screen effect) using JavaScript, the video tag, and canvas:

  1. The document establishes two canvas elements, with the IDs c1 and c2. Canvas c1 is used to display the current frame of the original video, while c2 is used to display the video after performing the chroma-keying effect; c2 is preloaded with the still image that will be used to replace the green background in the video.
  2. The JavaScript code is imported from a script named main.js; this script uses JavaScript 1.8 features, so that version is specified in line 22 when importing the script.
  3. When the document loads, the processor.doLoad() method in main.js gets run.

    The work to do the image manipulation is:

    computeFrame: function() {
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);
    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) { let r = frame.data[i * 4 + 0]; let g = frame.data[i * 4 + 1]; let b = frame.data[i * 4 + 2]; if (g > 100 && r > 100 && b < 43) frame.data[i * 4 + 3] = 0; } this.ctx2.putImageData(frame, 0, 0); return; } [/javascript] Fun stuff!

Posted by Dion Almaer at 12:44 pm
7 Comments

++++-
4.2 rating from 29 votes

7 Comments »

Comments feed TrackBack URI

Firefox 3.1 has some amazing performance. I’ve been doing a lot of tests between the Opera 10alpha, Safari 4, and Firefox 3.1, and am very happy with these next-generation browsers and their support of the tag. Seems like they all might be coming out within the next two months… get ready for some exciting new webapps :)

Comment by mudx — March 2, 2009

leptons –

great story. I believe you, I remember seeing a demo of something like that back in the day.

still – I’m fired up to try this at home tonight w/FF 3.1, and that this (video canvas element) will someday come to popular video sites instead of only via an add-on (flash).

Comment by blinkingmarquee — March 2, 2009

@leptons: I’ve been looking for references to early IE work like that. Were there Direct3D bindings that could be used within the browser? Were these ActiveX controls? Have any docs or examples? Do they work anymore?

Comment by Brad Neuberg — March 3, 2009

@Brad Neuberg:

This is from 2003 or 2004, and it’s IE only! hah

real-time video mixer with pixelation
http://www.atomicmandala.com

A Direct3D example – tested on IE 5.5 on WinME and IE6, texture-mapped 3D animation in the browser in 1999.

http://www.atomicmandala.com/direct3d/

Direct3D ended with IE6, it became a security risk or maybe silverlight killed off interest within MS for continuing to support it, and now it seems very difficult to find any information on the internet about Direct3D at all. I’d be very interested in seeing the docs for it if anyone knows how to access them.

Comment by leptons — March 3, 2009

That was awesome!!

Comment by drewlesueur — March 3, 2009

The good stuff was Direct Animation. There was an exploit in the code and Microsoft killed it instead of fixing. I lost over a hundred scripts because of that.
.
Some good stuff:
http://www.mgifos.demon.co.uk/directanimation.htm
http://www.mgifos.demon.co.uk/vault/vault.zip
http://www.sworks.com/keng/da.html

Comment by TNO — March 3, 2009

thanks TNO, I meant to mention direct animation. wow, its been a long time since i’ve played with it. Yeah, there were some pretty cool demos of it before MS took it out in IE7.

IE does still support custom CSS filters, which is what I used for the pixelation effect on an embeded media player using the ‘windowlessVideo’ property to render the video without overlay – thereby allowing the video to render with the CSS filter effects. there are all sorts of other filter effects, chroma-key being one of them, although I don’t know if its that useful in IE, it uses only a single color. The matrix transform is interesting though, it can be used to rotate images/video (or any html element) in IE.

I’m glad canvas is supporting some of these things with video, it gives more possibilities for interesting cross-browser content.

Comment by leptons — March 3, 2009

Leave a comment

You must be logged in to post a comment.