Kinect

Yesterday I took the train down to Brown see a set of cool vision talks. One of the presenters talked about "RGB-D" images, which mean different things to different people. To me, RGB-D is an image abstraction. It augments the red, green and blue color channels with a fourth channel which is the pixel depth (the "D" in RGB-D).

Although there are tons and tons and tons of methods that reconstruct scene depths from images, RGB-D folks are agnostic to these. They have an interesting abstraction subtext to their work, which could be summarized as: "Pick a vision reconstruction method that works really well. Lets have it implemented as a hardware black box . Lets now celebrate the fact that we don't have to worry about depth-reconstruction/shape-from-X again. We'll assume that we have perfect depth, and lets build some cool vision on top of that."

I think this attitude is awesome since it gets people to think beyond scene reconstruction.

One of the "black boxes" that give RGB-D images is the Microsoft Kinect. Whats interesting for micro computer vision is an interesting sub-component of the Kinect: a tiny IR projector.

Correction: The Kinect has no IR projector. See updates below.

You can see pics of it here. The Kinect has a stereo-pair-with-projector system. The projector adds texture for the stereo system, but its "projected light" is unnoticed since its in IR and so invisible (You can see the projected pattern here, where a video was taken with a night-vision camera.)

I believe the projector pattern is fixed. This means Microsoft could have gotten away with a very bright IR LED and a physical, printed texture pattern. Why did they use a projector? I'm not sure, but there is an opportunity to hack and control the projector. I'm surprised to have not found anything along those lines yet on the web. I'm particularly curious about the projector's frame rate, and whether high-speed applications are possible.

Update: We have confirmation that the Kinect does not have a projector at all. Thanks to AV for the update. (Also, people actually read the blog.)

Update 2: Thanks to GD for pointing out this website kinecthacks.net