Kinect

Yesterday I took the train down to Brown see a set of cool vision talks. One of the presenters talked about "RGB-D" images, which mean different things to different people. To me, RGB-D is an image abstraction. It augments the red, green and blue color channels with a fourth channel which is the pixel depth (the "D" in RGB-D).

Although there are tons and tons and tons of methods that reconstruct scene depths from images, RGB-D folks are agnostic to these. They have an interesting abstraction subtext to their work, which could be summarized as: "Pick a vision reconstruction method that works really well. Lets have it implemented as a hardware black box . Lets now celebrate the fact that we don't have to worry about depth-reconstruction/shape-from-X again. We'll assume that we have perfect depth, and lets build some cool vision on top of that."

I think this attitude is awesome since it gets people to think beyond scene reconstruction.

One of the "black boxes" that give RGB-D images is the Microsoft Kinect. Whats interesting for micro computer vision is an interesting sub-component of the Kinect: a tiny IR projector.

Correction: The Kinect has no IR projector. See updates below.

You can see pics of it here. The Kinect has a stereo-pair-with-projector system. The projector adds texture for the stereo system, but its "projected light" is unnoticed since its in IR and so invisible (You can see the projected pattern here, where a video was taken with a night-vision camera.)

I believe the projector pattern is fixed. This means Microsoft could have gotten away with a very bright IR LED and a physical, printed texture pattern. Why did they use a projector? I'm not sure, but there is an opportunity to hack and control the projector. I'm surprised to have not found anything along those lines yet on the web. I'm particularly curious about the projector's frame rate, and whether high-speed applications are possible.

Update: We have confirmation that the Kinect does not have a projector at all. Thanks to AV for the update. (Also, people actually read the blog.)

Update 2: Thanks to GD for pointing out this website kinecthacks.net

Stereoscope for the iPhone

The stereoscope was one the first 3D viewing devices. Its a low-tech device that basically places an opaque occluder between your eyes, forcing a separation between what your left and right eye can see. A stereo pair placed correctly on either side of the occluder gives the viewer an impression of depth.

Hasbro has created a very interesting update to the stereoscope for the 21st century. A binoculars-like device pretty much plays the role of an occluder. At the end of the "binoculars", an iPhone or smartphone is attached. A Hasbro app formats 3D content appropriately, so that each eye sees only the correct image through the "binocular" tubes.

This was one of those "why didn't I think of that!" moments for me. Very cool stuff from Hasbro. However, one can't help but think the way forward for mobile 3D devices is more of the autostereoscopic optics from the likes of Nintendo, with its 3DS. The Hasbro device is useful for people who already have an iPhone or similar device and who don't mind the device form factor.

New Micro Four Thirds models

The Micro four thirds cameras were launched by Olympus and Panasonic about three years ago. A quick recap: SLR cameras have large sensors and pixels and therefore low noise. They also have a "mirror in the loop" that allows the viewfinder to show *exactly* what is going to be photographed. This is done by beamsplitting the incoming light into two paths, one of which goes to the sensor and the other to the viewfinder. These two features force SLR cameras to have a large form factor.

The Micro four thirds system basically tries to be a "pocket SLR" by reducing the sensor size (but not by much) and removing the mirror in the viewing path. It also tries to give the consumer an "SLR feel" by having removable and interchangeable lenses. The nytimes has a nice article about the new cameras, and the sizes are 4.4x2.7x1.3 inches.

This blog always likes to see if innovations meant for the marketplace have an impact on research. I'm not seeing any particular feature of these cameras that we could use in research, since neither the small sizes nor the high quality of images are a game-changer. However, you can think of these cameras as a poor man's SLR. Perhaps we could exploit the cheapness factor in some areas of appearance capture, where many cameras are used...are there applications that need 100s of SLR-like cameras?

A note about image noise and sensor resolution. Large pixel sizes collect more light, so its obvious that this would increase SNR. However, when you purchase a camera, you rarely find two candidate cameras with the same resolution, but one with larger pixel sizes: that would be an easy choice. Instead, you may find one camera with a large sensor and small pixels, while another has a smaller sensor but with larger pixels. Each individual pixel in the second camera should have higher SNR, but the overall sensor size of the first one is larger, and that does have some benefit.

There are sites that explain these relationships in detail. I just want to point out that the right decision is not obvious since resolution and SNR get mixed up when you compare across both those quantities.

ICCP post 2

Yesterday was a packed day at ICCP 2011. Illah Nourbakhsh started out the day with an amazing talk on Gigapan imaging. The philosophical breakthrough he made was to contrast a gigapixel image to a real image.

If you view a gigapixel image you cannot see the "whole" picture, since the resolution is huge (unlike a snap of a family picnic). You have to zoom in and explore the image. Now imagine the image is of some place you haven't ever been to, lets say Mars. So now when you zoom in, Illah says that you are exploring that place, not just viewing a picture.

The optics session that followed was pretty amazing too. Oliver Cossairt had a great talk on the new gigapixel camera from Shree Nayar's lab. The final presentation in this session was Hand-Held Schlieren Photography with Light Field Probes, which won the best paper award at ICCP.

ICCP post 1

I'm sitting in the beautiful Rashid Auditorium listening to the opening of ICCP 2011!

Yesterday there were some tutorials.

I loved the talk by Peter Belhumeur on digital plant detection. The topic may sound a little esoteric to the general audience, but the results were beautiful and the paper was interesting. A nytimes article is here.

MicroCV at ICCP 2011

ICCP 2011 starts this week in our fair city of Pittsburgh. MicroCV will be there. Check this space for updates on cool talks, gadgets, presentations and other stuff.

I'll also be presenting a poster on my latest work on optical filtering for micro visual sensors.

More posts soon!

Nintendo 3DS

More updates on the Nintendo 3DS. If you've been following the blog, then you'll know the post we made previously about how awesome the 3DS is.

Nytimes has an article by a 3D skeptic who loves the display. Also, linked are other articles where reviewers gush about the awesomeness of the device. To paraphrase one reviewer, the 3DS might be cool enough to compete with Apple's effortless chic.

I wonder if they've opened up the 3DS for apps other than games. I guess you could write a "game" that had a more practical application...