Embedded vision is a clear "poster child" for digital signal processing silicon and software technology, beginning with the algorithms employed in image capture (including exposure, color balance, lens aberration correction, etc) through edge detection and pattern matching, and extending all the way to frame analysis and response (motion tracking, for example). And Microsoft's Kinect peripheral for the Xbox 360 game console, which sold 10 million units in its first five months on the market (not to mention 8 million units in its first 60 days), has quickly become the poster child for embedded vision technology's expanding application embrace beyond the historical computer vision niche.
Although Microsoft initially resisted enthusiasts' attempts to broaden the multi-image sensor device's reach beyond living room entertainment, the company quickly did an abrupt about-face to the enthusiastic response of academics, hobbyists and commercial developers alike. Coincident with Kinect's November 4, 2010 launch in the United States, a DIY advocacy and materials supplier called Adafruit Industries announced a $2,000 bounty to the first hacker who successfully reverse-engineered the peripheral's interface protocol:
What do we (all) want?
Open source drivers for this cool USB device, the drivers and/or application can run on any operating system―but completely documented and under an open source license. To demonstrate the driver you must also write an application with one "window" showing video (640 x 480) and one window showing depth. Upload all of this to GitHub.
Microsoft's response was swift and terse, as reported by CNET:
"Microsoft does not condone the modification of its products," a company spokesperson told CNET. "With Kinect, Microsoft built in numerous hardware and software safeguards designed to reduce the chances of product tampering. Microsoft will continue to make advances in these types of safeguards and work closely with law enforcement and product safety groups to keep Kinect tamper-resistant."
And Adafruit's reaction was equally rapid, not to mention rebellious; it promptly upped the bounty ante to $3,000.
Six days later, the winner was announced; Hector Martin developed his code on a Linux laptop and didn't even own an Xbox 360:
Here’s my take on the Kinect driver. Supports depth and RGB images and displays them on an OpenGL window. It’s very hacky right now but it does prove the concept.
From there, the academic, enthusiast and industry communities were off and running, rapidly developing a diversity of 3-D vision-augmented applications; visit the Kinect Hacks site for some truly stunning examples of engineering creativity.
Sensing a potential leadership opportunity lost, Microsoft quickly back-peddled, claiming that the prior remarks were due to the spokesperson's misunderstanding the corporate strategy, and explicitly noting that the USB connection between Kinect and the console was left open (i.e. unencrypted) by design. To wit, here are abridged comments from Microsoft's Alex Kipman (Xbox director of incubation) and Shannon Loftis (Game Studios manager), quoted on the November 19, 2010 edition of National Public Radio's Science Friday program during an interview conducted by Ira Flatow and also including Professor Katherine Isbister from New York University's Polytechnic Institute:
Mr. KIPMAN: The first thing to talk about is Kinect was not actually hacked. Hacking would mean that someone got to our algorithms that sit on the side of the Xbox and was able to actually use them, which hasn't happened. Or it means that you put a device between the sensor and the Xbox for means of cheating, which also has not happened. That's what we call hacking, and that's why we have put a ton of work and effort to make sure it doesn't actually occur.
Mr. KIPMAN: What has happened is someone wrote an open-source driver for PCs that essentially opens the USB connection, which we didn't protect by design, and reads the inputs from the sensor. The sensor again, as I talked earlier, has eyes and ears and that's a whole bunch of, you know, noise that someone needs to take and turn into signal.
FLATOW: So you left it open by design then? So you knew people could get into it.
Mr. KIPMAN: Yeah. Correct.
Prof. ISBISTER: And I just want to throw in that this is the kind of thing that's a dream for researchers like myself. I mean, I still haven't got an actually developer's kit for the Wii, but we use the open-source shareable inputs to the Wii-motes. And that's how we work with the Wii technology, and so I was very heartened to see that the Kinect's actual hardware was going to be available soon for researchers anyway to put stuff together and test in the lab.
FLATOW: So you have no problem with people using the open-source drivers then?
Ms. LOFTIS: As an experienced creator, I'm very excited to see that people are so inspired that it was less than a week after the Kinect came out before they had started creating and thinking about what they could do.
FLATOW: So no one is going to get in trouble?
Mr. KIPMAN: Nope. Absolutely not.
Ms. LOFTIS: No.
FLATOW: You heard it right from the mouth of Microsoft.
(Soundbite of laughter)
Three months later, Microsoft's beyond-the-Xbox 360 plans for Kinect got even more serious. As noted in a news entry on the Embedded Vision Alliance website on July 27, Microsoft's Craig Mundie (the company's chief research and strategy officer) and Don Mattrick (president of the Interactive Entertainment Business unit) announced at TechForum on February 21 the company's planned release of a non-commercial Kinect SDK later that spring. Developed and planned for release by Microsoft Research in collaboration with IEB, this free download was intended to "give academic researchers and enthusiasts access to key pieces of the Kinect system—such as the audio technology, system application programming interfaces and direct control of the Kinect sensor itself."
The freely downloadable SDK public beta appeared on June 16. Focused on non-commercial purposes ("a commercial version is expected to be available at a later date"), it's also tailored for developers using Windows 7-based computers (sorry, all you other-O/S folks, you'll need to continue using open source community-developed tools). Here's what Microsoft's site says about it:
The Kinect for Windows SDK beta includes drivers, rich APIs for raw sensor streams and human motion tracking, installation documents, and resource materials. It provides Kinect capabilities to developers who build applications with C++, C#, or Visual Basic by using Microsoft Visual Studio 2010.
This SDK includes the following features:
· Raw sensor streams: Access to raw data streams from the depth sensor, color camera sensor, and four-element microphone array enables developers to build upon the low-level streams that are generated by the Kinect sensor.
· Skeletal tracking: The capability to track the skeleton image of one or two people moving within the Kinect field of view make it easy to create gesture-driven applications.
· Advanced audio capabilities: Audio processing capabilities include sophisticated acoustic noise suppression and echo cancellation, beam formation to identify the current sound source, and integration with the Windows speech recognition API.
· Sample code and documentation: The SDK includes more than 100 pages of technical documentation. In addition to built-in help files, the documentation includes detailed walkthroughs for most samples provided with the SDK.
· Easy installation: The SDK installs quickly, requires no complex configuration, and the complete installer size is less than 100 MB. Developers can get up and running in just a few minutes with a standard standalone Kinect sensor unit (widely available at retail outlets).
More recently, in mid-July, Microsoft released the equally-freely-downloadable Kinect Services suite for Robotics Developer Studio 2008 R3. At the time, Stathis Papaefstathiou, General Manager of Microsoft's Robotics division, noted, "Kinect Services for RDS is built on top of the Kinect for Windows SDK. We have wrapped in the core Kinect SDK capabilities so you can use them with the CCR [editor note: Concurrency and Coordination Runtime]/DSS [Decentralized Software Services] programming model. However, this is only the beginning. Keep your eyes peeled for updates in the fall. We know that there are many practical issues for applying the Kinect technologies to robotics capabilities that we will be addressing in upcoming RDS releases."
Robotics Developer Studio 1.0 first became publicly available from Microsoft Research in December 2006. Industry adoption of it has to date been somewhat muted, albeit with a few notable application exceptions. Nonetheless, Papaefstathiou concluded his mid-July comments as follows, "We look forward to working with the developer community to create a common platform―using Kinect―to co-create consumer robotics projects and scenarios." I wholeheartedly concur. As noted in a separate news entry on the Embedded Vision Alliance website, on August 1, robust vision support is at the nexus of robust autonomous robot implementations. The combination of Robotics Developer Studio 2008 R3 and its companion Kinect Services suite will undoubtedly go a long way towards translating robotics developers' visions into reality.
Add new comment