The Pixel Farm

The “Whisper Deck” is an experimental voice-controlled augmented reality data visualization interface (wow, that’s a mouthful!)  The system allows you to don a pair head-mounted goggles and speak commands into a small microphone.  These commands are then recognized by a speech-to-text parser and executed in augmented reality in near real-time.  Think of it like a simplistic, very awkward non-tangible holodeck

As of today the system can help to visualize search volume using a tie in to Google Trends.  Simply speak the command “compare” followed by a series of keywords and the Whisper Deck will do the rest! Here’s a quick video that shows the prototype in action.

Now that I have a working prototype I hope to expand the systems functionality to do a number of things, including:

1. Access to additional APIs (Wikipedia, Google Images, Facebook, etc.)
2. New commands that provide visual, 3D search capabilities
3. Export options for Twitter and Facebook (say “tweet this” and a snapshot of your page + a brief spoken caption are uploaded to a twitter feed)

For the last year I’ve been on the lookout for a decently priced head mounted display with an integrated camera, and last week I found a device that fits this bill.  Introducing the iWear Vuzix VR920 + AR unit, which offers two 640 x 480 LCD screens as well as a clip-on USB powered webcam.

Just call me Geordi LaForge

The Vuzix VR920 + CamAR webcam (snapped onto the front)

Upon delivery I immediately “goggled” into the metaverse and thew together a quick augmented reality demo.  The system is fairly responsive, and I found that installing the optional “light filter” insert (basically a piece of molded rubber that blocks out ambient light from above and below your eyes) really helps in making the illusion feel more “believable.”  Here’s a quick video of me watching some cubes bounce around on my kitchen table.

As with all AR projects, the illusion is dependent on the presence of a marker of some kind.  Wearing the goggles gives a user much more freedom of movement, which means that you tend to lose sight of the marker more often than you would while using your laptop’s fixed webcam.  My next project will be to try and overcome this problem by creating a set of redundant markers that can act in concert to help provide overlap coverage for when a single marker goes out of view.

I just finished presenting a session on “Augmented Reality in the Classroom” at this year’s NJEDge.Net Conference. For those of you who stopped by, thanks! Here’s a quick recap of some of the videos that I referred to during the talk. My own Augmented Reality demos can be found by clicking on the “Augmented Reality” category tag to the left.

Link: The REACH Project: Infinite Games for Pediatric Rehabilitation

Augmented Reality – The Future of Education

A Visit to Stanford’s Virtual Human Interaction Lab

A Soldier’s Therapy Session: Using Virtual Reality to help Patients with PTSD

Topps Baseball Cards in Augmented Reality

Mixed Reality Learning Media: Effects on Classroom Instruction

In-Place Augmented Reality 3D Sketching of Mechanical Systems

Fashionista: Webcam Social Shopper

New York Nearest Subway iPhone 3GS Application

Unveiling the Sixth Sense: Wearable Augmented Reality

Xbox Natal

The Daily Show: Augmented Reality in Iowa Caucus

I’ve been experimenting with ways in which AR can be used to control other systems, including Second Life. Here is a quick demo of an AR cube that is tied to a Second Life “chair” that my avatar is sitting on. In the future I would like to set up a complete control system for an avatar using AR or color tracking. The system that is demonstrated below uses the following:

1. Single marker tracking to orient a cube in 3D space
2. When visible, the 3D transformation matrix of the cube is sent to my webserver
3. In Second Life, the “seat” that my avatar is sitting on calls my webserver and asks for the current transformation matrix. After unpacking the data it reorients the cube in Second Life 3D space.

My dog Prue co-stars in the demo video below. Perhaps this one should be titled “dog-mented reality” instead (I know, it’s a terrible joke!)

This week I decided to try and hook up an augmented reality scene so that it can be driven by SMS messages. The app below lets you vote for your favorite flower using your mobile phone – just text the appropriate code (”craigk red” or “craigk blue”) to 41411 to cast your vote!

The system is built using the public version of the TextMarks SMS aggregation system, a pair of back-end Perl scripts and, of course, Flash + Papervision.

Vote for your favorite flower!

Check out a video of the project in action: