The Pixel Farm

Overview

The Whisper Deck is a voice-controlled augmented reality data visualization tool that immerses users within a fluid information ecosystem of their own design.  The project is an experimental interface that explores new ways in which we can examine the vast amount of data being generated by the world on a daily basis.

Visualizing google trends via the Whisper Deck

Video

Description

Using an off the shelf Vuzix Cam-AR head mounted display, users can look around their local environment and examine the world through the integrated webcam unit on the front of the display.  Upon noticing a pre-defined symbol, a 3D world instantly appears.  As long as this symbol remains in view, this newly created augmented space will continue to persist and will allow the user to examine it from any direction by simply moving around it in real space.

Users can issue requests to the Whisper deck using a series of voice commands.  These commands will cause the world to reconfigure itself based upon your preferences.   For example, if you would like to have the world gather information about a topic you are interested in – say, Boston Terrier Puppies – simply say the command “search Boston terrier puppies” with the keyword “over” at the end of your sentence.  You can search almost any topic from ice cream to casino to cars or anything that interests you. The system will go out to the Internet and retrieve information relating to your request, including a spoken definition from Wikipedia as well as a set of images from various publicly accessible image search engines.

Image Search using the Whisper Deck

In addition, the Whisper Deck also allows visitors to compare the relative popularity of search term by interfacing with Google Trends.  Speaking the command “compare” will allow you to name any number of terms which will be visualized as a 3D bar chart that can be further inspected.

Technology

The Whisper Deck uses a number of different tools.  While most of the technologies described below are web-friendly, the voice controlled aspect of the system is handled via a desktop speech to text package.

• Flash ActionScript 3
  • FLARToolkit (marker detection)
  • Papervison 3D (3D rendering)

• Web Services
  • Yahoo!  Pipes (Flickr, Picasa & Google Images feed aggregation)
  • Perl + Python (Google Trends integration)

• Voice Recognition / Playback
  • Mac Speech Dictate
  • Perl + integrated Apple OS X text to speech engine

The “Whisper Deck” is an experimental voice-controlled augmented reality data visualization interface (wow, that’s a mouthful!)  The system allows you to don a pair head-mounted goggles and speak commands into a small microphone.  These commands are then recognized by a speech-to-text parser and executed in augmented reality in near real-time.  Think of it like a simplistic, very awkward non-tangible holodeck

As of today the system can help to visualize search volume using a tie in to Google Trends.  Simply speak the command “compare” followed by a series of keywords and the Whisper Deck will do the rest! Here’s a quick video that shows the prototype in action.

Now that I have a working prototype I hope to expand the systems functionality to do a number of things, including:

1. Access to additional APIs (Wikipedia, Google Images, Facebook, etc.)
2. New commands that provide visual, 3D search capabilities
3. Export options for Twitter and Facebook (say “tweet this” and a snapshot of your page + a brief spoken caption are uploaded to a twitter feed)

For the last year I’ve been on the lookout for a decently priced head mounted display with an integrated camera, and last week I found a device that fits this bill.  Introducing the iWear Vuzix VR920 + AR unit, which offers two 640 x 480 LCD screens as well as a clip-on USB powered webcam.

Just call me Geordi LaForge

The Vuzix VR920 + CamAR webcam (snapped onto the front)

Upon delivery I immediately “goggled” into the metaverse and thew together a quick augmented reality demo.  The system is fairly responsive, and I found that installing the optional “light filter” insert (basically a piece of molded rubber that blocks out ambient light from above and below your eyes) really helps in making the illusion feel more “believable.”  Here’s a quick video of me watching some cubes bounce around on my kitchen table.

As with all AR projects, the illusion is dependent on the presence of a marker of some kind.  Wearing the goggles gives a user much more freedom of movement, which means that you tend to lose sight of the marker more often than you would while using your laptop’s fixed webcam.  My next project will be to try and overcome this problem by creating a set of redundant markers that can act in concert to help provide overlap coverage for when a single marker goes out of view.

I’ve been experimenting with ways in which AR can be used to control other systems, including Second Life. Here is a quick demo of an AR cube that is tied to a Second Life “chair” that my avatar is sitting on. In the future I would like to set up a complete control system for an avatar using AR or color tracking. The system that is demonstrated below uses the following:

1. Single marker tracking to orient a cube in 3D space
2. When visible, the 3D transformation matrix of the cube is sent to my webserver
3. In Second Life, the “seat” that my avatar is sitting on calls my webserver and asks for the current transformation matrix. After unpacking the data it reorients the cube in Second Life 3D space.

My dog Prue co-stars in the demo video below. Perhaps this one should be titled “dog-mented reality” instead (I know, it’s a terrible joke!)

This week I’ve been playing around with the shared object feature of the Flash Media Server.  So far I’m really liking it!  I was able to whip up a pretty basic prototype of a web-based virtual world without too much fuss – feel free to check it out.

Just me and MC Hammer hanging out in the metaverse

As of today the world can do the following:

1. Avatar identification
2. Synchronous  text-based chat
3. Click-to-move functionality within the world
4. Appearance editor (change your own avatar’s look)

The appearance editor is still very rudimentary, and changes made to your avatar don’t yet ripple out to other users.  I was having a real tough time figuring out how to send out complex AS3 objects using the Shared Object feature of the FMS.  The documentation explicity states that sending any AS3 class via a Shared Object should work. What the documentation fails to tell you is that you need to prepare your classes in a very particular way in order for them to retain their integrity after being serialized by the FMS encoder.

I should have a fix in place for this real soon – I’m uber excited about being able to share non-trivial AS3 objects in real time!