waterin' the virtual plants

The game is currently a very rough prototype and is missing all sorts of niceties (i.e. sound effects, scoring mechanisms, etc.) – hopefully I can refine this framework so that it can serve as the basis for a host of color-drive 2D AR games.

If you’d like to try it yourself feel free to click on the ‘Start’ button below.  You will need a webcam and a solid colored object about 2″ by 2″ in order to play.  When prompted, simply hold up this object so that it fits within the small rectangle in the middle of the screen.  Once the object appears within this region, click on the button that appears to cause the game to “memorize” the color value of your object.  From here you will be able to wave your object around in front of your webcam as though it was a mouse cursor.

The voice control system that I created is based on a client-side software package called “MacSpeech Dictate.” Voice recognition works as follows:

1. Launch the MacSpeech Dictate recognition engine
2. Place cursor focus in a text box at the bottom of the Whisper Deck interface.  This text box is not visible in the Youtube demo of the project.
3. Speak into the microphone.  Recognized words are transcribed by MacSpeech Dictate and placed into the text box in the Flash movie.
4. Flash listens for an Event.CHANGE event to fire on the text box.  When it does, it starts parsing the text that was transcribed by MacSpeech Dictate.  Here are the general steps my parsing routine goes through:
   1. Convert the entire spoken string to lowercase (”this.mytextfield.text = this.mytextfield.text.toLowerCase();“)
   2. Parse out any leading spaces
   3. Split the transcribed sentence into individual array elements based on the placement of spaces (i.e. “hello world how are you” would parse out to a new array with the following elements
      1. myarray[0] = “hello”;
      2. myarray[1] = “world”;
      3. myarray[2] = “how”;
      4. myarray[3] = “are”;
      5. myarray[4] = “you”;
   4. Look for the last element of the array to be the world “over” – this is used as a trigger to tell Flash to process the command in its entirety.
   5. If the “over” command is present, look at the first element of the array.  Current the Whisper Deck can recognize two commands (”search” and “compare”) – if either of these commands is present, pass the command to the appropriate AR rendering class.

Originally I did not include the “over” keyword as part of the system – instead I used a period of microphone inactivity as a cue to tell the program that I was done speaking.  Unfortunately this did not provide very stable results – machine transcription, even under quiet conditions, is fraught with errors, which led to a lot of inaccurate searches.  “Over” was included as a safety buffer to let me “proofread” my voice command before I asked the program to process it.  It works well for demo purposes, but I can see that it’s a limitation of the system that I will need to work out if the project was to move forward.

At some point I would love to play around with a web-accessible machine translation routine, similar to what Didier Brun has accomplished in the video below.  Unfortunately I was pressed for time on this application, and MacSpeech Dictate worked very well given the design requirements for this project.

Voice Gesture from didier.brun on Vimeo.

The Whisper Deck is a voice-controlled augmented reality data visualization tool that immerses users within a fluid information ecosystem of their own design.  The project is an experimental interface that explores new ways in which we can examine the vast amount of data being generated by the world on a daily basis.

Visualizing google trends via the Whisper Deck

Video

Description

Using an off the shelf Vuzix Cam-AR head mounted display, users can look around their local environment and examine the world through the integrated webcam unit on the front of the display.  Upon noticing a pre-defined symbol, a 3D world instantly appears.  As long as this symbol remains in view, this newly created augmented space will continue to persist and will allow the user to examine it from any direction by simply moving around it in real space.

Users can issue requests to the Whisper deck using a series of voice commands.  These commands will cause the world to reconfigure itself based upon your preferences.   For example, if you would like to have the world gather information about a topic you are interested in – say, Boston Terrier Puppies – simply say the command “search Boston terrier puppies” with the keyword “over” at the end of your sentence.  The system will go out to the Internet and retrieve information relating to your request, including a spoken definition from Wikipedia as well as a set of images from various publicly accessible image search engines.

Image Search using the Whisper Deck

In addition, the Whisper Deck also allows visitors to compare the relative popularity of search term by interfacing with Google Trends.  Speaking the command “compare” will allow you to name any number of terms which will be visualized as a 3D bar chart that can be further inspected.

Technology

The Whisper Deck uses a number of different tools.  While most of the technologies described below are web-friendly, the voice controlled aspect of the system is handled via a desktop speech to text package.

• Flash ActionScript 3
  • FLARToolkit (marker detection)
  • Papervison 3D (3D rendering)

• Web Services
  • Yahoo!  Pipes (Flickr, Picasa & Google Images feed aggregation)
  • Perl + Python (Google Trends integration)

• Voice Recognition / Playback
  • Mac Speech Dictate
  • Perl + integrated Apple OS X text to speech engine

Augmented Gnomes popping out of a sheet of paper!

While it’s not fully completed, I do have a working version that lets you construct simple scenes.  The current version can do the following:

• Add 2D planes to a 3D space
• Texture these planes using transparent PNG files
• Orient objects and adjust rotation and scale
• Create dynamic “cut-out” shadows based on the original material
• Handle timing to allow items to pop out in a specific sequence
• Save the file to an external server for permanent storage
• Reconstruct the scene in augmented reality

Here is a brief video that showcases the pop-up authoring environment as well as a finished product in action.

If you’re interested in seeing portions of this project in action, feel free to stop by the ITP Winter Show this weekend.  I will be showing off a voice controlled augmented reality project called the “Whisper Deck” which incorporates my AR pop-up books as one of its many interactive features.

Over the last few months we have refined this prototype and have folded it into a larger set of synchronous web-based tools designed to help enhance communication between instructors and students in an online learning space.  Using the Social Syllabus course delivery framework as a platform, John and I enhanced the system’s existing communication structure to take advantage of a range of next-generation communication and collaboration tools, including:

1. Audio commenting system.  The audio commenting system gives users of Social Syllabus the ability to easily post audio “notes” into almost any facet of an online course.  Using an embedded Flash interface, faculty and students can record their voice using their computer’s microphone.  In addition, the voice notes tool also lets users edit their audio note before saving it to the public class space. Currently this functionality is limited to the course blog and calendaring modules of Social Syllabus, but we are working to expand it to other areas of the site, including giving faculty members the ability to create their own ad-hoc podcasts on the fly.
   

   in-line audio player for Social Syllabus

2. Lecture streaming system. The lecture streaming system gives instructors of a course the ability to stream out their lecture without having to download or install any software on their machine.  Using an embedded Flash movie, professors can easily broadcast their webcam and microphone to students from within Social Syllabus.  Professors can get up-to-the minute statistics on the number of live viewers, and we are working to build in an “archive” feature that lets instructors store their lectures for later delivery.
   

   Broadcasting a class

3. Social Search: The Social Search tool is a meta-search engine that enables instructors to rapidly search multiple Internet sources for relevant course content.  Typing a keyword into the social search interface will cause Social Syllabus to return resources from a wide range of external sources, such as Delicious, Digg, Youtube, Twitter, Google Scholar, Amazon Books and Scribd.  Materials found via the social search can be incorporated into an online course and made available to students with a single click.
   

   The 'social search' meta-search feature

The “Whisper Deck” is an experimental voice-controlled augmented reality data visualization interface (wow, that’s a mouthful!)  The system allows you to don a pair head-mounted goggles and speak commands into a small microphone.  These commands are then recognized by a speech-to-text parser and executed in augmented reality in near real-time.  Think of it like a simplistic, very awkward non-tangible holodeck

As of today the system can help to visualize search volume using a tie in to Google Trends.  Simply speak the command “compare” followed by a series of keywords and the Whisper Deck will do the rest! Here’s a quick video that shows the prototype in action.

Now that I have a working prototype I hope to expand the systems functionality to do a number of things, including:

1. Access to additional APIs (Wikipedia, Google Images, Facebook, etc.)
2. New commands that provide visual, 3D search capabilities
3. Export options for Twitter and Facebook (say “tweet this” and a snapshot of your page + a brief spoken caption are uploaded to a twitter feed)

Just call me Geordi LaForge

The Vuzix VR920 + CamAR webcam (snapped onto the front)

Upon delivery I immediately “goggled” into the metaverse and thew together a quick augmented reality demo.  The system is fairly responsive, and I found that installing the optional “light filter” insert (basically a piece of molded rubber that blocks out ambient light from above and below your eyes) really helps in making the illusion feel more “believable.”  Here’s a quick video of me watching some cubes bounce around on my kitchen table.

As with all AR projects, the illusion is dependent on the presence of a marker of some kind.  Wearing the goggles gives a user much more freedom of movement, which means that you tend to lose sight of the marker more often than you would while using your laptop’s fixed webcam.  My next project will be to try and overcome this problem by creating a set of redundant markers that can act in concert to help provide overlap coverage for when a single marker goes out of view.

1. The vast majority of hits to my site came from locations within the USA. However it also seems as though I have a small but loyal fan club from Barcelona, Spain who have visited the site 205 times during the last month.
2. The search term “craig kapp girlfriend” was used to find my site 6 times in the last week. The computer that originated the request lives somewhere in Brooklyn, NY, and this individual stayed on my site for an average of 4:22 minutes, much longer than the average visit over that time (2.18 minutes.) I have no idea who this person is!
3. A surprising amount of traffic directed to my site is focused on finding information about “kid friendly virtual worlds.”

In the interest of further mining this rich set of data I have constructed a flash-based inspector that parses individual search terms used in locating my blog. The program breaks down search terms into individual words and ranks them by frequency of use through a dynamically generated bar chart. Mousing over a bar will display all search variations that use that particular word (i.e. “craig” has been used in the search terms “craig kapp,” “craig kapp itp” and “sex chat rooms blog.craigkapp.com” – yikes! Let’s hope that last one was a spambot …

The tool is currently running off a snapshot of last month’s data, but I’d like to hook it up to a live feed for up to the minute analysis. Feel free to check it out.

Link: The REACH Project: Infinite Games for Pediatric Rehabilitation

Augmented Reality – The Future of Education

A Visit to Stanford’s Virtual Human Interaction Lab

A Soldier’s Therapy Session: Using Virtual Reality to help Patients with PTSD

Topps Baseball Cards in Augmented Reality

Mixed Reality Learning Media: Effects on Classroom Instruction

In-Place Augmented Reality 3D Sketching of Mechanical Systems

Fashionista: Webcam Social Shopper

New York Nearest Subway iPhone 3GS Application

Unveiling the Sixth Sense: Wearable Augmented Reality

Xbox Natal

The Daily Show: Augmented Reality in Iowa Caucus