The Pixel Farm

Last spring I had the opportunity to work with Nisma Zaman on a series of assistive technology projects for inpatient kids at NYU’s Rusk Institute of Rehabilitation Medicine.  The overarching goal for this project was to try and find ways to create entertaining technology driven-experiences for patients to play with in the center’s main recreation room.  The following video shows our first game, called “PairBurst Memory,” in action.

PairBurst Memory from Nisma Z on Vimeo.

“PairBurst Memory,” is an augmented reality matching game that asks kids to try and find pairs of 3D animals that are “hidden” behind pieces of a wooden puzzle.  The game setup consists of a ceiling mounted projector, a Mac Mini, a wooden puzzle board and a webcam that is mounted slightly above and behind the player.  The game software is designed to track a series of nine symbols that exist underneath each puzzle piece.  When visible, the PairBurst software will augment the uncovered symbols with one of four 3D animals.  Upon finding a match the animals will fly off of their spaces and land in the center of the board.  When all four matches are found the animals fly back and celebrate by having a 3D dance party.

For more information please visit the project site at http://www.pairburst.com.

Here’s a quick demo shows an experimental AR system that lets people construct their own augmented reality spaces by manipulating a series of printed cards.  The screenshot and video below showcase how this kind of system can be used to grow a small virtual garden on your desktop.

Creating a technocolor garden by moving around a series of printed cards

The eventual goal for this project is to create a series of physical manipulatives that can be used in a pediatric rehabilitation setting for recreation & therapy purposes.  I’ll be sure to provide more information as we begin testing the AR environment with actual users in the weeks ahead.

Here’s a working copy of the most recent version of the ‘Grab-a-Star’ assistive technology game.  New features include:

• Instruction panel to help players physically orient themselves before the game starts
• A visual indicator that reminds players about their orientation throughout the game
• Vertical star motion
• Quick score indicator that gets triggered by a successful star grab
• (somewhat) less annoying sounds!

Video:

Live version:

Below is the most recent working prototype of our “Grab the Star!” assistive technology tool.  The program is designed to function as follows:

1. The system provides a “mirror-like” interface that allows a patient to see his or her reflection
2. A star appears at the top of the screen.  The object of the game is to “grab” the star repeatedly – once the star has been touched it will reorient itself to the opposite side of the screen at a random location.
3. The system keeps track of the number of grabs as well as how fast a patient is able to react to a star in its new position.
4. At the end of a game the system will provide detailed statistics, including a graphical representation of a patients reaction time and average seek times based upon the position of the star.

Feel free to give it a try! Just click the ‘Start’ button below to get started.

The following is a motion detection “shell” that I wrote in Flash over the weekend.  It’s designed to act as the basis for a number of upcoming assistive technology projects that our group is planning to implement throughout the rest of the semester.  You can try it out by doing the following:

• Click the ‘Start’ button to begin
• When your webcam video shows up you will be presented with your image in addition to three squares that are tiled along the top of the screen.  These squares are “hot spots” and are constantly looking for motion within their boundaries.  Upon detection they will change to a gold color.
• The squares are also “draggable” – dropping a square somewhere else on the canvas will cause that square to recalibrate its “control” value.
• You can also manually recalibrate the video by clicking on the ‘Properties’ panel at the bottom of the screen.  From here you click ‘Recalibrate’ to cause all the squares to resample their portion of the incoming video.
• The ‘Properties’ panel also contains a tolerance interface – this allows you to make the system more or less sensitive based on your preferences.