The Pixel Farm

As I’ve mentioned in a number of previous posts, I’ve been actively working on a new version of the ZooBurst 3D pop-up book creator.  This new version is currently scheduled for release at the end of August 2010, and will be chock full of new features, including:

• Privacy options:  Designate your books as public to the world, password protected or completely private
• Customizable backgrounds:  No more canned backdrops!  You can color your own backgrounds as well as drag and drop images onto the background to make your own landscapes for your books.
• Sound effects:  Choose from a library of sound effects to bring your characters to life.
• Record your own voice:  Use your computer’s microphone to record your own voice to have your characters “speak” when they are clicked
• Classroom management:  Teachers will be able to easily set up private spaces for their students inside of ZooBurst.  Classes in ZooBurst can be public, private or password protected, and teachers can set up usernames and passwords for all students in their class using a simple classroom dashboard tool.

Here’s a quick video shows some of these new features in action.

Hello, blog!  Sorry I’ve been so lax with posting.  I just got back from my summer teaching assignment out in Mallorca Spain. This year, as in previous years, I taught a section of “Educational Applications of Technology” for a group of amazing international school teachers and administrators.  Our class blog is available here if you want to check out what we covered during the summer term.

The summer palace of King Juan Carlos (visible from my old apartment!) - This is where Michelle Obama is meeting the royal family this week on her vacation to Spain.

Mallorca Graduation Summer 2010

TCNJ Mallorca Graduation Celebration, July 2010

Now that I’m back I’m focusing on getting the new version of ZooBurst up and running.  This new version contains a host of new features that were suggested by the comunity of over 3,900 (!) teachers who have signed up since May 2010.  These features include:

• Sound effects via a searchable sound library
• Audio recording to allow your characters to “speak” using your own voice
• Customizable backgrounds
• Advanced page management features (reorder, drag and drop, duplication, etc.)
• Privacy options
• Classroom management mode for teachers
• … and some other surprises!

We’re shooting to have this new framework up and running in time for the beginning of the school year here in North America.  I’ll keep you posted as the release date gets closer.

Ok, back to work!

Over the last few weeks a number of ZooBursters have gotten in touch to ask if there was a way in which they could interact with their 3D pop-up books without having to print out an augmented reality marker.  Many people don’t have easy access to a decent quality printer, and while you can use an external device in place of a marker (see Greg Borenstein’s exploration into using an iPad in place of an AR marker) the glare issues that arise from using a backlit display can be a bit difficult to overcome.  Plus it’s hard to enjoy your 3D creation while holding something up in front of your screen!

In response to these issues I’ve built an experimental features that lets you display your ZooBurst book on the screen without needing to have an AR marker handy.  Once activated, this feature will cause your book to fall out of the sky and bounce into place along the bottom of the video field (see image below).  In addition, a new control tab also appears at the top of the page – this tab “listens” for “swipes” in the video feed.  When detected, these swipes can be used to control the page turning action.

Controlling a ZooBurst book using physical gestures

Here’s a quick video of how the new feature works:

ZooBurst has been getting tons of use over the last few weeks, and I’ve been spending most of my free time making updates to the system and thinking of ways to move forward with the environment. It’s really amazing how fast it’s taken off – as of today we have over 1,000 people actively using the system to create their own 3D pop-up books!

There are a ton of features on my “to-do” list for ZooBurst, but one of the most pressing is to find a way to make the augmented reality feature more usable.  Right now you can watch as your own books fly out of the printed symbol and into the world around you, but in order to interact with the book (i.e. turn the page, make a character speak) you still need to use the mouse.

ZooBurst augmented reality mode

I’ve toyed around with a number of potential interaction methods, including using the rotation of the marker to control various elements, but I’m not 100% happy with them just yet.  I want to try and preserve the feeling of a real book as much as possible, and creating additional mappings that don’t necessarily apply to a physical book is something that I’d like to try and avoid.

One experiment I’ve been playing around with is the idea of using physical gestures to actually turn the pages of the book.  I whipped up a quick 2D demo of this in action – basically I use motion detection in very small regions of the screen to create a matrix of “touchable” sensors.  From there I can create a simple algorithm to interpret gestures based on the order in which these regions are activated.  Here’s an example that allows you to simulate a coverflow-style interface with a series of images.  I’m hoping to adapt this functionality into the 3D world of ZooBurst by the next release of its AR viewer component.

2D coverflow using a webcam

ZooBurst v1.0 was launched last week, and over the past 5 days over 60 people have signed up to start building their own 3D Pop-Up books.  It’s amazing to see what kinds of books people are creating – as of now there are over 44 user-created books, including an instructional story on how kids can stay safe on the Internet, personal narratives and even a 3D Mother’s Day card!

The ZooBurst community has also been providing me with tons of feedback on how I can make the tool more useful, and some of the top requests so far have to do with customization of materials and animation.  I figured that I would start my experimentation with a few animation techniques to try and make the ZooBurst world feel a little more “alive.”  I began by taking a look at the various book backgrounds to see how I could enhance the user experience a bit.

ZooBurst background

Skyboxes

ZooBurst backgrounds are rendered using a technique called a “skybox”.  A skybox is a trick used by 3D artists and developers to give a 3D space a sense of “depth” without having to add large numbers of 3D assets to a given scene.  Skyboxes attempt to simulate far-off details by “trapping” the user’s camera within a large cube.  Once inside, the interior faces of the cube are tiled with a series of images that are designed to give the illusion of depth, as can be seen in the following diagram:

Skybox setup

With a little tweaking you can easily create your own skybox textures out of a regular flat image.  For example, using a PhotoShop plugin called ‘Flexify 2‘ by Flaming Pear, I was able to turn this image of a beach in Spain into a series of skybox tiles.

My favorite cove in Mallorca

The same cove in Mallorca turned into a foldable "cube"

It looks pretty cool to stand inside of a skybox made using this technique for about 3 seconds.  After that you start to realize that the world you are standing in is completely devoid of motion, which is exactly what I was trying to avoid when having visitors come inside of a ZooBurst book.

After playing around with static images I realized that I really wanted to do was to create a series of animated skybox textures.  These textures could then be scripted so that I could cause the background to reform itself on a continuous basis to breathe a little more life to a ZooBurst book.

The Skybox Experiments

In theory this seemed simple enough.  Flash is great at handling all sorts of animation, and Papervision 3D has the ability to take an animated Flash movieclip and use it as a “texture” for 3D constructs (cubes, spheres, planes, etc.)

I started off by creating a very long movie clip (2400 pixels wide by 600 pixels high) that I planned to “slice” into the four side walls of my skybox.  From there I planned on scripting a series of dynamic animations that could be cut apart at run-time to provide a seamless set of textures.  Here’s my first stab at trying to get this up and running – click on the image below to launch the demo.

And here’s how I approached the problem:

1. Create a single large movieclip that contains the animation that I want to tile on the 4 side faces of a skybox.
2. Create four smaller movieclips that will each hold 25% of the total animation.
3. Use Flash’s built-in ‘bitmapData.draw()’ method to extract out the necessary visual elements and draw them into the appropriate “holder” movieclips.  In my example the source movieclip was 2400 x 600, allowing me chunk it out into blocks of 600 x 600.
4. Finally, I need to deal with the “wraparound” issue.  When an item goes off the screen in any direction it should wrap and appear on the opposite side.  This isn’t that difficult to do when working on a screen that has defined edges, but a skybox is built to give the illusion of seamlessness, meaning that when an item goes off the screen it can’t just “jump” over to the other side.  The portions of a character that are off the screen must be drawn on the other side of the skybox while the portions that remain on the screen continue to be drawn in place.  I accomplished this by using two additional calls to ‘bitmapData.draw()’ to capture objects that went out of bounds and draw them in their appropriate spots.

The first experiment didn’t turn out half bad, but next I had to figure out whether my “slice and paste” algorithm would hold up inside the a resource-intensive 3D environment.  My first stab at this yielded a mixed result.  Things were a bit slow when I applied the animated background to the skybox surrounding a ZooBurst book.

As I’ve worked with Flash I’ve found that most movies run significantly faster on a PC than a Mac.  This is a shame, since I prefer to do all my development on a Mac and want to make sure that users on any platform have a good experience with the content I create.  With that said, the performance hit for this technique is pretty high (the CPU regularly pegs at 100%), especially when viewed through Safari on a Mac (Firefox is a bit better for some reason.)  The whole simulation runs beautifully on a PC in any browser.  Go figure!

So, with that said, here is test #2, which uses the same method and texture demonstrated in test #1 and applies it to a Papervision 3D skybox surrounding a ZooBurst 3D pop-up book.

Test #2: Animated skybox background demo using ZooBurst (click image for a live demo)

You’ll also notice that the skybox is fairly pixelated (stare at one of the fish for a few seconds and you’ll see what I mean).

My next task was to try and clean the animation up as much as possible while also trying to improve rendering performance and playback speed.  Here’s what seems to have done the trick:

1. Flash does have some run-time blurring capabilities that let you smooth out jagged edges of bitmaps.  However, applying this technique to fifty-some fish while rendering them in 3D proved to be an incredible waste of resources.  Simply pre-blurring the characters in PhotoShop and importing the PNGs back into the Flash movie seemed to do the trick nicely.
2. The current version of ZooBurst only renders the 3D scene when the user decides to rotate around it.  This results in a huge savings of CPU cycles.  I didn’t want to abandon this technique entirely for this implementation, so I altered my rendering policy to do the following:
   1. Render the whole scene when the user is rotating around the book
   2. Only render the skybox and the “chat bubbles” when the book is at rest (this allows the bubbles to remain clickable while the rest of the book goes dormant).  This was accomplished using Papervision’s “renderLayers” method.
3. I reduced the number of polygons devoted to the skybox from 400 per face down to 25.  This caused some distortion that looked surprisingly like you are underwater!  However, since I was working on a system-wide solution for animated backgrounds I figured this wasn’t something I wanted to keep in the simulation.  I ended up using precise texture mapping and texture smoothing to compensate (MovieMaterial.precise and MovieMaterial.smooth)

The final iteration can be seen here.  Notice that the fish no longer just swim from one side to the other.  I coded them to behave using a “reverse flocking” algorithm that causes each fish to try and remain an optimal distance away from every other fish.

Experiment #3: Optimized animated skybox + flocking creature behavior (click image for live demo)

Next Steps

I’d love to get these new backgrounds into ZooBurst right away, but I feel that I still need to thoroughly test the implementation before releasing it into the ZooBurst environment.  As I move forward I also want to address the following issues, including:

1. Delivery.  Currently the ZooBurst skyboxes are delivered using static PNG or JPG files.  In order to fully implement an animated skybox in a non-hacky way (i.e. not hard-coding the animation routines into the ZB viewer) I need to explore some alternate delivery options.  Right now I’m leaning towards dynamically loading in animated skybox SWF files that contain the necessary simulation code and graphical elements combined into an easy to digest package.
2. Performance.  I’m not 100% sure that I want ZooBurst to consume so much CPU time.  I need to spend some time optimizing the routines to things down to a more manageable level.
3. Interactivity.  Is it possibly (or even desierable?) to make these animated backgrounds interactive in some way?  Should you be able to feed the fish, open a giant clam or dive for buried treasure?  Or does that take away from the pop-up book expereience?
4. Options.  Whatever I end up implementing needs to come delivered with the ability to gracefully turn itself off if need be.  Users should be given the choice as to whether they want to experience a book with or without animation and sounds.

Update:

The following demo shows how you can use the mouse to interact with dynamic background elements within a 3D skybox.  In this example your mouse causes bubbles to appear in the water, and the bubbles have a “lifting” effect on the crab that crawls along the floor.  I’m still not sure if this is a good idea for ZooBurst – it may end up being more of a distraction than anything else.

Interactive animated background (click for demo)