The Pixel Farm

T-minus 9 days until the premier educational technology event of the year – the annual New Media Consortium Summer Conference! This year’s program is looking better than ever with sessions and workshops focusing on a wide range of topics, such as educational gaming, virtual worlds, mobile applications, geolocation technologies, cloud-based learning systems, digital storytelling techniques and yes, of course, Augmented Reality!

NMC 2010 Summer Conference

This year I will be presenting a half-day hands-on session in which we will explore emerging educational applications of augmented reality as well as introducing ways in which educators can build their own AR environments.  I’m busily preparing a number of fun tutorials for this event, including:

• Hands-on activities that will let participants experience augmented reality first-hand (printed markers included!)
• A series of “quick start” code libraries that are designed to help educators construct their own AR learning applications, and step-by-step instructions on how to get up and running fast.  All participants will leave the session with their own working AR project on their blog or website.
• A preview of the new version of the ZooBurst digital storytelling tool which includes a number of new features, including live audio recording capabilities, classroom management features and even an augmented reality presentation mode.

If you’ll be in Anaheim CA next week and would like to join us feel free to register!

ZooBurst has been getting tons of use over the last few weeks, and I’ve been spending most of my free time making updates to the system and thinking of ways to move forward with the environment. It’s really amazing how fast it’s taken off – as of today we have over 1,000 people actively using the system to create their own 3D pop-up books!

There are a ton of features on my “to-do” list for ZooBurst, but one of the most pressing is to find a way to make the augmented reality feature more usable.  Right now you can watch as your own books fly out of the printed symbol and into the world around you, but in order to interact with the book (i.e. turn the page, make a character speak) you still need to use the mouse.

ZooBurst augmented reality mode

I’ve toyed around with a number of potential interaction methods, including using the rotation of the marker to control various elements, but I’m not 100% happy with them just yet.  I want to try and preserve the feeling of a real book as much as possible, and creating additional mappings that don’t necessarily apply to a physical book is something that I’d like to try and avoid.

One experiment I’ve been playing around with is the idea of using physical gestures to actually turn the pages of the book.  I whipped up a quick 2D demo of this in action – basically I use motion detection in very small regions of the screen to create a matrix of “touchable” sensors.  From there I can create a simple algorithm to interpret gestures based on the order in which these regions are activated.  Here’s an example that allows you to simulate a coverflow-style interface with a series of images.  I’m hoping to adapt this functionality into the 3D world of ZooBurst by the next release of its AR viewer component.

2D coverflow using a webcam

It was only a matter of time.  My poor iPhone was mortally wounded back in January at the Lambertville Beer and Chili Festival.  While it continued to function like a trooper, I figured it was about time to let it move on to greener pastures.  Plus I didn’t want to have to continue to deal with all the pitiful (disgusted?) looks that I was getting from people on the subway whenever I had my phone out.

cracked screen = no fun

I decided to switch it up a bit and see what life is like in Android land, and I’ve got to say I’m really enjoying it so far.  I ended up getting the new HTC Incredible which sports a 1ghz Snapdragon processor, 512mb of ram and an amazingly crisp 8 megapixel camera.  The difference between browsing speed on AT&T versus Verizon is unbelievable, especially in NYC.  Pages that would chug on my iPhone load quickly on the Incredible.  In addition, I found an amazingly helpful application (’The Missing Sync‘) that lets me wirelessly sync all of my desktop content – including my music – to my phone whenever the two devices are on the same wireless network.  No more USB syncing with iTunes – yay

The built in camera takes great photos and sports a software based digital zoom as well as a “tap to focus” feature that lets you isolate subjects in your shot easily.  Here’s a scaled down image of my pup, Prue, that I took using the built-in cam the other night.

my pup, prue

One of my favorite features of the phone is that it runs Flash Lite.  ZooBurst even loads on the phone, which really surprised me since I use a ton of external 3D libraries that I thought wouldn’t be compatible.  With that said, let me say that I’m as excited as everyone else to see what HTML 5 will eventually bring to the table, but right now there are things that Flash can do on the web easily that just aren’t possible with other languages (including all of my augmented reality experiments).  I’m really psyched about the full version of Flash mobile that is supposedly coming out this summer.  I wonder how easy it will be to port some of my AR demos to run on Android … I’ll keep you all posted!

ZooBurst v1.0 was launched last week, and over the past 5 days over 60 people have signed up to start building their own 3D Pop-Up books.  It’s amazing to see what kinds of books people are creating – as of now there are over 44 user-created books, including an instructional story on how kids can stay safe on the Internet, personal narratives and even a 3D Mother’s Day card!

The ZooBurst community has also been providing me with tons of feedback on how I can make the tool more useful, and some of the top requests so far have to do with customization of materials and animation.  I figured that I would start my experimentation with a few animation techniques to try and make the ZooBurst world feel a little more “alive.”  I began by taking a look at the various book backgrounds to see how I could enhance the user experience a bit.

ZooBurst background

Skyboxes

ZooBurst backgrounds are rendered using a technique called a “skybox”.  A skybox is a trick used by 3D artists and developers to give a 3D space a sense of “depth” without having to add large numbers of 3D assets to a given scene.  Skyboxes attempt to simulate far-off details by “trapping” the user’s camera within a large cube.  Once inside, the interior faces of the cube are tiled with a series of images that are designed to give the illusion of depth, as can be seen in the following diagram:

Skybox setup

With a little tweaking you can easily create your own skybox textures out of a regular flat image.  For example, using a PhotoShop plugin called ‘Flexify 2‘ by Flaming Pear, I was able to turn this image of a beach in Spain into a series of skybox tiles.

My favorite cove in Mallorca

The same cove in Mallorca turned into a foldable "cube"

It looks pretty cool to stand inside of a skybox made using this technique for about 3 seconds.  After that you start to realize that the world you are standing in is completely devoid of motion, which is exactly what I was trying to avoid when having visitors come inside of a ZooBurst book.

After playing around with static images I realized that I really wanted to do was to create a series of animated skybox textures.  These textures could then be scripted so that I could cause the background to reform itself on a continuous basis to breathe a little more life to a ZooBurst book.

The Skybox Experiments

In theory this seemed simple enough.  Flash is great at handling all sorts of animation, and Papervision 3D has the ability to take an animated Flash movieclip and use it as a “texture” for 3D constructs (cubes, spheres, planes, etc.)

I started off by creating a very long movie clip (2400 pixels wide by 600 pixels high) that I planned to “slice” into the four side walls of my skybox.  From there I planned on scripting a series of dynamic animations that could be cut apart at run-time to provide a seamless set of textures.  Here’s my first stab at trying to get this up and running – click on the image below to launch the demo.

And here’s how I approached the problem:

1. Create a single large movieclip that contains the animation that I want to tile on the 4 side faces of a skybox.
2. Create four smaller movieclips that will each hold 25% of the total animation.
3. Use Flash’s built-in ‘bitmapData.draw()’ method to extract out the necessary visual elements and draw them into the appropriate “holder” movieclips.  In my example the source movieclip was 2400 x 600, allowing me chunk it out into blocks of 600 x 600.
4. Finally, I need to deal with the “wraparound” issue.  When an item goes off the screen in any direction it should wrap and appear on the opposite side.  This isn’t that difficult to do when working on a screen that has defined edges, but a skybox is built to give the illusion of seamlessness, meaning that when an item goes off the screen it can’t just “jump” over to the other side.  The portions of a character that are off the screen must be drawn on the other side of the skybox while the portions that remain on the screen continue to be drawn in place.  I accomplished this by using two additional calls to ‘bitmapData.draw()’ to capture objects that went out of bounds and draw them in their appropriate spots.

The first experiment didn’t turn out half bad, but next I had to figure out whether my “slice and paste” algorithm would hold up inside the a resource-intensive 3D environment.  My first stab at this yielded a mixed result.  Things were a bit slow when I applied the animated background to the skybox surrounding a ZooBurst book.

As I’ve worked with Flash I’ve found that most movies run significantly faster on a PC than a Mac.  This is a shame, since I prefer to do all my development on a Mac and want to make sure that users on any platform have a good experience with the content I create.  With that said, the performance hit for this technique is pretty high (the CPU regularly pegs at 100%), especially when viewed through Safari on a Mac (Firefox is a bit better for some reason.)  The whole simulation runs beautifully on a PC in any browser.  Go figure!

So, with that said, here is test #2, which uses the same method and texture demonstrated in test #1 and applies it to a Papervision 3D skybox surrounding a ZooBurst 3D pop-up book.

Test #2: Animated skybox background demo using ZooBurst (click image for a live demo)

You’ll also notice that the skybox is fairly pixelated (stare at one of the fish for a few seconds and you’ll see what I mean).

My next task was to try and clean the animation up as much as possible while also trying to improve rendering performance and playback speed.  Here’s what seems to have done the trick:

1. Flash does have some run-time blurring capabilities that let you smooth out jagged edges of bitmaps.  However, applying this technique to fifty-some fish while rendering them in 3D proved to be an incredible waste of resources.  Simply pre-blurring the characters in PhotoShop and importing the PNGs back into the Flash movie seemed to do the trick nicely.
2. The current version of ZooBurst only renders the 3D scene when the user decides to rotate around it.  This results in a huge savings of CPU cycles.  I didn’t want to abandon this technique entirely for this implementation, so I altered my rendering policy to do the following:
   1. Render the whole scene when the user is rotating around the book
   2. Only render the skybox and the “chat bubbles” when the book is at rest (this allows the bubbles to remain clickable while the rest of the book goes dormant).  This was accomplished using Papervision’s “renderLayers” method.
3. I reduced the number of polygons devoted to the skybox from 400 per face down to 25.  This caused some distortion that looked surprisingly like you are underwater!  However, since I was working on a system-wide solution for animated backgrounds I figured this wasn’t something I wanted to keep in the simulation.  I ended up using precise texture mapping and texture smoothing to compensate (MovieMaterial.precise and MovieMaterial.smooth)

The final iteration can be seen here.  Notice that the fish no longer just swim from one side to the other.  I coded them to behave using a “reverse flocking” algorithm that causes each fish to try and remain an optimal distance away from every other fish.

Experiment #3: Optimized animated skybox + flocking creature behavior (click image for live demo)

Next Steps

I’d love to get these new backgrounds into ZooBurst right away, but I feel that I still need to thoroughly test the implementation before releasing it into the ZooBurst environment.  As I move forward I also want to address the following issues, including:

1. Delivery.  Currently the ZooBurst skyboxes are delivered using static PNG or JPG files.  In order to fully implement an animated skybox in a non-hacky way (i.e. not hard-coding the animation routines into the ZB viewer) I need to explore some alternate delivery options.  Right now I’m leaning towards dynamically loading in animated skybox SWF files that contain the necessary simulation code and graphical elements combined into an easy to digest package.
2. Performance.  I’m not 100% sure that I want ZooBurst to consume so much CPU time.  I need to spend some time optimizing the routines to things down to a more manageable level.
3. Interactivity.  Is it possibly (or even desierable?) to make these animated backgrounds interactive in some way?  Should you be able to feed the fish, open a giant clam or dive for buried treasure?  Or does that take away from the pop-up book expereience?
4. Options.  Whatever I end up implementing needs to come delivered with the ability to gracefully turn itself off if need be.  Users should be given the choice as to whether they want to experience a book with or without animation and sounds.

Update:

The following demo shows how you can use the mouse to interact with dynamic background elements within a 3D skybox.  In this example your mouse causes bubbles to appear in the water, and the bubbles have a “lifting” effect on the crab that crawls along the floor.  I’m still not sure if this is a good idea for ZooBurst – it may end up being more of a distraction than anything else.

Interactive animated background (click for demo)

ZooBurst is a web-based digital storytelling tool that is designed to let anyone easily create their own customized 3D pop-up books.   As of today I am pleased to report that version 1.0 of ZooBurst is up and running, and is accessible to the world at http://alpha.zooburst.com.

Here’s an example ZooBurst book (requires the Adobe Flash player plugin – sorry iPad/Pod/Phone users!)

… and here are a few screen shots for those of you who are viewing this post via an aggregator:

Zooburst 3D Pop-Up Book

Zooburst 3D Pop-Up Book - Augmented Reality

The current version of ZooBurst supports the following features:

• Web-based access to user created books
• Embeddable content that allows a ZooBurst book to exist on any website that accepts HTML embed code
• Augmented Reality mode that lets you experience a book in the palm of your hand (requires a webcam)
• Web-based creation of new books
• Integration with the Open Clip Art Library of public-domain vector images
• Local file upload feature to allow you to use your own artwork within a book

Over the next week I will be contacting individuals who have signed up for the public beta version of ZooBurst with an invitation code.  We are still accepting new users on a rolling basis – to sign up for an account please visit this form and tell us a little more about yourself and why you would like to give ZooBurst a try.