Just a quick test of a light-sensitive Augmented Reality environment that will soon become inhabited by a colony of virtual snails! Different colored lights will affect the environment in different ways, changing the system’s color balance and thus making certain snails easier to detect by predators.
In this demo you can see a blue light making small forests grow up out of the ground. Eventually this behavior will be triggered by a green light [I accidentally left all of my LEDs in NYC this weekend! ]
Scientific Concept
My selected area of science is focused on fundamentals of evolutionary biology. More specifically, I am focusing my attention on natural and artificial selection pressures and how they can affect a population of organisms.
Setting for Project
I imagine that this project could be used in a science classroom that is outfitted with a computer and LCD projector unit. I would also like to try it out in a more public setting, such as the Spring ITP show.
Audience I envision this project being used by students aged 15-22 who are studying the fundamentals of evolutionary biology. In addition, the project is targeted at biology teachers and professors as a supplemental exercise that can be used when talking about natural selection.
Proposed Forms
I am currently debating two forms of delivery. My preference is heavily weighted in the favor of a “true” augmented reality experience, but if I run into technical limitations I may modify the project outlined below to work without a head-mounted display.
Update 4/15/2009: The head mounted display is definitely out of the question. The prototype I constructed made me nauseous almost instantly! The current format I will be working with will only include a fixed camera and a projector. This will allow visitors the ability to “walk up” to the scene rather than have it thurst upon them via a head mounted dipslay.
Augmented Reality Imagine being able to watch as your local environment is transformed into an epic evolutionary struggle between predator and prey! In this project I am proposing to create just such a system that would allow a visitor to see and experiment with the concept of natural selection from the comfort of his or her own home.
The more do-able (and less nauseating!) version of the project will work as follows:
Visitors will be able to walk up to large projection screen or LCD monitor in order to interact with the display. A fixed camera will be placed in front of the monitor to capture a video feed of the visitors as they look at the display.
The video feed from the camera will be transferred to a computer where it will be “augmented” with a 3D scene based upon the placement of a series of visual markers. The feed will then be transferred back to the large projection / LCD display.
The markers will define various 3D “anchor points” of a fictitious 3D environment that is inhabited by small multi-colored organisms (snails). The visitor will watch as sight-hunting predators enter the environment and feast upon the easily detectable snails.
Evolutionary theory will come into play once organisms mature to the point where they can reproduce, and organisms that successfully survive to this stage will “breed” and cross their genetic material with other surviving organisms. The offspring will contain similar color variations to their parents, and the process will continue.
Status updates will be presented in the form of 3D graphs scattered around the edge of the environment. I envision having a dynamically updatable color distribution chart along with information about the surviving organisms.
I’m still working on ways to incorporate the visitor into the experience and give them more of an active role in the simulation. Below is a quick video that outlines the progress of the system as of 4/15/2009. The system currently includes the following:
Dynamically generated environments. While the demo only shows a green landscape, the environment can be configured to use the entire range of red / green / blue color values. This will become important if the system is upgraded to support for changing seasons and external environmental factors, such as global warming and rising sea levels.
Autonomous snails “live” on the environment. Their life is consists of:
Birth: Snails begin as microscopic eggs that are barely visible upon hatching.
Adolescence: Snails grow until a predetermined age, at which time they are considered “mature.” Snails in the adolescent stage cannot breed.
Mature: Snails who have reached their full size can begin to breed. Snails currently have a 10% chance per “cycle” to enter into a breeding mode.
Breeding: A small cube with a heart on it signifies that a snail is ready to breed. Snails in the wild take part in both hermaphroditic and sexual reproduction, but the simulation shown below only contains hermaphroditic reproduction – offspring are clones of their parents. Future plans include the implementation of a sexual reproduction routine as well as a “mutation” factor that affects offspring so they are not absolute carbon copies of their parents.
Death: The lifecycle of a snail ends after a period of time. This is simulated by the snail being absorbed by the ground.
The originally proposed system, which includes the idea for a head-mounted AR display, would have worked as follows:
Visitors will be able to don a specially created headpiece that will function as wearable display. A small camera will be mounted on top of the piece to capture a video feed from directly in front of the person.
The video feed from the camera will be transferred to a computer where it will be “augmented” with a 3D scene based upon the placement of a visual marker on a table. The feed will then be transferred back to the headpiece to allow the visitor to see the entire scene using a miniature “pico” projector embedded in the headpiece.
The marker will define the center point of a fictious 3D environment that is inhabited by small multi-colored organisms. The visitor will play the role of the “predator” in the scene, and will use a device (probably a red fly swatter) to eliminate as many organisms as possible.
Evolutionary theory will come into play once organisms “escape” from the edge of the environment. Escaping signifies that the organism has matured to the point where it can reproduce, and organisms that successfully survive will “breed” and cross their genetic material with other surviving organisms. The offspring will contain similar color variations to their parents, and the process will continue. The hope is that the player will “attack” the most easily visible organisms, thus pressuring the virtual colony to evolve towards being less visible.
Status updates will be presented in the form of 3D graphs scattered around the edge of the environment. I envision having a dynamically updatable color distribution chart along with information about the surviving organisms.
The entire scene will also be visible to passers-by through a live video feed of the player’s perspective being projected on the wall.
John Kuiphoff and I worked together this week to create a media enhancement for the “Decide” board game that we played in our Art/Science Data Collisions class a few weeks ago. Our redesign focused on two specific aspects of the experience:
Expanding the amount of technical information that is available to players during the game
Creating a way to keep track of discussion points associated with a specific card or issue
To address these issues we created a web-based application that does the following:
Players can print out “augmented” versions of the standard Decide playing cards. Our version of the cards contain a special symbol which can be used to interface with the application.
Next, a player can open up their browser and visit the application homepage. After loading the player can hold up a card to their webcam – the site will recognize the augmented symbol and will display card-specific information, including relevant audio and video clips.
The system is also set up with a commenting field – this field can be used to record important notes that come up during the discussion portion of the game. The system is set up to remember these comments on a card-by-card basis, so visitors can switch back and forth between cards without losing any information.
Screenshot of the Decide web interface
We have a live demo version available – here’s how you can try it:
You may need to grant the application access to your webcam. Note that if you’re on a Mac you may also need to right-click on the movie and select “Settings -> Webcam -> USB Class Video”.
Hold up a card to the screen – information about the card should appear along with a video and a comment box.
Comments are “sticky” and will persist even if you hold up another card to your webcam.
One of the major items that struck me as I read through the evolutionary biology literature was the fact that natural selection (or more specifically in the case of the overfishing example, directional selection) can cause changes in an observable phenotype that can become apparent within only a few generations. I was always under the impression that evolutionary processes operated over a very long period of time and were more or less unobservable at the micro (generational) level. However, it seems that my initial assumptions were incorrect, and the fact that a scientist was able to design an experiment that could show the effects of directional selection as well as the corresponding rebounding of a population within just a few generations is amazing.
The next thing I would like to explore is how these processes play out over longer periods of time. The experiment I covered in class lasted 10 generations, and I wonder whether there would be statistically significant differences in the various populations after, say, 100 generations of “recovery.” This would lend more credence to the theory that we could reverse the damage we have done by overfishing large fish populations if we were to stop and give these populations in the wild some time to rebound.
Exploring this experiment as part of the Science Salon helped to prepare me for my initial meeting with my scientific expert. During this meeting we determined that our mutual interest was in finding ways to more effectively convey difficult scientific concepts to students as well as find ways to make Biology more engaging and appealing. Our initial plan is to develop a simulation that uses Augmented Reality techniques to create an immersive environment that can be used as a teaching tool.
Our initial game plan for developing such a system is as follows:
Develop a 2D simulator that models the concept of directional selection on a simple population. For the purposes of this project we have selected the somewhat famous case of the “Cepaea nemoralis” snail and its predator, the Song Thrush. (http://www.weichtiere.at/Mollusks/Schnecken/drossel.html). The simulator will have the following characteristics:
An environment made up of a heterogeneous background color will be established for our snails.
Snails of various colors will be introduced and will be able to move around this environment.
Song Thrush predators will be established and will keep track of how “hungry” they are. When they reach a certain hunger level they will enter the environment and select the “easiest to spot” snail within a small region. The ease of detecting a snail is computed by comparing the snail’s coloring to the coloring of the background.
Snails that are eaten are removed from the simulation.
After a predetermined time the remaining snails enter a breeding period. Offspring will have coloring that is computed based on the coloring of their parents. Cepaea nemoralis is a hermaphroditic species, so two mating snails act as both mother and father for the next generation.
Statistics are gathered throughout the simulation, including the frequency and color of snails eaten as well as the coloring and frequency of the surviving snails.
Once the simulator behaves in a stable and reliable manner, we will work to convert it into a 3D augmented reality installation. The AR display will (probably) work as follows:
A static camera will focus on a tabletop that contains an AR marker tile.
The marker will serve as the center of the environment and will be detected by an AR computer vision algorithm.
The augmented video stream will be projected on a screen behind the table. This will allow passers by to see themselves standing in front of a table that is teeming with 3D snails and birds locked in an epic evolutionary struggle!
A series of additional AR tags will be available to allow visitors to interact with the display. These can include tags that can be held up to display:
Current statistics on the color values & frequency of eaten snails
Current statistics on the color values & frequency of surviving snails
Possibly allow visitors to “drop” new snails into the simulation or to start off the simulation with customized starting parameters
]
