The Pixel Farm

Software Development

While working on the development of the pixel grid hardware, we were simulteanousely developing a series of software tools to that would give life to our creation as well as give us the ability to rapidly develop software for the platform that we were creating.

Serial Communication

From the beginning of the project we knew that we needed to drive the hardware device with visual data that would be generated on an external computer.  While it would be possible to write the software on the microcontroller itself, we were concerned about performance issues that may arise from having both application and display logic running on the same device.  In addition, splitting out the software development environment from the hardware microcontroller allowed us to develop application code without the need for a working hardware device at our disposal.

Our first attempt at setting up serial communication utilized a simple call/response methodology.  First, the microcontroller would contact the computer via a serial connection with a predetermined string of bytes.  The software on the computer would respond with another predetermined string of bytes, and after which the computer would begin to send display data to the microcontroller using a series of bytes.  We decided to attempt to take full advantage of the fact that the TI5940 chip supported PWM, so we started by sending a full byte of information per LED.  This would yield a maximum possiblity of 256 different value per color, giving us a whopping 16 million possible colors per pixel.

Of course this didn’t work.

We found that sending this much data (243 bytes per frame at 30 frames per second) was beyond the ability of the serial connection.  On top of this, we quickly realized that our call/response methodology was inadequate for this purpose – we needed a way to keep the display in sync with the computer in order to prevent dropped frames and ensure that the display was always in sync with the driver software.  We established a new protocol that works as follows:

• Computer sends:  ping? (later shortened to “p” to help improve the data transfer rate)
• Arduino receives:  ping?
• Arduino sends:  pong! (later shortened to “o”)
• Computer receives:  pong!
• Computer sets a flag that indicates that it’s OK to send data.  The program is allowed to caluclate the next frame of display data.
• The next frame is broken down into bits based on color values.  For our example we broke down each pixel into three bits – red (on/off), green (on/off) and blue (on/off).  These bits are packaged into bytes and are sent to the Arduino at the end of the frame calculation cycle.
• Arduino recieves:  specified number of bytes and uses a series of bitmasks to extract the color values for each of the TI5940’s output pins.  The Arduino then updates the display accordingly.
• Arduino sends:  “success” to indicate that the display has been updated
• Computer receives:  “success” and sets a flag to indicate that the microcontroller is ready for another set of bytes.  This flag permits the system to generate another frame of visual data, and the process repeats itself.

This process worked great, and kept the display in sync with the computer.  Once the basis of this system was in place we went ahead and tweaked the serial baud rate to increase performance, and at 56kbps we were able to achieve almost 30 frames per second of video on our display.

Grid API

In order to extend the functionality of the Wand Pong project, we wrote a grid API object that provides a standard means of interacting with our hardware device.  Written in Processing, this API object provides a basic toolset for the setting, clearing and loading of pixel data between a sketch and the display.  The API is fully integreated with the call and response serial protocol outlined above, which allows the end programmer to focus on creating applications rather than worrying about the mechanics of how to send visual output to the display.

Human Interface

Now that we had the ability to drive the display and had a standard framework in place for software development we were able to focus on human interaction.  We toyed around with a few ideas, but the standard ones that came to mind (buttons, force sensors, IR beams, etc) seemed to lack something.  We wanted to try and capture the “fun” of multicolored ping pong balls, so Bryan took the lead and developed two wands that light up using green and blue LEDs.

Unlit wands

Let there be light!

We borrowed some previous work we made on the augmented reality display to track these colors using a live video feed.  The goal was to use these wands as a simple way to physically interface with the display – a webcam would be mounted on the top of the display which could determine if a wand was in it’s field of view.  If it was, we would enter into an interactive mode of play, namely the classic game of “Pong!”

Vision

We ran into a few challenges during our initial exploration into the development of an augmented reality device.  The most sigifncant of these challenges was the ability to reliably track an object using a live video feed.  Unfortunately we were unable to surmount this issue for that project, but we did find that one of our ideas – diffusing visible light using a ping pong ball – resulted in a really captivating display, especially when implemented using a multi-colored LED.  We decided to explore this concept further and attempt to develop a matrix of ping pong balls that were illuminated with LEDs – the eventual goal of this matrix would be to create a fully functional display unit that could be hung on a wall and dazzle passers-by with a variety of visualizations, simulations and games.

Obtaining Hardware Components

The first major challenge we encountered was figuring out what kinds of hardware components we would need to implement the ping pong ball display.  First and foremost we needed to figure out a way to address and power a large number of LEDs.  The Arduino microcontroller only has a limited number of outputs, and if we wanted to create a standard size display (say, 10×10 balls) that would mean we would need 100 output ports – much more than we could get from what was delivered on the Arduino itself.  To make matters more complicated, we wanted our display to be comprised of multi-colored LEDs, which have a separate pin for each color (Red, Green & Blue).  This means that we would need three time the number of outputs, or 300 independently addressible pins.

We brought up this issue to Tom, and he encouraged us to explore the TI5940 chip, which is a 16 channel pulse width modulation (PWM) LED driver.  There is a fairly simple library written for the chip, and it can easily be attached to an Arduino by using only 5 digital output pins.  In addition, the chip supports “daisy chaining,” meaning that we had the ability to string multiple chips into a long line (online resources claim that up to 40 chips can be daisy chained without signal degradation).  The chip also has the ability to generate PWM output signals, meaning that we could send varying levels of electriciy to an individual LED.  This meant that we could, theoretically, perform color “mixing” in our software to create a wider range of colors than just red, green and blue.

Tom loaned us a single TI5940 and we were able to construct a basic program that allowed us to address the 16 individual PWM outputs without too much trouble.  Inspired by our success, we ordered 25 chips (enough to address 133 individual RGB LEDs, or an 11×11 grid.

A single TI5940 implementation using standard LEDs

We had already ordered 100 RGB LEDs (which we had gotten at a great price) but we found that they would not be compatible with the TI5940.  The LEDs that we had on hand were common cathode LEDs, meaning that the lights contained a common ground and three power leads (one for each color).  The TI5940, however, is a “sink” style chip, meaning that it is designed to address an LED by allowing electricity from another source to flow through and ground itself via the chip.  Given that our chips had only a single ground wire there would be no way for us to address an isolated color in a given LED.  We needed to obtain a reversed version of this setup, known as a common anode LED.  As soon as we discovered the issue we put in an order for 125 units immediately.  Unfortunately this delay set us back by about a week – while we could test the system with standard LEDs, we weren’t able to explore the nuances of using super bright RGB LEDs in an actual application.

Assembly

Given this constraint we forged ahead in two directions simultaneously.  First, we worked to chain together the TI5940 chips and test their operation using single colored LEDs.  At the same time we began to develop software routines that would eventually serve as the backbone for interfacing with our device, as well as a standard framework API which could be used to facilitate the development of applications for the display.  We also drilled holes in 100 ping pong balls and selected some foam backing to serve as the installation “frame” for the display.

We started by attempting to chain together two TI5940’s into a single unit.   The process was fairly straightforward – the second chip needs to be set up in the standard way, including power, ground and the necessary resistors.  All signal wires need to be chained from the first chip onto the second chip, with the notable exception of the “Serial Out” line which chains into the “Serial In” pin on the second board.  Next, the tlc_config.h file needs to be updated to reference the number of chips that will be chained together.  One very important part of this step is to delete the Tlc5940.o file after making this change – otherwise you will recompile your Arduino code using the previous number of chips and will be setting yourself up for heartache and angst

Two chips worked without too much trouble once we realized the Tlc5940.o issue, and we expanded our efforts to include 4 chips.  We found that after about 4 chips we began to see “flickering” in chips near the end of the chain.  We initially thought that this was due to signal degradation, but we eventually found out that we were having power issues (we were still powering the array off of the Arduino’s 5 volt power supply).  After some rewiring, we hooked up an external power supply (a varying power module that we dubbed the “big box of death”) and were able to reduce flicker to a minimum in the later chips.

16 TI5940s daisy chained on breadboards

The big box of death

Two LEDs soldered and color coded

Interior of frame

all the LEDs plugged in and ready to go

All of the LEDs arranged in order

Get the latest Flash Player to see this player.

Identified tasks (hardware)

• obtain foam backing
• prepare enclosure
• obtain more ping pong balls
• drill LED access holes in ping pong balls
• experiment and determine wiring solution (soldering, using connectors) – experimentation completed, working on viable large-scale implementation
• obtain 100 common anode multi colored LEDs (ordered, currently being shipped)
• obtain 25 TI5940 LED driver chips (ordered, currently being shipped)
• experiment with chaining multiple TI5940 chips in series (will be possible once chips have arrived)
• experiment with various input devices (paddles, video camera input, etc)
• implement a reliable powering solution for 100 LEDs

Identified tasks (software)

• establish serial communication with Processing (call & response)
• establish driver routines on the Arduino for working with multiple TI5940 chips
• design software API in Processing for working with pixel matrix)
• develop software suite to include
  • pixel mirror (screensaver)
  • pong
  • ripple visualization
  • text display tool
  • painting game
• package software suite into easy to use distribution

Aly Wolff-Mills, Bryan Lence & I decided to work as a team on our final Physcomp & ICM projects. Our overarching idea was to create an augmented reality system that would create a three dimensional overlay onto a live video stream.

Our prototype system contained a digital camera, Nintendo Wiimote, Macbook Pro Laptop and a set of Myvu LCD goggles.  A visual schematic of the system can be seen in the flash movie below:

We began by experimenting with 3D graphics in Processing, which was surprising simple.  We then overlaid an arbitrary set of graphics on top of an incoming video stream and found that the video performance remained high.

Basic 3D overlay on live video

Next we needed a way to reliably overlay the 3D image on top of a fixed point that was visible in the incoming video stream.  We also needed a way to accurately gauge the distance of the fixed point from the observer, which would allow us to scale the 3D objects and keep them in proportion to the 2D landscape.  We decided to try using three recognizable “spots” on a flat surface of a known size in order to do this.  Initially these spots would be visual (color) cues that the digital camera would pick up, but after a bit of research we decided to try to use Infrared light instead, as it was less visibly intrusive and could easily be detected by the Nintendo Wiimote, plus we could avoid detection collisions with similar colors.

In order to test IR light detection, we built an IR debugging program in Processing that visualized visible IR spots using colored ellipses.  We used the Darwin Remote bluetooth server (see my Wiimote Painting project for more information) and the system performed reasonably well.  As a side note, the Wiimote can only see a maximum of 4 IR spots at any one time, but this wasn’t a limiting factor for this project as we would only need three spots to accurately triangulate the surface of our table.

We then drilled three holes in a box and inserted 3 LEDs in a triangular pattern (see the Flash movie above) and started streaming live video into a new Processing sketch.  We used the positions of the LEDs to determine the location of the box and could then render a 3D cube on top of this space.  The two spots that were closest together represented the center of the box, and the outlying spot represented the edge of the box.  Using a little trigonometry we could determine the Y axis rotation of the box based on the incoming orientation of the box.  Here’s a quick example of the tracking we were able to get out of the system using this process (click the triangle to start the video):

Unfortunately we found that IR LEDs have a fairly limited range and that we would lose visual contact with them after about 5 feet or so.  We also found that the visible angle put off by the IR LEDs was fairly limited (about 45 degrees or so) which meant that you could only track them reliably by looking down at the LEDs.  Because of this we attempted to diffuse the light in a number of ways, including putting the LEDs inside of semi-opaque plastic cups and ping pong balls.  This yielded a slightly better range, but nothing substantial enough to give us the range we needed for a project such as this.  We also attempted to place multiple IR LEDs inside a ping pong ball in hopes of trying to trick the Wiimote into thinking it was seeing a single LED, but we were unsuccessful in this as well.

Finally we tried placing visible light LEDs inside the ping pong balls and attempted to track them via the digital camera.  We were able to do this fairly well, but we ran into a number of issues, including the matching objects that were in the field of vision that were not the intended targets.

Given these challenges we felt that we needed to abandon this as our final project as the performance didn’t live up to our expectations.  We did find, however, that the visible light LEDs embedded in the ping pong balls produced a great effect that we wanted to explore in further detail.  We brainstormed for a bit and decided to shift gears and attempt to build a 2D grid of LED illuminated ping pong balls that could act as a display device.  More information on this project is forthcoming in a separate blog posting!

Connecting a DC motor to a potentiometer with a diode and transistor.
Following the lab we connected, the DC motor to a TIP120 transistor and a 1N4001 diode.
The potentiometer values determine the speed of the DC motor. Turning the knob makes it go faster or slower–all the way to a stop.

The only change in the lab code was to add a Serial.prinln to monitor the value of the potentiometer variable.

For the second part of the lab, see Aly’s blog.