A scene rendered by ZeroRay illustrating reflection and refraction. The glass orbs both reflect and transmit light through them.

ZeroRay began its life as a ray tracer. Building a ray tracer from scratch was one of the large projects for a graduate level computer science course I took called Image Computation. I ended up using ZeroRay in a lot of other projects and it has evolved into a ray tracer/computer vision (CV) toolkit with a suite of deep applications that are both interesting, and illustrate how to use ZeroRay.

I have released ZeroRay as open source software, under the BSD license. The BSD License is pretty much the most liberal open source license. The basic jist is that you can use the code for anything you want, be it open or closed source, commercial or non-commercial. All you have to do is give the author credit for using their library. You can visit the ZeroRay project page at sourceforge. At the time of this writing, I haven’t prepared any convenient downloadable packages so you will have to check the code out of the svn repository. To do so click Develop on the project page and follow the instructions there.

If you’re wondering, “What’s a Ray Tracer?” check out my previous article What is Ray Tracing?

Back to what exactly ZeroRay is. ZeroRay is a software library (or more precisely a set of software libraries). That means it is a set of tools that are used by programmers to create applications. A simple application could be one that, when executed, draws a pre-set picture and saves it to an image file. A more complex application could be a program that allows you to set up a 3D scene by dragging objects into it a low quality rendering of the scene and then invokes the ZeroRay ray tracer to generate a high quality version of the image. As a bit of an aside: ray tracing is a way to generate very high quality computer graphics, however it is significantly more computationally intensive than the techniques used in real-time renderers, like the sort that are used in video games. To make the ZeroRay ray tracer widely useful to computer artists (who generally are not programmers), it would need an application similar to the second example. But ZeroRay is more than a ray tracer.

More Than a Ray Tracer

There are a number of high-quality, mature ray tracers in the world. Why build another? Well, as I mentioned, I started it as a project for a class, but what ZeroRay really offers is its combination of 3D rendering and a computer vision (CV) toolkit. Both libraries are a part of ZeroRay and use the same set of foundational classes, vectors, matrices, images, etc. The image class is full-featured and implements a number of useful algorithms as well as simple tools like annotating or marking images as is often needed to test CV algorithms. The CV toolkit also implements convolution/correlation providing both brute force algorithms and Fast Fourier Transform (FFT) based algorithms built with FFTW.

ZeroRay is not nearly as fully-featured as OpenCV, the current go-to CV toolkit, but ZeroRay provides a well-designed, modern, object-oriented foundation for CV. It contains all the building blocks that a student of CV would have to spend several months writing before they started their work.

A ZeroRay Rendering of glass spheres lit by various coloured light sources.

The Image class is especially useful. It is implemented in a speed-optimized but still relatively straightforward manner. Users can load an image and begin working with pixel data directly without having to read a chapter in the manual first (as one might to begin working with OpenCV images). ZeroRay reads and saves in png format. In memory, the Image class (zr::Image) can represent many different types of pixel data, greyscale, RGB, RGBA, all in discrete (4 bits per channel) or floating point (32 or 64 bits per channel) format. Floating point images are important because many CV algorithms and transformations need to be performed with much more numerical precision than images are usually stored in. Without this extra precision iterated calculations would result in very large rounding errors as results are rounded off at each step of the way.

Applicatons Module

The applications module contains a set of example programs that use either the zeroray (ray tracing) or zrvision (CV) libraries. However, these aren’t just simple example programs, each one is a project I completed for my CV coursework. They’re pretty cool and show what can be done with ZeroRay.

zrExample – This is the ray tracer demo. It has a handful of scenes (defined in code) that it can render either to the screen or to a file, at arbitrarily large resolutions. This is the place to look to learn how to set up and render a scene with the ray tracer. It is easy to modify one of the pre-defined scenes, re-compile and render the new scene.

AeroQuest – The task was to take a shaked, hand-held, video of a plane landing and stabilise the picture so that the plane stays in the center by using correlation. The end result is a command line executable that takes in a set of images, a template bounding box in the first frame, and outputs a set of images with the template held in the center. Or at least it tries to. The technique fails if the object in the template changes size or shape dramatically (this is an expected shortcoming).

WineAndCheese – A classifying algorithm than gives a set of images scores on a “Grapes vs. Cheese” score. For a given set of pictures where some are grapes and some are cheese it gives all the grape images a higher “grape score” than the cheese. It works by using a circular hough transform to count the number of circles in an image. The more circles, the higher the grape score. Not expected to work generally on all arbitrary images of grapes or cheese.

VidendoAedificare – See my previous article Videndo Aedificare.

The PolyViz Viewer displaying three polyhedra.

“PolyViz is a tool for visualizing n-dimensional polyhedra. Motivated by the difficulty of reasoning about the iteration spaces of nested loops with many levels, PolyViz allows users to visualize polyhedral representations of those iteration spaces.”

- PolyViz Sourceforge project description

I started writing PolyViz for an advanced high performance computing (HPC) course last spring (early 2010). The course focused on the polyhedral model, a way of thinking about tightly nested loops in computer programs as geometric shapes. In the polyhedral model, each loop has an index variable, like in a standard for loop, such as i, j or k. Each index variable iterates over a range of values, this set of values is called its domain. Each index variable is interpreted as a dimension of the polyhedron. Values that are included in the domain are within the polyhedron, all other values are not.

PolyViz on Sourceforge – See the files section to download source packages or the manual, which is very good and covers background materials and the mathematics involved, along with instructions on how to use the software.

For simple loops this usually results in some manner of rectangular shape (though it may be 3, 4 or higher dimensional depending on how deep the loop nest goes). However, for loops with more complex domains, the polyhedrons become more complex themselves.

The polyhedral model shows its worth when one tries to optimise a loop nest. There are many program transformations that may improve the performance of a loop nest, such as shifting the order of loops or skewing the loop in some way. These transformations are especially important when trying to parallelize a loop nest. Traditionally, these transformations are applied manually to the loop code by skilled programmers. In the polyhedral model, many of these transformations can be expressed as functions that can be applied to the polyhedral domain to arrive at the modified program. Of course, the program must be translated into and out of the polyhedral model in order to leverage this mathematical elegance.

This is where PolyViz can help. It can be very difficult, even for experts, to look at loop bounds in a nest of for loops and imagine the shape of the implied polyhedral iteration space. However, it is necessary to do this in order to reason about the effects of program transformations on important program attributes like data dependencies.

PolyViz allows programmers to type in the bounds of a polyhedron, in the form of a series of inequalities, and instantly view a model of the polyhedron. If the polyhedron is high dimensional (more than 3 dimensions), the user chooses which dimensions he or she would like to view directly and which will be parameterized. Parameterized dimensions are fixed at a given value. The result is essentially a rendering of a 3 dimensional slice of a higher dimensional object. See the manual for a great analogy (involving a koosh ball) that explains this concept.

Jon Roelofs and I co-wrote the PolyViz library software as part of the HPC course. I wrote the PolyViz Viewer GUI application last summer for Prof. Sanjay Rajopadhye with the support of the National Science Foundation.

PolyViz uses Ogre, an exceptional open source real-time 3D graphics engine, to provide visualizations. The PolyViz Viewer uses wxWidgets to provide a native GUI experience on OS X, Windows and Linux (GTK).

Aw, Snap! Chrome tab crashes.

I think Google Chrome is a great browser, but even great browsers sometimes fail. In fact, I just had two tabs crash on me. They happened to both be Google Docs tabs so Google can’t pin the blame on anyone else. However, instead of just sitting there dumbly, Chrome informed me that the tabs weren’t responding and gave me the option to close them. This is part of Google’s whole strategy for Chrome. The renderer for every tab is a separate process, so that if something goes wrong, they can be killed individually rather than crashing the whole browser.

While being able to kill the tabs was nice, it’s not what I’m writing about. What I’m writing about is Chrome’s brilliant strategy of, when it fails, being too cute for you to be mad at it. It’s a master stroke. Software that exploits people’s emotions to get people to like it even when it misbehaves! Software has finally evolved to the level of a two year old. The image above is a screenshot of the page Chrome showed me after I chose to kill the tabs. How can you be mad when looking into those poor pixel-art eyes?

Apache logo, courtesy of The Apache Software Foundation

For an unrelated project, I’m writing a section about open source licenses and I happened across language in the Apache license that gives users the right to “publicly display [and] publicly perform” the work. Immediately after reading those words, I sat there for a good five minutes thinking up creative ways to turn the source code of the most popular open-source web server in the world into a performance piece. Unfortunately none my ideas for a song and dance rendition of the Apache source code were very good (in fact, the very concept is wretched. It’d be intolerable to sit though). Though a copyleft, source-code based, visual art installation could be pretty cool… with pages of code pasted to the walls, floor and ceiling. Errant CRTs jutting from the mass of lacquered paper, scanning over the codebase in amber on black. The guts of the enabler of the modern web on display in all their hideous monotony.

A screenshot of the Google N-Gram Viewer in action. Plot shows the relative frequency of the words internet, computer, telephone and robot from 1850 to 2000.

For those unfamiliar with the project, Google Books is an attempt to digitize every book ever written. The project began in 2002 and many libraries, universities and publishers got on board. In 2004 the project stirred up a bit of controversy with lawsuits against Google charging that Google Books (then called Google Print) violated the copyrights of the books it scanned.

History aside, the reason I’m writing about this is that Google has released a load of data on the frequency of occurrence of words and phrases in the entire body of books it has scanned. The goal of publishing the data seems to be to allow academics to research the evolution of language, as used in books. However, beyond just making the raw data available, Google has provided a neat little webapp that is easy enough for anyone to use. It’s fun to play with! As pictured above, I made a plot of a few technology related words (with rather predictable results).

An image of the statute of Athena rendered by ZeroRay.

There are two approaches to the computer graphics problem:

1. For each object in a scene, project it onto the screen, then color in all the pixels that it covers.
2. For each pixel on the screen figure out which object in the scene it points at, then color the pixel the color of that object.

The first approach is the one used by mainstream, real-time graphics toolkits like OpenGL and DirectX. The second approach is ray tracing. This may seem like a fine distinction, but this choice determines what sort of things end up being hard or easy later.

Camera Models

Inquisitive Reader: So how does that #2 thing you said make a picture?

Jess: I’ll tell you how.

IR: So you keep saying.

J: Ahem. Well. To begin, you must have a model of a camera. By model, I mean a geometrical way of describing how an image is actually formed. In real life, this is somewhat complicated. But since we’re not in real life, we can assume very simple model: the pinhole camera.

An illustration of a pinhole camera courtesy of Wikipedia.

IR: That’s a nice picture, and accurate I’m sure, however, I couldn’t help but notice that the tree inside the box is upside down.

J: Quite observant of you! The place where the image of the tree is formed is called the image plane. We’ll cleverly do away with this problem by placing the image plane in front of the pinhole by the same distance that it is behind in the illustration. This way, the light rays will not have crossed paths at the pinhole yet, and the image will not be upside down.

IR: Wait, that doesn’t seem quite right… I’m sure there’s a problem here but I can’t put my finger on it.

J: The reason we can get away with this is because, as I mentioned before, we aren’t in real life. In real life the light-sensitive imaging surface must be contained in a dark box, so that light coming from other directions, doesn’t affect the image. Since we are simulating the light ourselves, we can choose to only simulate the light in front of the image plane.

IR: Simulating light… maybe we’re finally getting to the good part.

J: That we are. We call the point where the rays come together in the illustration, the focal point. To generate an image we create rays that begin at the focal point and go through each point on the image plane. We then trace them back out into the scene and see what they hit. This isn’t really simulating light, as I said earlier, but working backwards from the camera and saying, “Now if light were to hit the image plane here, where must it have come from?” Where the light came from determines what color it is, ergo what color we will make a given spot on the image.

J: To make an image out of all of this, we map our image onto the image plane. Since an image is essentially a grid of pixels, we imagine that grid superimposed on the image plane. Then we cast rays from the focal point through each grid cell, moving over the entire image plane, each ray determining the color of one pixel.

IR: It still seems as if there are a few things missing here. How do you figure out what the ray hits in the scene? How do you know what color light the ray should be anyway? How do lights and shadows work? I’m not at all satisfied by your hand waving.

J: That’s it. I’m tired of your carping. I’ll answer your questions, but I’ll do so in non-dialog form.

IR: No! Wait! I.. [zop!]

With a spray of orange sparks and a puff of acrid smoke, he was gone, and I was left to my narrative, uninterrupted.

Intersecting the Scene

A wireframe render of the Master Chief model from Halo 2.

Once you have a ray, travelling from the focal point, through the image plane, out into the scene, the next thing you need to know is what object the ray hits. The simplest way to determine this is to check each object in the scene and see if the ray intersects it.

Every type of object that exists in the scene must have a function to test whether a ray intersects it and to find where the intersection occurs. ZeroRay supports a number of primitive shapes, planes, spheres, rectangular prisms and billboards to be precise. Furthermore, it supports any mesh made up of triangles. This is pretty much the same set of objects supported by a non-ray-tracing graphics engine. Most of the objects you see in computer graphics are meshes of triangles (sometimes quadrilaterals are allowed, they are easily broken into 2 triangles along the diagonal). Some advanced rendering engines support more exotic primitives, but we’re not going to talk about them.

We are only interested in the intersection nearest to the camera. Any intersections that are further away will be blocked from the camera’s view by the closer object. The object of nearest intersection is the object that determines the color of the point in the image (so far. There are other considerations that are left for later).

Lights and Materials

Now that we know which object controls the color of a pixel, how do we figure out what the color is? Every object also has a property called the surface material, so named because it represents what type of material the object is made of. The surface material and lighting are the basic contributors to the color that we will ascribe to the ray. A surface material itself has a number of properties, the simples of which is diffuse color. Diffuse color is the color that light takes on when it is scattered diffusely (in all directions) after hitting the object. Another color property is specular color. For more details on diffuse vs specular lighting see my article Dynamic Skies and Normal Mapping (scroll down to the sections entitled Some Background on Lighting in 3D Graphics).

The surface material tells us the basic color of the object. To produce more realistic renderings we need to take lighting into account. A light is a very simple construct. It has simply a position and a color (and maybe a few attenuation properties, i.e. how strong its light is some distance away). The way the light is used is a bit more complex. Once we’ve determined the point of our camera ray intersection, we move on to lighting calculations. For each light in the scene, we cast a ray from the light to the camera ray intersection point. We find the nearest intersection in the scene for the light rays. If the nearest intersected object isn’t the same object as the camera ray intersection object, then that point is shadowed from that light by some other object. If the point is shadowed by another object, that light has no contribution to that point’s color. If the object is the same, then we know that there is nothing blocking that light and we perform the calculation described in the Dynamic Skies and Normal Mapping article to determine how much light that point receives.

Reflection, Refraction: Recursion

An early ZeroRay render showing the checkerboard floor reflecting on the shiny planet orbs.

The beauty of the ray tracing approach to graphics is that we can perform this whole process again to calculate reflections and refractions. We can use the angle of the surface the ray hit and the angle of incidence to determine which direction the ray would reflect off the surface. Since we’re tracing the light backwards, what we’re actually doing is figuring out the direction that light must have come from, if it were to have hit that point from somewhere else and reflected into the camera. So we send our algorithm off in that direction as if this ray were the original ray from the camera. Once we get a color back from that calculation, we add it to the color at the original point subject to a reflectivity parameter. More reflective objects show the reflected light brighter while less reflective objects show the reflected light more mutely.

This process of setting up a calculation that then performs the same calculation within itself is called recursion, and it is a very powerful method of computation. If we allowed infinite levels of reflection our ray tracer would never finish calculating the color of any point, so we usually set some hard limit on the number of recursive reflection calculations (like 3 or so).

A scene rendered by ZeroRay illustrating reflection and refraction. The glass orbs reflect and transmit light through them.

Another way we can exploit this same trick is if we have objects that transmit light through themselves, like glass. We can use the ray angles to calculate the ray that the light would travel after passing through the object and recursively calculate the contribution of light passing through the object to the color of the point.

Introduction

It’s late at night. Your eyes are blurry from hours of reading at your computer screen but you have pages more to go before tomorrow’s deadline. Wouldn’t it be nice if you could close your eyes and have your computer read those last few pages to you?

You just finished that paper you had so much trouble writing. You don’t have time to find someone to proofread it before tomorrow and you’ve been staring at it for so long you know you’ll read right over any mistakes. Wouldn’t it be nice if your computer could read it to you out loud, making those silly grammatical mistakes sound obvious?

If you‘re using Apple’s OS X, you can do both of those things easily. Apple was one of the early adopters of speech synthesis in 1984 and support for text to speech has been in their operating systems ever since. OS X has been shipped with all Macintosh computers since 2002. Unfortunately Apple is fond of moving the location of speech related menu items between versions, making users find them again. This document will teach you how to assign speech actions to a quick key combination in OS 10.6 “Snow Leopard” and how to use the command line tool “say” to create audio files of text to listen to at your leisure.

Universal Access

Many of OS X’s speech features are presented within the context of Universal Access which comprises extensions to the user interface that aid users with vision disabilities. The Universal Access tools are designed for users with disabilities, so they assume that if you want text on the screen spoken to you, you will want all of the text on the screen spoken to you all of the time. VoiceOver is a feature with Universal Access that narrates whatever you pass your mouse cursor over. Since we just want the computer to speak some specific things, we will not use VoiceOver or Universal Access but they are worth addressing because many sources that you may come across regarding Apple’s speech tools will direct you to Universal Access.

Speak Selected Text Shortcut

A screen shot showing how to access the System Preferences on OS X

Next we will create a keyboard shortcut to speak highlighted text in any application. First, open the System Preferences. You can do this by opening the Apple menu at the top left of the screen and selecting “System Preferences…” (see :ref:`Figure 1 <fig1>`). Now select the “Speech” icon from the system preferences. You will now be confronted with the Speech preferences dialog. Check the third checkbox (see :ref:`Figure 2 <fig2>`) and press the “Set Key…” button directly to the right of the third checkbox. Finally, press a key combination that you would like to use for this action, preferably one that is not used by any other common action. Control-S is a simple choice that is not commonly used elsewhere (not to be confused with Command-S which is very commonly used).

You now have a keyboard shortcut to speak any text you like. If you are reading this document on your computer try it by highlighting this sentence and pressing your chosen keyboard shortcut. Here’s a fun sentence to try: “I am a computer. Boop beep boop be doop beep!” If you want to stop the computer in the middle of speaking press the shortcut again.

The Speech Preferences dialog. See the third checkbox labeled "Speak selected text when the key is pressed."

Advanced Tools

An Introduction to Say

Now that we’ve covered the basics, there are some more advanced tools that can be useful. The program “say” is included with OS X. It is designed to be run at the command line and can convert text you type directly or the entire contents of a text file to speech that is either immediately played or saved to a file. To use a command line tool you will need to use a program called Terminal. To find it go to the Applications folder, then the Utilities folder, then double click on Terminal. Or go to the spotlight search bar on the top right of the screen and type Terminal. When the Terminal app appears on the search list, press enter.

Once we have a terminal window open we can play with say. Try typing:

say "I can make you say whatever I want."

You should hear the computer speak the text in quotes. The default mode, as you just heard, is to speak the text directly through your speakers (or your selected sound device if you have changed it). Lets say we want to make an audio file to listen to later, or to send to someone else, or perhaps because we’d like to be able to pause and resume the speech. We can do that using the following command:

say "I can make you say whatever I want." -o speech.wav.

The text in quotes will be spoken into a file called speech.wav. You can also use Apple’s AAC compression file format by using the extension m4a rather than wav. The output audio file is saved in the folder that you executed “say” in. If you are not sure where that is, type:

pwd

This will print out Terminal’s *working directory*, the address of the folder that Terminal is currently in. The address is a list of all the folders leading to the working directory separated by slashes, much like what you see in web addresses after the .com (or .org, or .net).

‘Say’ an Entire Document

Making your computer speak a short sentence that you have to type at the command line is not much of a convenience. However, you can tell “say” to create an audio file of an entire text file. This is much more useful as you can convert the entire document and sit back and listen. If you have to get up you can pause it, if you miss something you can rewind a bit and hear it again. The one snag is that “say” can only handle simple text files. It will not work on pdf or Microsoft Word documents. If you have a pdf or Word document that you want to have spoken to you, you can open the document and copy the text into TextEdit (in the Applications folder). You may need to clean up the pasted text as many pdf documents have headers and/or footers that will get pasted in the middle of the text of your document. Having a sentence interrupted to hear that you’re on page three of seven can be quite distracting. You can manually scan and delete unwanted sections in TextEdit and save the cleaned up file.

This next part may be a bit unfamiliar to those who do not often use the command line. You will have to change Terminal’s working directory to the folder where you have saved the document you want to “say”. To do this, you can use the “cd” (change directory) command to move to the desired folder. You can enter the entire path at once such as:

cd Documents/SayStuff

or you can move incrementally with the commands:

cd Documents

cd SayStuff

Both of these sequences will bring you to the same directory. Once you are in the folder with your text document, use “say” like this:

say -f MyDocument.rtf -o MyDocument.m4a

The file name after -f is the name of your text document (the extension may be other than rtf). The file name after -o is the name that the audio output file will be given. The output file name does not need to be similar to the input text document’s name. For long documents “say” may take a while to process.

Conclusion

OS X has an array of text to speech tools that are useful even to those without vision impairments. Those tools are not always easy to locate and many users are unaware of their existence. Now that you know how to speak highlighted text and create audio files of documents it’s time to give your eyes a rest and let your ears do the walking.

Like most state-of-the-art structure from motion techniques, Pan, Reitmayr and Drummond’s approach is feature based [1]. Image features is an entire class of research in computer vision. The premise is that rather than operating on the entire image, in the form of raw pixels, it is smarter to pick “interesting” parts of the image and just consider the information around those spots. The process of finding interesting spots is called feature (or interest point) detection. There are many types of feature detectors (bearing all sorts of different names). Some look for bright or dark blobs, some look for points with a lot of local texture and some decide whether a point is interesting on some other criteria entirely. ProFORMA uses the FAST corner detector as their feature detector.

There are two fundamental components to every feature:

1. Feature location
2. Feature descriptor.

Feature location is discovered during feature detection.  The feature descriptor is some way of describing what that feature is, as opposed to simply where it is. The most obvious feature descriptor is simply the pixels around the feature. Computer vision researchers have developed an array of less obvious feature descriptors that have neat properties, but a discussion of them is beyond the scope of this article. ProFORMA uses the 5×5 grid of pixels centered on the feature location as its feature descriptor (mean and variance normalized pixels that is).

Why features? ProFORMA needs feature detection because they want to find the same spots on the model from frame to frame, even though that spot will not be at the same pixel coordinates. Finding the same spots and seeing how much they have moved in pixel coordinates is how they determine how the model must be moving. Now, say that we’ve found 50 features in each frame 1 and frame 2. Some of the features have moved. Since they’re not at the same positions in the image, how do we know which feature in frame 2 was feature x in frame 1? This is where feature descriptors come in. We can compare the feature descriptor of feature x from frame 1 with the feature descriptor of every feature in frame 2. When we find one that matches we can assume that it is the same feature as in frame 1 and we can measure how far that spot on the model moved between frames. In this way we can figure out how many points on the model moved between frames. From this, given what we know about the shape of the model, we can infer the motion of the entire model.

Author’s Note: This article is one of three that compose my report on the Videndo Aedificare project. The other two are Background Subtraction and Videndo Aedificare.

References

[1] Q. Pan, G. Reitmayr, and T. Drummond. ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition. In Proc. 20th British Machine Vision Conference (BMVC), London, September 2009. Download

[2] Pan, Qi. “ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition.” 18 July 2009. Online video clip. YouTube. http://www.youtube.com/watch?v=vEOmzjImsVc

An example of a background/foreground segmentation from Janusz Konrad at Boston University.

After working for quite a while on the “motion detection” algorithms described in my last article, I was clued in to background subtraction. Or rather, it finally hit me why, when explaining what I was working on, people kept saying, “Oh, you’re doing background subtraction.”

Background subtraction is the keyword for a relatively well explored nook of computer vision. The motivation for background subtraction research is that, in an image, there is usually a part of the image that you care about (the foreground), and a part that you don’t care about (the background) and it would be nice to focus only on the parts that we care about. There are many justifiable ways to divide a single image into foreground and background sections if we use the criteria that the foreground is “things we care about” and the background is “things we don’t care about.” The colloquial distinction is that the foreground is usually closer to the camera, in focus, and more interesting than the background. The last is where subjectivity enters the equation. In the field of background subtraction and in the context of video, the consensus is that the background is the part of the image that does not belong to a sizable moving object. This is still a big vague but different algorithms have different ideas about what constitutes a background.

Running Gaussian Average

The simplest background model I found uses a running gaussian average. That’s fancy talk for, at every frame, each pixel of the background model is equal to a weighted sum of the pre-existing background pixel and the same pixel in the new frame. In pseudocode:

imgBG := the background image (blank to start)
a := weight that balances fast updating and background stability
For every frame:
    imgF := the current frame
    For every pixel p:
        imgBG[p] = a * imgBG[p] + (1 - a) * imgF[p]


Azarbayejani, Wren and Pentland created a person tracker called Pfinder in the mid 1990′s that utilized a running gaussian average for background subtraction [1]. Pfinder located and tracked people and parts of people such as their head and hands, for gesture recognition. They even had an interface to Wolfenstein3D where the user held a toy gun and moved their bodies to control the game (think Microsoft Kinect circa 1996).

Temporal Median Filter

The temporal median filter is almost the same as the background model I use in Videndo Aedificare (what I called the Evolving Background Model, EBM, in the last post). In similar words, my EBM would be called a temporal mean filter, as I used the arithmetic mean rather than the median.

A temporal median filter, as described by Cucchiara in the 2003 paper Detecting Moving Objects, Ghosts, and Shadows in Video Streams, takes N frames of previous video and, for each pixel, finds the median all N values [2]. This median becomes the value of the background model at each point. Rather than simply considering the last N frames, Cucchiara samples the video stream every 10 frames. In other words, when a new frame is added to the background model at time t, the set of frames considered consists of frames,

{t, t – 10, t – 20, … , t – 10 * n }.

As I mentioned, this is similar to how my background model works, but there are a few differences. Instead of finding the median (the value in the middle) of n frames, I calculated the mean (average) of all n frames. Rather than adding a new frame to the background model every 10 frames, or even every K frames, I added a frame to the background model every 1 second. I chose to use wall-clock time rather than frames so that the real-time response of my algorithm would be more independent of the speed of the computer running it.

Pixel Comparison and Color Spaces

A spinning cellphone being detected. I feature examples with my cell phone because it spins nicely, allowing me to get my hands out of the frame while it is still in motion.

Another difference between my method and most in the literature is in how they compared pixels. All of the examples in the literature, that operated on color images, used a color space distance to compare pixels. That is, they treated a color value as a 3D point in space and when comparing two pixels, they compare the straight-line (euclidean) distance between the two points. For instance RGB color space is just like a regular 3D space with the axis labelled r, g, b rather than the conventional x, y, z. In this way, they can consider two pixels different if their distance is above a certain threshold. I didn’t think of doing this! Instead, I compare each color channel pairwise between two pixels and decide that the pixels are different if the difference between ANY of the color channels is above a threshold. This is slightly faster than computing euclidean distance. The difference in the computation amounts to this, the straight line color space distance effectively draws a sphere about a point and says that any point within this sphere is not different enough to consider a change, while all the points outside of it are different. My distance measure effectively draws an axis-aligned cube about the point, in color space, and says any pixel falling within the cube is similar enough, all points outside are different.

The Wrap Up

Being dense and re-inventing a few background subtraction techniques ended up buying me a better understanding of this problem. If I had gone straight to the literature I would have picked out an algorithm straight from someone else’s paper. It might have saved me a lot of time, but not only would I not have learned as much, but I probably would have picked an overly complicated algorithm. Beyond those that I wrote about above, there are some pretty complicated background subtraction algorithms based on techniques such as mixture of gaussians, kernel density estimation or eigenbackgrounds [3]. Some of these more complicated techniques are very effective and I probably would have thought that the simpler techniques would be insufficient. As it turned out, my simple temporal mean filter was sufficient for my problem. Upon reading the literature it was validating to discover that many people had arrived at similar conclusions (and in papers less than ten years old!).

Author’s Note: This article is one of three that compose my report on the Videndo Aedificare project. The other two are Videndo Aedificare and Feature Based Approaches: ProFORMA.

References

[1] C. Wren, A. Azarbayejani, T. Darrell, and A.P. Pentland, “Pfinder: real-time tracking of the human body,” IEEE Trans. on Patfern Anal. and Machine Intell., vol. 19, no. 7, pp. 780-785, 1997. Download

[2] R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, “Detecting moving objects, ghosts, and shadows in video streams,” IEEE Trans. on Pattern Anal. and Machine Infell., vol. 25, no. 10, pp. 1337-1442, 2003. Download

[3] M. Piccardi, “Background subtraction techniques: a review,” IEEE Intl. Conf. on Systems, Man and Cybernetics, pp 3099-3104, 2004. Download

Introduction

The project was inspired by a paper by Pan, Reitmayr and Drummond called ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition. Actually, it might be more honest to say that the project was inspired by the video that the ProFORMA authors posted on YouTube.

For a discussion of ProFORMA and feature based approaches see my article, Feature Based Approaches: ProFORMA.

Videndo Aedificare (VA) does not use a feature based approach. I had wanted to at first, but decided that implementing a state of the art structure from motion system is beyond what is feasible in a semester project for a one person team (read: was talked down by my professor). Videndo Aedificare is built on my 3D graphics/computer vision toolkit, ZeroRay (which is a topic on my article todo list for this blog that keeps getting put off!). Its primary goal is to present a framework for exploring real time structure from motion algorithms. It provides a neat API to subscribe listeners to a connected webcam and classes to display results, either 2D images (which may be intermediate results for debugging), or 3D scenes. Videndo Aedificare uses the Ogre open source rendering engine to provide 3D views. Camera listeners implement a receiveFrame method, by which they are passed the current camera frame, and given time to operate on it. Often camera listeners have their own views to display results side by side with the raw camera feed.

Simple Builder

VA's Simple Builder. The webcam video stream picturing my monitor wearing a festive hat (left) beside the 3D rendering of the model constructed from the scene (right).

As a proof of concept, and to test my framework, I implemented the most naive scene reconstruction algorithm I could think of. It assumed that the intensity of a pixel was inversely proportional to the distance of that point to the camera. In other words, bright pixels are close to the camera, dark ones are far away. Simple Builder generates a polygonal mesh from each frame by first making the image greyscale and then interpreting it as a height map. The maximum and minimum heights are parameters and the greyscale values are interpreted between them. The visual effect is rather interesting.

Since each frame is changed into geometry in real time, the grey patch you see on the right hand side of the image above would shimmer and shift as you watched it. The human vision system infers a lot of information even from the height map. My first reaction to the Simple Builder was “Wow… that looks good! How did such a stupid idea work?!” But it actually didn’t. Rotating the 3D view around, you can see that the height map only rarely reflects the true geometry of the scene, even in a simplified sense. The one exception was skin. Skin often had the correct contour as in the hand in the image above, but as you can see the wall in the background appears in the foreground of the height map. The speckle texture is an effect of camera noise as no noise reduction was performed ont he image before extracting the height map.

Scene Builder

The other builder that I worked on, I creatively called Scene Builder. My roadmap was to have it proceed in the following manner:

1. Background Detection
2. Locate Motion/Object
3. Refine Object

I ended up spending quite a while on background detection, as it seemed crucial to have a robust way to locate the portion of the image that was moving. The reason that I bring the word “robust” into it, is that the video stream captured by the iSight camera in my MacBook Pro was very noisy. I’m not sure how noisy it is compared to cameras in general, or even other webcams, but I was surprised by the amount of gaussian (random) noise in my images. The noise could be reduced by good lighting conditions. Unfortunately, good lighting conditions seemed to be synonymous with daylight and I often ended up working on this project at night. Needless to say, developing a noise-tolerant algorithm became a priority.

With the goal of computing which regions had changed between two frames I needed two things: a way to obtain an image that contained only the background and a way to compare two images that returned a region of pixels. I’ll talk about the comparison method first.

Image Comparison

At first, I chose the simplest algorithm I could think of. A toleranced pixel by pixel comparison of the pixel intensities where I saved the minimum and maximum points that were different by at least the tolerance, epsilon. This yields an axis-aligned bounding box around the changed region. To be noise tolerant this technique required an epsilon of around 54 to get the detected region to stop jumping at shadows (er, at noise, it technically should jump at shadows). Since pixel values range between 0 and 255, an epsilon of 54 is 21% of the total range.

I was dissatisfied with how large the tolerance had to be so, after receiving a suggestion, I added the requirement that, for a pixel to be “moving” all of its neighboring pixels must also have changed. This is reasonable because it is unlikely that a real object moving in front of the camera will only affect one pixel. A one-pixel change is much more likely to be noise. Even if the change is due to a real object, it is so tiny that we probably aren’t interested in it anyway (this is not a fly tracking algorithm). This change vastly improved the noise robustness of the comparison. Previously I’d had to use pixel intensity as it was more noise tolerant than my intended comparison, comparing each colour channel and considering the two pixels different if any of the differences between channels was at least epsilon. With the neighbor consulting comparison, the stability actually improved by looking at all channels rather than the intensity. I was also able to drop epsilon back from 54 to around 10 to 15.

The evolving background model's detection of a box thrown into the field of view. The red rectangle is oversized because the box had just travelled into the frame from the upper left and bounced once.

Background Models

The section describes the series of background models I created as I explored this problem. This is a background subtraction problem. For a discussion of background subtraction in general and how my approaches relate to others in the literature see my article Background Subtraction.

The Simple Background Model

The background model went through several iterations. My first attempt, which I called the Simple Background Model (SBM) took N frames during an initialization period. During this period the user is required to present the camera with only the background. This led to a somewhat awkward testing workflow which began with me hiding under my desk during the initialization period and then popping up to wave my hands at the camera. This earned me a lot of weird looks and some interesting conversations at the office. The frames captured during the initialization phase are stored, and at the end of the phase they are averaged (using the arithmetic mean) into a single background image. This accumulation and averaging removed a lot of the noise from the background model.

Surprisingly, this simple model performed quite well. It found a rectangle around any new objects introduced into the scene and the rectangle was stable around the new object (it didn’t jitter or jump around). The problem that arose is that this model was very sensitive to tiny changes in the “background” of the scene over time. The SBM considered any difference from the background an “object” (very loosely). For example, in the above screenshot, the papers at the far side of the desk move very gradually as air currents in the room shifted. This is a very tiny change, but the model is sensitive to any change of even a few pixels. So after a few minutes, what was logically still the background, changed enough that the SBM thought it wasn’t background. I needed a more sophisticated model of the background of an image.

The Clever Background Model

For the next background model, I thought I was being very clever. It turned out that the clever idea I’d thought of didn’t work. So I scrapped the whole idea and went back to the simple idea that did work, the SBM, and tried to figure out how to make it evolve over time.

The Evolving Background Model

I called the third background model the Evolving Background Model (EBM). It operated on the same principles as the SBM; it captured a series of images and averaged them. However, the EBM kept all of its images through the entire run in a queue. After a brief initialization period where it filled its frame queue, it took another image every few frames. When it took a new image, it threw out its oldest image and multiplied every pixel in the image by 1 / N, where N is the number of images it was keeping. That way, when asked to make a comparison, it simply summed all N images together to arrive at the average background image. As images are constantly being replaced every few frames, the background model adapts to changes in the background. In fact, if you sit in front of the camera and hold very very still, it will begin to classify you as part of the background. This seemed great. No more hiding under the desk, no more weird looks around the office. The problem was, that now I was detecting motion, not just a change from the background. I thought this was what I wanted, but it turned out that it had its own pitfalls.

The reason I’m trying to detect foreground objects in the video feed is that I want to determine their size and location so that I can begin to generate a 3D model of them. The problem with detecting motion is that if a new object is placed in the scene, unless it is constantly perturbed, after a time it is no longer detected (since it isn’t moving). It becomes part of the background. This effect causes the detected region to jitter and move a lot, even if you hold up an object and rotate it in front of the camera. Some regions of the object have the same appearance and as you move the object it is possible for a solid coloured region to pass through the same pixel for several frames, resulting in a conclusion that that pixel is not moving. The EBM does very well at detecting motion and telling you where it is, but not at forming a neat, stable, bounding box around the object that is causing that motion.

Jitter Solutions

My first idea to solve this problem was to keep a running average of the detected region, or of the generated model geometry. But the more I thought about it, the more I began to think that this was a bad solution. Introducing a running average in my background model was going to necessitate introducing a running average into my object location? I would be chasing my tail. Going back to the SBM results in a stable detected object region, but reintroduces the problems of small background changes throwing off the detection. The compromise that I settled on was using the EBM with slow update rates, for instance, giving the EBM a new frame only every second rather than every few frames. This slows the jitter of the detected region without giving up the ability to adapt to gradual background changes.

Fitting Geometry

Scene Builder detecting a spinning cell phone (left) and fitting a cylinder between the camera and the background and texturing it with a portion of the live video (right).

I was very interested in this part of this project, but I ended up spending much of my time thinking about background models instead. Currently what the SceneBuilder does is generates a cylinder in front of the background image that roughly occludes the detected area. The cylinder is then textured with the part of the image in the detected region.

Future Extensions

• The texture that is placed on the object cylinder needs to be deformed so that the image does not appear distorted by the curved surface.
• Detect features within the changed region and match them between frames to determine whether the object has rotated.
• If the object has rotated, estimate the pose and add the new frame’s changed region’s texture to the part of the cylinder that was previously not exposed to the camera. In this way, texture the entire object as it is rotated.

Author’s Note: This article is one of three that compose my report on the Videndo Aedificare project. The other two are Background Subtraction and Feature Based Approaches: ProFORMA.