Flash & Server Side object/face recognition
Ever had a project where you needed to process a webcam image? We had a project for Heineken.nl called cloning. We wanted to do object recognition for objects like beer cans, bottles, etc… The objects would be placed in front of the webcam. When recognized the user would see a ‘clone’ of the object. As we came to found out this process requires some serious processing power. At that time Flash (7/8) wasn’t up for the job. I even doubt the current ActionScript Virtual Machine (AVM2) in Flash Player 9 is powerful enough to handle such calculations.
The solution we came up with was to process the image by the server, but we wanted a fast way to send an image to the server. Unlike the method of storing a bitmapdata structure to the server we wanted to use the oh so fast built in Sorenson Squeeze encoder. Flash Media Server came to the rescue to handle this process, this is how it would look in theory (and afterwards in practice):
The Flash player makes a connection to the FMS and publishes it’s webcam stream to the server. After this process FMS makes a soap/xml/loadvars connection to the object recognition service which in turn runs through the stored flv and returns the objects recognized. Now comes the interesting part.. :-)
To share this method with the community I’ve hacked together an example of this recognition process. The programming part isn’t complicated at all (at least not on my part :-)), to follow my source code reusing philosophy it’s a matter of combining what’s already out there. Our example server will be able to process an flv, recognize the faces (and eyes) which are in the flv and return the data in xml. This xml must of course be served through a http daemon so FMS can do something with it.
Ok, so what do we need to put this all together?
- MingW GCC is the best open source compiler available.
- FFMPEG (LGPL or GPL) handles the Sorenson Squeeze codec
- Null HTTPD (GPL) provides us with the built-in web server to serve the xml
- The Machine Perception Toolbox (BSD License) does the actual face recognition
- POSIX Threads for Win32 (LGPL) Null HTTPD uses posix threads to handle the load
And here is the result of combining the whole lot together, my colleague Paul in front of the webcam :-)
I will not be going into any details of the actual c/c++ source code. But to get things compiling on your machine you need to do the following (please note, we’re compiling on windows here, but all libraries used are unix compatible):
- Extract the source code into one folder
- Download and compile the latest FFMpeg version into the ffmpeg folder
- Download and compile the latest version of the WIN32 POSIX threads library into the pthreads folder
- Run make within the nullhttpd folder
- To finish things off run a ‘make’ in the root folder
- You should now have a working ‘flv processing httpd daemon’
If the process above was to complicated for you can download the binaries here. And to top it of download the flash client source and the FMS sources as well.
Two big site notes here:
- The source code hasn’t been thoroughly tested for thread safety issues, memory leaks, etc… so don’t try run this in your production environment.
- This is a proof of concept solution only, if you want to start using this then respect the possible differences in licensing.
Good luck!