Flash and real-time Face recognition
To follow up on my augmented reality experiment, here is one that does real-time face tracking within Flash! It’s really a matter of using the low-level (c/c++) image libraries that are already out there! This time I used the commercial face tracking library from FaceOnIt, and was up and running in a couple of hours!
Sources are as always available on the Mammoth homepage, although you need to register on the FaceOnIt website to get this plug-in to compile.
And thanks to my Kruder & Dormeister CD and my girlfriends ‘elegance’ you don’t have to look at my face :-)
Flash and Augmented Reality
The whole idea of low level video/audio processing started with my object recognition experiment. But to actually take this a step further we need Flash connected to a media server, preferably one that can decode and analyze on the fly :-). With Mammoth (formely openFMS) released we can now unleach the power of the server cpu and use the low level (c/c++) image/audio processing libraries already available! Pretty cool huh?
In this experiment I use papervision on the client-side and an augmented reality library named ARToolKitPlus as a server plugin.
Here is the surprisingly fast result with a server cpu of almost zero:
Source code can be found on the Mammoth homepage, but please note license wise ARToolKit (GPL) and Mammoth (LGPL) are incompatible, it’s for experimentation use only.
OpenFMS announcement
I’m pleased to announce that a new Open Source Flash Server project is taking form (OpenFMS). Although still in alpha, OpenFMS will be a full blown high concurrency C++ Flash Server running on windows and *nix. Its main purpose is to spend as much time in kernel as possible.
features:
- streams all possible Flash codecs: h263, h264, mp3, vp6, etc…
- thanks to FFmpeg we have container support for almost every thinkable format: mov, flv, mkv, mp3, etc..
- built from the ground up while looking and learning from red5 code
- easy vhosts structure in the form of plugins
- live / record publishing (uses FFmpeg’s flv muxer to store FLV format)
- fully asynchronous with one thread per CPU
- uses a modifed version of boost channel for message broadcasting (gives us easy future access to edge/origin like configurations)
I still have a long way ahead, but if you care to help out, please do.
Using Google Maps within CS3
A friend of mine asked me if I knew a way to get google maps working within CS3. At the moment google made this flex only. If you’ve carefully read my previous articles you already know the drill.
This is what you need to do to get it working within CS3:
1. download the sdk from: http://code.google.com/apis/maps/documentation/flash/
2. copy the google swc to the flexsdk/frameworks/libs folder, where the other swc’s reside
3. create a library.as (read my previous article) from the HelloWorld google example and compile an swf and swc from that
4. run HelloWorld.as within your CS3 IDE
Download my example here to see it in action!
Good luck!
Intrinsic Library maker for Flash CS3
Jesse Warden had a post about the new Gaia Beta framework created by his friend Steven Sacks. In his post he commented and talked about my library solution. Let’s quote him:
There are just certain audiences that totally agree with what your doing, and others who are like, “That’s too much trouble”. We need to satiate those peeps. Yes, compiling a separate library is hard; it’s an extra step. If you make this shiz easy for Flash peeps, it rocks. Stay tuned…
To be honest, in the end I totally agree with what he says. So I sat down with my colleague Paul and we came up with a solution that automates the process of creating this ‘intrinsic’ library, in the form of a panel within Flash CS3!
Here are the screenshots that do all the talking:
 |
|
This is the first version, it has no Apple support just yet. In the next version I promiss we’ll add Apple support and an auto import feature my other colleagues came up with.
To make the install as easy as possible we’ve created an mxp. So for now, download version 0.0.1 here.
Using Flex compiled code within Flash!
Why did I put effort in combining Flash and Flex generated swf’s?
A major AS 3.0 obstacle for me is the fact that the flex RPC sources are not available. This means we can only use rpc.swc supplied within the Flex framework or use a third party library. Another big thing I’ve been messing around with is the idea of using cairngorm within the Flash authoring environment. As we all know cairngorm depends heavily on databinding, one can simulate databinding using simple macros and a preprocessor or one can let the flex compiler do all the hard work and embed the flex generated library into flash. I’ve been trying hard and succeeded in both, just read on…
As I was messing around with flex (compc) generated swc’s and things like playerglobal.swc, flex.swc, rpc.swc, etc… I came to found out that Flash actually compiles code against swc’s!! Yes, that’s compiling, this means NO linking! So, the linking part has to be done manually… Do you remember ‘intrinsic classes’ back in the good old AS 2.0 days? This result is the same, but you’re not dealing with exclusion xml’s and or the ASI files!
In short, what do you need to do?:
- First create a class that will import your needed dependencies (mx.*, whatever you need!)
- Now create your component library from your class file with ‘compc’, for ex:
‘compc -source-path . -include-classes FooLib -output=./example.swc’
You should now have an swc file which you can use to compile against. In the flash authoring environment the swc is only used for type checking, think of it as an intrinsic package! - Now you’ll compile the library that you’ll dynamically load to access your classes:
mxmlc -source-path . -o ./bin/library.swf ./FooLib.as - Before you’re ready to compile against your newly created library you need to load the library at runtime. As far as I can tell there are two options, my favorite is to link against it as a shared library. This allows all your swf’s to run quickly without any actual code! First you make a link to the library and when you’re done you put the movieclip on the first frame of your application. The other option (which is excellent as a preloader btw.) is to use a standard flash.display.Loader. (Just don’t forget to set the ApplicationDomain in the loadercontext or you’re classes will not be accessible!)
- And finally you’re ready to compile your Flash classes against the classes flex compiled for you.
As a proof of concept I’ve converted (read quick & dirty) the cairngorm cafetownsend mxml views to Flash. The views have dependencies on cairngorm and the flex framework (mx.rpc.*, mx.validators.StringValidator, etc..), but they compile without any of these classes in the path just because the swc sits besides the fla! You can download this package here. After you’ve downloaded the flex sdk to your favorite location you need to run ‘compile.cmd’ (put the flex sdk ‘bin’ folder to your path, or you need to edit compile.cmd and at the path in front of compc and mxmlc), this will automate step 2 & 3. Now you can open cafetownsend.fla and compile without any errors.
If you’re only interested in getting webservices support in Flash then download the examples here.
Now let’s use cairngorm within our Flash authoring environment :-)
For ‘Jon Bradley’ from the FlashCoders maillist I tested some flex controls. And yes, that’s possible as well although you need some style classes initialized before adding controls to your view. My first attempt involved custom flex initialization, download that one here. My second attempt has a better workflow: you create a basic flex application which is compiled against your library.swc. This basic flex application is used for initialization when you want it to! The trick involves the overriding of the flex application. If you want to look into this method further then download a working example here.
As far as I (or google) can tell this is the first proper solution to use your flex classes within Flash!
[Edit1]As Geoff van den Ouden found out, this solution does not work instantly with the Flex 3 SDK! You need to add “-compute-digest=false” to as an extra parameter to compc. Or you should download flex 2.0.2.[/Edit1]
[Edit2]Julian Kussman got AIR crashing while using this method. It turns out AIR doesn’t like the shared library option. You can download a working AIR rpc example here.[/Edit2]
Good luck!
Simple Actionscript Preprocessor
I was looking for a ’simple’ C preprocessor for actionscript. The best thing out there which is out of the box compatible with actionscript is “EnLarge” from Asual.
But even EnLarge is to complicated for me! I don’t want to make an ant build for every little thing I do. All I want is a preprocessor that takes an input folder and processes every single file to a specified output folder. So I took a peek on google and found “GPP“, which is a general-purpose preprocessor ‘almost’ perfect for my needs. Yes, ‘almost’ perfect because it’s missing some of my fundamental needs to process a folder for ex. So after some small modifications I’ve recompiled GPP to fit my needs. Download my binaries here, and get my patches for GPP 2.24 here.
GPP now supports the following syntax:
gpp.exe –source .\src_p –dest .\src –nocurinc –stdinclude .\macros
We’re using this in FlashDevelop using a shortcut key (F11) within the FlashDevelop menu, if you want this as well then copy gpp.exe to your FlashDevelop tools folder and modify MainMenu.xml with the following:
<button label="Preprocess src_p" click="RunProcess" shortcut="F11" tag="$(Quote)$(BaseDir)\Tools\gpp\gpp.exe$(Quote);--source $(Quote)$(ProjectDir)\src_p$(Quote) --dest $(Quote)$(ProjectDir)\src$(Quote) --nocurinc --stdinclude $(Quote)$(BaseDir)\Tools\gpp$(Quote)" />
to super or not to super
Sometimes there are reasons for not calling the base class constructor. A little trick in AS3.0 to not super() your base class is the following line of code:
if (0) super();
NellyMoser decoding support in ffmpeg!
I’ve been following the thread on the conversion of the anonymous nellymoser project to native support within ffmpeg. Since the 15th of october the patch has been approved and applied to the ffmpeg svn trunk. This is great news, we can now finally convert our FMS stored audio to anything we like!
Flash & Server Side object/face recognition
Ever had a project where you needed to process a webcam image? We had a project for Heineken.nl called cloning. We wanted to do object recognition for objects like beer cans, bottles, etc… The objects would be placed in front of the webcam. When recognized the user would see a ‘clone’ of the object. As we came to found out this process requires some serious processing power. At that time Flash (7/8) wasn’t up for the job. I even doubt the current ActionScript Virtual Machine (AVM2) in Flash Player 9 is powerful enough to handle such calculations.
The solution we came up with was to process the image by the server, but we wanted a fast way to send an image to the server. Unlike the method of storing a bitmapdata structure to the server we wanted to use the oh so fast built in Sorenson Squeeze encoder. Flash Media Server came to the rescue to handle this process, this is how it would look in theory (and afterwards in practice):
The Flash player makes a connection to the FMS and publishes it’s webcam stream to the server. After this process FMS makes a soap/xml/loadvars connection to the object recognition service which in turn runs through the stored flv and returns the objects recognized. Now comes the interesting part.. :-)
To share this method with the community I’ve hacked together an example of this recognition process. The programming part isn’t complicated at all (at least not on my part :-)), to follow my source code reusing philosophy it’s a matter of combining what’s already out there. Our example server will be able to process an flv, recognize the faces (and eyes) which are in the flv and return the data in xml. This xml must of course be served through a http daemon so FMS can do something with it.
Ok, so what do we need to put this all together?
- MingW GCC is the best open source compiler available.
- FFMPEG (LGPL or GPL) handles the Sorenson Squeeze codec
- Null HTTPD (GPL) provides us with the built-in web server to serve the xml
- The Machine Perception Toolbox (BSD License) does the actual face recognition
- POSIX Threads for Win32 (LGPL) Null HTTPD uses posix threads to handle the load
And here is the result of combining the whole lot together, my colleague Paul in front of the webcam :-)
I will not be going into any details of the actual c/c++ source code. But to get things compiling on your machine you need to do the following (please note, we’re compiling on windows here, but all libraries used are unix compatible):
- Extract the source code into one folder
- Download and compile the latest FFMpeg version into the ffmpeg folder
- Download and compile the latest version of the WIN32 POSIX threads library into the pthreads folder
- Run make within the nullhttpd folder
- To finish things off run a ‘make’ in the root folder
- You should now have a working ‘flv processing httpd daemon’
If the process above was to complicated for you can download the binaries here. And to top it of download the flash client source and the FMS sources as well.
Two big site notes here:
- The source code hasn’t been thoroughly tested for thread safety issues, memory leaks, etc… so don’t try run this in your production environment.
- This is a proof of concept solution only, if you want to start using this then respect the possible differences in licensing.
Good luck!
keep looking »