ABSTRACT:
Over the past few years, there has been an increased interest in automatic facial behavior analysis and understanding. We present OpenFace an open source tool intended for computer vision and machine learning researchers, affective computing community and people interested in building interactive applications based on facial behavior analysis.
OpenFace is the first open source tool capable of facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation. The computer vision algorithms which represent the core of OpenFace demonstrate state-of-the-art results in all of the above mentioned tasks.
Furthermore, our tool is capable of real-time performance and is able to run from a simple webcam without any specialist hardware. Finally, OpenFace allows for easy integration with other applications and de-vices through a lightweight messaging system.
OPEN FACE PIPELINE
In this section we outline the core technologies used by OpenFace for facial behavior analysis First, we provide an explanation of how we detect and track facial landmarks, together with a hierarchical model extension to an existing algorithm. We then provide an outline of how these features are used for head pose estimation and eye gaze tracking.
Facial Landmark Detection and Tracking:
OpenFace uses the recently proposed Conditional Local Neural Fields (CLNF) for facial landmark detection and tracking. CLNF is an instance of a Constrained Local Model (CLM), that uses more advanced patch experts.
Sample Registrations on 300-w and Mpiigaze Datasets.
Head Pose Estimation:
Our model is able to extract head pose (translation and orientation) information in addition to facial landmark detection. We are able to do this, as CLNF internally uses a 3D representation of facial landmarks and projects them to the image using orthographic camera projection. This allows us to accurately estimate the head pose once the landmarks are detected by solving the PnP problem.
For accurate head pose estimation OpenFace needs to be provided with the camera calibration parameters (focal length and principal point). In their absence OpenFace uses a rough estimate based on image size.
Eye Gaze Estimation:
CLNF framework is a general deformable shape registration approach so we use it to detect eye-region landmarks as well. This includes eyelids, iris and the pupil. We used the SynthesEyes training dataset to train the PDM and CLNF patch experts. This model achieves state-of-the-art results in eye-region registration task.
Action Unit Detection:
OpenFace AU intensity and presence detection module is based on a recent state-of-the-art AU recognition framework. It is a direct implementation with a couple of changes that adapt it to work better on natural video sequences from unseen datasets. A more detailed explanation of the system can be found in Baltrusaitis et al. In the following section we describe our extensions to the approach and the implementation details.
EXPERIMENTAL EVALUATION
In this section, we evaluate each of our OpenFace sub sytems: facial landmark detection, head pose estimation, eye gaze estimation, and facial Action Unit detection. For each of our experiments we also include comparisons with a number of recently proposed approaches for tackling the same problems (although none of them tackle all of them at once).
Furthermore, all of the approaches we compared against provide only binaries with pre-trained models and not the full training and testing code (except for EyeTab and regression forests ).
Fitting on the Wild Datasets Using the Clnf Approach Included in Openface Compared Against State-of-the-art Methods.
INTERFACE
OpenFace is an easy to use toolbox for the analysis of facial behavior. There are three main ways of using Open-Face: Graphical User Interface, command line, and real-time messaging system (based on ZeroMQ). As the system is open source it is also possible to integrate it in any C++ or C based project. To make the system easier to use we provide sample Matlab scripts that demonstrate how to extract, save, read and visualize each of the behaviors. The system is cross-platform and has been tested on Windows, Ubuntu and Mac OS X.
CONCLUSION
In this paper we presented OpenFace – a first fully open source real-time facial behavior analysis system. OpenFace is a useful tool for the computer vision, machine learning and affective computing communities and will stimulate research in facial behavior analysis an understanding.
Furthermore, the future development of the tool will continue and it will attempt to incorporate the newest and most reliable approaches for the problem at hand while remaining a transparent open source tool and retaining its real-time capacity. We hope that this tool will encourage other researchers in the field to share their code.
Source: University of Cambridge
Authors: Tadas Baltru saitis | Peter Robinson | Louis-Philippe Morency
>> Matlab Projects Fingerprint Recognition and Face detection for Final Year Students