Facial Expression Recognition and Analysis Challenge 2015

• Michel Valstar, University of Nottingham
• Jeff Cohn, University of Pittsburgh/Carnegie Mellon University
• Lijun Yin, Binghamton University
• Gary McKeown, Queen’s University Belfast
• Marc Méhu, Webster University Vienna
• Maja Pantic, Imperial College London

Data Chairs:

• Jeff Girard, University of Pittsburgh
• Timur Almaev, University of Nottingham
• Bihan Jiang, Imperial College London

Programme committee:

Jeff Girard, 
Pittsburgh University, USA
Hatice Gunez, 
Queen Mary University London, UK
Qiang Ji, 
Rensselaer Polytechnic Institute, USA
Brais Martinez, 
University of Nottingham, UK
Louis-Philippe Morency, 
Carnegie Mellon University, USA
Ognjen Rudovic, 
Imperial College London, UK
Nicu Sebe, 
University of Trento, IT
Yorgos Tzimiropoulos, 
University of Nottingham, UK

The second Facial Expression Recognition and Analysis challenge FERA 2015 , held in conjunction with FG 2015 is calling for contributions in the form of papers and competition in the challenge. The challenge has two components: detection of the presence of a set of FACS Action Units (facial muscle actions), and estimation of FACS Action Unit intensity. Participants are invited to participate in one or more of the sub-challenges.

For more information about the challenge, see http://sspnet.eu/fera2015/ . To register for the challenge data please create an account for both the FERA 2015 dataset and the SEMAINE database.

Submission deadlines can be found below.

In addition to challenge contributions the FERA 2015 workshop calls for papers that do not compete in the challenge but that address related topics, e.g.:

* AU activation detection
* AU intensity estimation
* Temporal dynamics of AUs
* Benchmarking of facial expressions
* Novel applications using automatic Action Unit analysis
* Publicly available AU databases

Regarding the review process, papers competing in the challenge have lower novelty requirements than would be expected from a normal FG workshop. Besides correctness and readability, both performance on the challenge and novelty will be taken into account in the peer review process. Papers not competing in the challenge will be evaluated as usual.

Many details about the data, performance measures, and baseline features and prediction results are included in the baseline paper, which is now available for download. Note that the paper is work in progress. While it may not be complete, we aim to keep it factually correct at all times.

Download the data 

Create an account to download the training and development partitions from the FERA 2015 data website.

You will have to sign two EULAs: one for the BP4D data, and one for the SEMAINE database. You should first create a SEMAINE account to sign that EUlA, before creating the FERA 2015 account and signing the BP4D EULA. Send the latter to the organisers by email. Please also register your team at this point.

Details of the datasets 

BP4D labels

For the BP4D data there are two types of FACS annotation files: occurrence and intensity files.

Each video file has a single occurrence annotation file which contains the annotations for all action units. The occurrence files are in the CSV format and are essentially numerical matrices in which each column corresponds to a single action unit and each row corresponds to a single video frame. The files also contain a header row with action unit labels and a header column with frame numbers. The proper way to parse these files is to drop the headers and then index for the video frame(s) and action unit(s) of interest. The following examples in MATLAB code may be helpful:
OCC = csvread(’F001_T1.csv’); %import annotations for one video file

FRNO = OCC(2:end,1); %get all frame numbers

CODES = OCC(2:end,2:end); %get codes for all action units

AU12 = CODES(:,12); %index action unit 12 codes for all video frames

FR10 = CODES(FRNO==10,:); %index all action unit codes for video frame 10

AU12_FR10 = CODES(FRNO==10,12); %index code for action unit 12 for video frame 10

The occurrence codes themselves are either 0 for absent, 1 for present, or 9 for missing data (unknown). Codes will be missing when the video frame was not coded by the manual annotator (not all frames and AUs were).
Each video file has multiple intensity annotation files: one per action unit. The action unit annotated in each file is indicated in its file name. The intensity files are also in the CSV format and are essentially numerical vectors in which each row corresponds to a single video frame. These files have no header rows or columns. The row number corresponds to the actual video frame number. The following examples in MATLAB code may be helpful:

INT = csvread(’2F02_01_AU12_DR.csv’); %import annotations for one video file

AU12_FR10 = INT(10); %index code for action unit 12 for video frame 10
The intensity codes themselves are either 0 for absent, 1 for present at the A level, 2 for present at the B level, 3 for present at the C level, 4 for present at the D level, 5 for present at the E level, or 9 for missing data (unknown). Codes will be missing when the video frame was not coded by the manual annotator (not all frames were) and when the face was occluded during the video frame.

SEMAINE labels

We have now prepared Matlab code for reading SEMAINE AU annotations, please find it attached below.

As you already know in this year challenge half of the data comes from the SEMAINE database. These data have been annotated using the ELAN annotation toolbox (https://tla.mpi.nl/tools/tla-tools/elan/), which stores annotations in XML structures saved in files with “.eaf” extension. Despite the extension, ELAN annotation files could be opened with any text editor and parsed with almost any XML parser available today. There are also files with “.pfsx” extension accompanying annotations, which are safe to ignore.

Inside the XML structure ELAN stores annotations in so-called tiers. There is a tier for speech and two tiers for every AU, one is for activations and another one is for intensity levels. Given that SEMAINE has been annotated for the total of 32 AUs, this gives 65 tiers. Since SEMAINE data is not going to be used in the intensity sub-challenge, we are going to ignore all intensity tiers as well as speech tier. We are also going to ignore most of the activation tiers as only the subset of 6 AUs (2, 12, 17, 25, 28, 45) has been chosen for this challenge from this database.

Regarding annotation rules during speech: when there’s speech we do not code the lower face AUs. This is the same procedure as we used for FERA 2011. The annotation files provide annotation for the occurrence of speech. For the purpose of score calculation, lower face AU scoring will be ignored during periods of speech.

Each tier is a combination of activation periods represented with two time stamps – when it starts and when it stops. Each time stamp refers to a particular time slot, table of which is defined in the beginning of the annotation file. A time slot has a numerical value in milliseconds which could be converted to frames by dividing time in ms by 1000 and multiplying the resulting value by the framerate. All SEMAINE videos have a constant framerate of 50 FPS.

The sample code written in Matlab takes a complete path to an ELAN annotation file, loads the XML structure, reads all time slots and converts time in ms into frames. It then goes through the above subset of 6 AUs and returns a numerical matrix of size NUMBER OF FRAMES by NUMBER OF ACTION UNITS, where each row represents a frame and each column is an AU. The matrix holds binary activation status for each frame / AU combination. The matrix also has a column header showing which AU corresponds to which column as well as a row header displaying original frame indexes.

The code requires an open source 3rdparty XML parser to work properly, which is included with the package.

Participant Registration

Please register your team with the organisers, by emailing Timur Almaev. When submitting your EULA for the data, please also provide a team name, the team leader, and for every person on the team a name and affiliation.

Participant System Submission

In FERA 2015, all participants must send in their fully working program. The program must be able to process data of the same format as the training and development partitions. We will later provide a function call signature that must be included in your program. We aim to support systems developed for Mac, Windows, and Linux.

Because of this flexibility, we ask you to work with us to get your programs working on our systems. We accept both compiled and source code entries. Please submit your first version two weeks before the paper submission deadline, to allow us time to set up your code. After your first submission, your scores will be sent to you within 48 hours of submission of a new system.

Participants can submit up to five systems. This is regardless of whether they are small tweaks of the same approach, or wildly different approaches. Any low-level comparisons and parameter optimisations should be done using the training and development partitions.