challenges

FERA2011 Challenge

On the 25th of March 2011, the first Facial Expression Recognition and Analysis challenge (FERA2011) was held in Santa Barbara, California. The aim of the challenge was to hold a competition where all participants competed on a level playing field, with a clear, predetermined protocol. It is described in detail in a paper published in the proceedings of FG’11. Two sub-challenges were defined: the first was in the detection of Facial Action Units (AUs), which encode atomic facial muscle actions in terms of (groups of) muscle contractions in the face. The second sub-challenge was in a more popular yet more contested area: the detection of expressions of discrete emotions.

The data used was the GEMEP-FERA dataset, a subset of the GEMEP database specially created for this challenge. Although the GEMEP database is still not publicly available, the GEMEP-FERA dataset with the AU activation labels and emotion labels, is now available for any scientist wishing to benchmark their AU detection and/or emotional expression recognition system. The data is considered challenging mainly for two reasons: firstly, there is a lot of speech during the recordings, which makes the detection of lower-face AUs a lot harder. Secondly, there is a large amount of head movements. As the challenge results showed, those who attained high competition rankings also had robust methods to deal with this head motion.

The AU-detection sub-challenge was done on a frame-by-frame basis. This was possible and valuable because in the duration of a single video, multiple AUs are activated at different times. It is also not uncommon for the same AU to be activated more than once during the same video. The emotional expression recognition sub-challenge was done on event basis, with each video given a single, discrete label corresponding to an emotional description. This was necessary, because the videos were cut such that they would start immediately with the expression, and ended still in that expression.

The challenge was a great success, with over 90 attendants to the workshop in Santa Barbara (even though it was the first day with decent weather, and held after the main conference). In total there were 15 participants, and 11 accepted papers. The person-specific emotional expression recognition problem appeared to be solved: with the top-three ranking participants attaining 94%, 96%, and 100%. The AU detection problem appeared to be much more difficult, and it seems that there is still a long way to go before this problem is solved. Only 5 participants contributed to this sub-challenge, and the highest scoring team attained an F1-measure of 62% .

The presentation that I gave with a meta-analysis of the challenge and the competition results is now available. The results can also be found on the FERA2011 website.