FaceReader 5.0 by Noldus IT company is without a doubt the best program for automatic analysis of facial expression of emotion. We employed it for the project "The impact of the capitalization of positive emotions on cardiovascular activity in a situation of stress", financed by the National Science Centre (N N106 016740), in which we wanted to combine the self-reported subjective and objective indicators of the affect. In this article, we would like to share our experience with FaceReader, so that other researchers may choose if this is a tool that can meet their expectations.
Distinguishing features of FaceReader:
CONTINUOUS analysis. FaceReader allows for rating the intensity of all emotions on the basis of each frame separately, or an average of about 15 frames/second, except of course for those moments when the study participants turn or lower their faces, cover their faces with their hands or there are other disturbances. If none of the following conditions does occur, observation charts showing the variation of emotional intensity in time remains impressive (some of them can be seen in the provided sample recording).
NON-INVASIVE analysis. Of course, if we assume that pointing the camera (or two) placed on the monitor in participants' faces might be considered non-invasive. Without a doubt, the degree of psychological and physical interference is much smaller than, for example, in the case of using a face muscle electromyography (EMG), which requires fixing multiple electrodes to participants' skin, along with the necessity of prior cleaning of the skin.
ACCURATE analysis, although one needs to remember that:
- very strong expression of emotions allows for greater validity
- FaceReader software copes especially well with a smile (joy is also the only positive emotion, which FaceReader recognizes)
- In the absence of expression of emotion program often wanders searching for the traces of emotions that are not there. This is not a major problem because even for short periods of time, this noise is generally averaged. You can also turn on the option that reduces the problem of these fluctuations by averaging the values of the last few frames, which gives a greater feeling of record stability.
The issues to be considered during the preparation procedures are as follows:
FACE. It is necessary for the participants to be looking straight toward the camera, or in other words, one needs to ensure that the camera follows the participants' gaze at parts of the experiment that are of importance.
GLASSES. The program is doing quite well with skipping glasses, as long as the frame does not cover the eyes. By creating a model face, the software simply blurs the glasses frame, which makes unpleasant / comic impression that the study participant has a black eye, but it seems that this does not affect the accuracy of expression measure. However, when glasses have thick frames, they might pose a big problem. The authors recommend (as do we) to stock up on a few sets of corrective glasses with different lenses and offer them to the participants. It is worth considering whether to ask participants to use lenses while taking part in the study, if possible.
LIGHTING. This is absolutely crucial. The light must fall from the front and be dispersed. In the attached recording it can be seen that in our case, after testing many different solutions, we decided to use two rectangular paper lamps from IKEA store, arranged them at the two sides of the screen at an average power level. Image quality (assessed with the FaceReader algorithm) was thus almost the maximum. In addition, this solution is very aesthetic.
HAIR. FaceReader cannot deal with hair that covers the eyebrows. This kind of hairstyle makes the FaceReader software completely helpless. All that remains is asking the study participants to use the elastic band in that case. If the headphones are used during the study, you can ask the participants to use them to make sure the hair arrangement is appropriate. Unfortunately, such requests make the participants aware that their face is a point of interest during the study.
HANDS. When part of the face is covered by the palm of a hand, FaceReader begins to create a grotesque facial model and the interpretation is completely wrong. Therefore, you should review all the recordings and such artifacts should be removed before analysis. The participants should be instructed to refrain from touching their faces - for some people, it might be difficult because touching the face is a strong habit.
DETAILED DATA ANALYSIS. Should you be interested in a detailed record of changes frame by frame (and most psychologists will be interested in that), you need to deal with the fact that FaceReader exports raw data to a simple .txt file. Its analysis will require much invention, eg. developing a set of complex macros for use in Excel. A detailed report gives tremendous opportunities for data analysis, but FaceReader 5.0 does not provide any tools for their easy use.
TIME-CONSUMING ANALYSIS. In psychological research, in most situations, the video of the study will be recorded to disk and analyzed later, eg. after the completion of the study. For our research, we used very fast computers (Dell Workstation T3600 Precision, and PC i7 2600K processor overclocked to 4.4 GHz). In each of these cases, the speed of the analysis was not greater than 10-15 frames per second, due to the fact that the software uses only one processor core. Thus, analysis of half-hour recording takes approximately 45 minutes. This, in turn, means that a very good computer working at top speed is able to process about 30 recordings per day. It is, therefore, necessary to save a few days to conduct the analysis.
MODULAR CONSTRUCTION. The modular structure might work very well, eg. in the case of buying furniture. However, in the case of software, it can be a great burden. Unfortunately, FaceReader 5.0 is not a unitary product that would provide all of the features needed. To perform the analysis using a method based on the FACS system, you should purchase an additional module for a price similar to the basic version. To perform a little more complex analysis, you need to buy another module called Project Analysis. To record a video from several sources at the same time, you should buy additional Media Recorder 2.0. And so on. It seems that this is due to marketing idea rather than care about user experience. It would be much easier to have it all from the beginning and to the end, especially if you are not familiar with the product. It might be difficult to grasp its functions, and the buyers expect fully operational product, not just bait for further purchases.
To sum up, FaceReader 5.0 is a very useful program with accuracy which clearly improves with each new version (current version 5.0 came in early 2013). However, it requires a great deal of preparation and appropriate conditions for video recordings, as well as one's own idea for data analysis based on detailed reports.
(Łukasz Kaczmarek)