Automatic Human Affect Analysis

Emotional intelligence (EQ) is an aspect of human intelligence that has been argued to be a better predictor than IQ for measuring aspects of success in life, especially in social interactions, learning, and adapting to what is important. When it comes to machines, not all of them will need emotional skills. Yet to have machines like computers, broadcast systems, and cars, capable of adapting to their users and of anticipating their wishes, endowing them with the ability to recognize user’s affective states is necessary.

Automatic recognition of human affective states is an important research topic for video surveillance as well. Automatic assessment of boredom, inattention and stress will be highly valuable in situations where firm attention to a crucial, but perhaps tedious task is essential, such as aircraft control, air traffic control, or simply driving a ground vehicle like a truck, train or car. An automated tool could provide prompts for better performance based on the sensed user’s affective states. In addition, monitoring and interpreting affective behavioural cues are important to lawyers, police and security agents who are often interested in issues concerning deception and attitude.

The current focus of our research in the field is automatic recognition of non-prototypic affective states (e.g., pain, fatigue, confusion, boredom). Furthermore, in contrast to the existing approaches to machine analysis of human affective states, we want to develop an human affect analyzer that is multimodal (combining facial expressions, nonverbal vocalizations, and head, hand, and body movements), sensitive to the dynamics of displayed behavioural cues (performing temporal analysis of the sensed data that are previously processed in a joint feature space), and context-aware.

Individual Final Projects in this area of research consider the following topics:

Sleepiness, Boredom and Inattention Detection

The focus of the research in this field is on spotting the differences between behavioural cues like facial expressions, head movements, and body/hand movements shown when somebody is sleepy / inattentive / bored and that shown when that person is attentive / interested and using this knowledge to develop sleepiness / inattentiveness / boredom detector. No research has been conducted yet in this field and there are no readily available datasets that can be used for the study. The data should be first acquired and annotated (in terms of activated AUs). The data can be acquired in human-computer interaction scenarios, or by recording students’ behaviours during lectures, or by recording your own behaviour throughout the day (by means of a wearable camera, mounted on a cap that one can wear the whole day).

Fatigue and Confusion Detection

Similar research as in the case of sleepiness / inattention / boredom detection. No research has been conducted yet in this field and there are no readily available datasets that can be used for the study. The data should be first acquired and annotated (in terms of activated AUs). The data can be acquired in human-computer interaction scenarios.

Audiovisual Emotion Detection

The focus of the research in this field is on recognition of a small set of emotions (e.g., positive / negative emotions or six basic emotions such as anger, surprise, and happiness) based on both vocal expressions of emotions and facial expressions of emotions. Fusion of different modalities should receive a particular attention. It would be interesting to test the final system on both spontaneous audiovisual expressions of emotions and deliberately displayed audiovisual expressions of emotions.

Emotion Detection from Non-linguistic Vocalizations

Humans seem to be rather accurate in decoding some non-basic affective states such as distress, anxiety, and boredom from non-linguistic vocalizations like laughs, cries, sighs, and yawns. The focus of the research on this topic will first be on development of automatic methods for extracting auditory features from input audio signals to be used for detection of non-linguistic vocalizations like laughs, cries, sighs, etc. Then, the focus of the research will be on emotion detection based on sensed non-linguistic vocalizations. No research has been conducted yet in this field and there are no readily available datasets that can be used for the study. The data should be first acquired and annotated (in terms of displayed non-linguistic vocalizations). The data can be acquired in human-human interaction scenarios.

Multimodal Fatigue Detection

The focus of the research is on spatiotemporal analysis of multimodal behavioural cues (facial expressions, nonverbal vocalizations, and head, hand, and body movements) and on fusion of these observations for fatigue detection. No research has been conducted yet in this field and there are no readily available datasets that can be used for the study. The data should be first acquired and annotated (in terms of displayed behavioural cues). The data can be acquired by recording people performing aerobic exercises.

Contact:

Dr. Maja Pantic
Computing Department
Imperial College London
e-mail: m.pantic@imperial.ac.uk
website: http://www.doc.ic.ac.uk/~maja/