Even the best models for reading facial expressions may be partly subjective

by University of Glasgow

Credit: University of Glasgow

Does someone look angry or sad? You can probably offer an answer to that question based on the information you can see just by looking at their face. That’s because facial expressions—or a combination of different small facial movements—can be read by other humans to help understand what a person might be feeling at that exact moment.

Since Darwin’s seminal work on the evolutionary origins of facial expressions of emotion, scientists have been trying to find out which specific combinations of facial movements best represent our six basic emotions: happy, surprise, fear, disgust, anger and sadness. So far, researchers have offered a range of theories—or models—to define which facial movements best match each emotion, but until now no one has been able to show which one is most accurate.

Now, a new study by a team of European researchers led by the University of Glasgow and University of Amsterdam has begun to answer that question. The new study, which is published in Science Advances, shows that even the best models for predicting emotions from facial expressions fall short of the judgment of real human participants. Moreover, different humans themselves may read different emotions from the same facial expression, making it even harder to pinpoint exactly which facial movements are systematically linked with certain emotions.

The research indicates that our brains take in more information than just facial movements when understanding emotions in another person and, importantly, that personal characteristics (such as one’s gender and culture) may also influence this process.

The research team asked more than 120 participants, 60 Western and 60 East-Asian, to categorize a series of facial animation videos as having one of the six classic basic emotions, if they perceived one to be there. The researchers then compared these human answers to the predictions of the existing models, to see how accurate they were. They found that even the best models were far from perfect, highlighting that facial characteristics, such as gender or age, may also be important sources of information beyond simple facial movements.

Researchers also found that existing models perform better for Western than for East Asian participants, highlighting the persistent bias towards Western representations of emotional facial expressions in current theories in the field.

Lead author of the study, Dr. Lukas Snoek from the University of Glasgow’s School of Psychology & Neuroscience, said, “The question of how we read emotions from faces is important, and sometimes controversial. Since Darwin, there have been hypotheses about which facial movements correspond to a specific emotion, for example that a frown in combination with compressed lips signals ‘anger.’

“In our research we were able to show that not all humans will perceive the same emotion from the same set of facial movements. Our results indicate that these individual differences are in part due to one’s cultural background and we show that incorporating culture in emotion perception models improves their performance substantially.”

The paper, “Testing, explaining, and exploring models of facial expressions of emotions,” is published in Science Advances.