On the use of Action Units and fuzzy explanatory models for facial expression recognition

Autoři: E. Morales-Vargas aff001;  C. A. Reyes-García aff001;  Hayde Peregrina-Barreto aff001
Působiště autorů: Instituto Nacional de Astrofisica, Optica y Electronica, Luis Enrique Erro 1, Santa Maria Tonantzintla, 72840 Puebla, Mexico aff001
Vyšlo v časopise: PLoS ONE 14(10)
Kategorie: Research Article
doi: https://doi.org/10.1371/journal.pone.0223563


Facial expression recognition is related to the automatic identification of affective states of a subject by computational means. Facial expression recognition is used for many applications, such as security, human-computer interaction, driver safety, and health care. Although many works aim to tackle the problem of facial expression recognition, and the discriminative power may be acceptable, current solutions have limited explicative power, which is insufficient for certain applications, such as facial rehabilitation. Our aim is to alleviate the current limited explicative power by exploiting explainable fuzzy models over sequences of frontal face images. The proposed model uses appearance features to describe facial expressions in terms of facial movements, giving a detailed explanation of what movements are in the face, and why the model is making a decision. The model architecture was selected to keep the semantic meaning of the found facial movements. The proposed model can discriminate between the seven basic facial expressions, obtaining an average accuracy of 90.8±14%, with a maximum value of 92.9±28%.

Klíčová slova:

Database and informatics methods – Decision making – Deformation – Emotions – Face – Face recognition – Semantics – Lips


1. Fasel B, Luettin J. Automatic facial expression analysis: a survey. Pattern Recognition. 2003;36(1):259–275. doi: 10.1016/S0031-3203(02)00052-3

2. Pantic M, Rothkrantz LJM. Automatic analysis of facial expressions: the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2000;22(12):1424–1445. doi: 10.1109/34.895976

3. Khanam A, Shafiq MZ, Akram MU. Fuzzy Based Facial Expression Recognition. In: Congress on Image and Signal Processing, 2008. CISP’08. vol. 1; 2008. p. 598–602.

4. Cohn JMV J F. Depression, smiling and facial paralysis. Facial palsies Amsterdam, the Netherlands: Lemma Holland. 2005;.

5. Kanade T, Cohn JF, Tian Y. Comprehensive database for facial expression analysis; 2000. p. 46–53.

6. Storer JS, Brzuskiewicz J, Floyd H, Rice JC. Review: Topical Minoxidil for Male Pattern Baldness. The American Journal of the Medical Sciences. 1986;291(5):328–333. doi: 10.1097/00000441-198605000-00008

7. Cohn JF, Zlochower AJ, Lien J, Kanade T. Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. Psychophysiology. 1999;36(1):35–43. doi: 10.1017/s0048577299971184 10098378

8. Lucey P, Cohn J, Lucey S, Matthews I, Sridharan S, Prkachin KM. Automatically detecting pain using facial actions. In: 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops; 2009. p. 1–8.

9. Sucar LE, Azcárate G, Leder RS, Reinkensmeyer D, Hernández J, Sanchez I, et al. Gesture Therapy: A Vision-Based System for Arm Rehabilitation after Stroke. In: Fred A, Filipe J, Gamboa H, editors. Biomedical Engineering Systems and Technologies. Communications in Computer and Information Science. Springer Berlin Heidelberg; 2009. p. 531–540.

10. Byrne PJ. Importance of facial expression in facial nerve rehabilitation. Curr Opin Otolaryngol Head Neck Surg. 2004;12(4):332–335. doi: 10.1097/01.moo.0000134829.61048.64 15252257

11. García-Casal JA, Goñi-Imizcoz M, Perea-Bartolomé MV, Soto-Pérez F, Smith SJ, Calvo-Simal S, et al. The Efficacy of Emotion Recognition Rehabilitation for People with Alzheimer’s Disease. J Alzheimers Dis. 2017;57(3):937–951.

12. Shan C, Gong S, McOwan PW. Facial expression recognition based on Local Binary Patterns: A comprehensive study. Image and Vision Computing. 2009;27(6):803–816. doi: 10.1016/j.imavis.2008.08.005

13. Mohammadi MR, Fatemizadeh E. Fuzzy local binary patterns: A comparison between Min-Max and Dot-Sum operators in the application of facial expression recognition. In: 2013 8th Iranian Conference on Machine Vision and Image Processing (MVIP); 2013. p. 315–319.

14. Zhao G, Pietikainen M. Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2007;29(6):915–928. doi: 10.1109/TPAMI.2007.1110 17431293

15. Iglesias F, Negri P, Buemi ME, Acevedo D, Mejail M. Facial expression recognition: a comparison between static and dynamic approaches. In: International Conference on Pattern Recognition Systems (ICPRS-16); 2016. p. 1–6.

16. Ekman P, Friesen W. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press; 1978.

17. Coan JA, Allen JJB. Handbook of Emotion Elicitation and Assessment. Oxford University Press; 2007.

18. Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2001;23(6):681–685. doi: 10.1109/34.927467

19. Ghasemi R, Ahmady M. Facial expression recognition using facial effective areas and Fuzzy logic. In: 2014 Iranian Conference on Intelligent Systems (ICIS); 2014. p. 1–4.

20. Bengio Y, Courville A, Vincent P. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2013;35(8):1798–1828. doi: 10.1109/TPAMI.2013.50 23787338

21. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I. The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition—Workshops; 2010. p. 94–101.

22. Boureau YL, Ponce J, Lecun Y. A Theoretical Analysis of Feature Pooling in Visual Recognition. In: 27th International Conference on Machine Learning, Haifa, Israel; 2010.

23. Reyes-Galaviz OF, Pedrycz W. Granular fuzzy models: Analysis, design, and evaluation. 2015;64:1–19.

24. Priyono A, Ridwan M, Alias AJ, Rahmat RA O K, Hassan A, Mohd Ali MA. Generation of Fuzzy Rules with Subtractive Clustering. Jurnal Teknologi. 2005;43(1). doi: 10.11113/jt.v43.782

25. Li H, Philip CCL, Huang HP. Fuzzy Neural Intelligent Systems: Mathematical Foundation and the Applications in Engineering. 1st ed. Boca Raton, FL, USA: CRC Press, Inc.; 2000.

26. Jang JSR. ANFIS: adaptive-network-based fuzzy inference system. IEEE Transactions on Systems, Man, and Cybernetics. 1993;23(3):665–685. doi: 10.1109/21.256541

27. Ekman P. Facial expression and emotion. American Psychologist. 1993;48(4):384–392. doi: 10.1037//0003-066x.48.4.384 8512154

28. Matthews I, Baker S. Active Appearance Models Revisited. International Journal of Computer Vision. 2004;60(2):135–164. doi: 10.1023/B:VISI.0000029666.37597.d3

29. Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, Knippenberg Av. Presentation and validation of the Radboud Faces Database. Cognition and Emotion. 2010;24(8):1377–1388. doi: 10.1080/02699930903485076

30. Ramanan D. Face Detection, Pose Estimation, and Landmark Localization in the Wild. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). CVPR’12. Washington, DC, USA: IEEE Computer Society; 2012. p. 2879–2886. Available from: http://dl.acm.org/citation.cfm?id=2354409.2355119.

31. Amos B, Ludwiczuk B, Satyanarayanan M. OpenFace: A general-purpose face recognition library with mobile applications. hgpuorg. 2016;.

32. Zavarez MV, Berriel RF, Oliveira-Santos T. Cross-Database Facial Expression Recognition Based on Fine-Tuned Deep Convolutional Network. In: 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI); 2017. p. 405–412.

33. Ali G, Iqbal MA, Choi TS. Boosted NNE collections for multicultural facial expression recognition. Pattern Recognition. 2016;55(Supplement C):14–27. doi: 10.1016/j.patcog.2016.01.032

34. Zhao H, Wang Z, Men J. Facial Complex Expression Recognition Based on Fuzzy Kernel Clustering and Support Vector Machines. In: Third International Conference on Natural Computation (ICNC 2007). vol. 1; 2007. p. 562–566.

35. Zhao K, Yang S, Wiliem A, Lovell BC. Landmark manifold: Revisiting the Riemannian manifold approach for facial emotion recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR); 2016. p. 1095–1100.

Článek vyšel v časopise


2019 Číslo 10
Nejčtenější tento týden