O.V. Melnik1, V.A. Sablina2, G. Burresi3, A.V. Savin4

1,2,4 Ryazan State Radio Engineering University n. a. akad. V.F. Utkin (Ryazan, Russia)

3 Siena University

The automated estimation of the psycho-emotional state of the human and their emotional reactions to the different influences using the video image analysis is an urgent task in different fields, such as: safeguarding in manufacturing, aviation, transportation, prevention of the crimes and terroristic threats, marketing researches etc. A promising direction is the facial micro-expression analysis. The facial micro-expressions are not under conscious control and reflect the objective emotional reaction. One of the key stages of the procedure of the automatic emotion estimation by the facial micro-expressions is the correct facial landmark detection. It is a complex task because of the presence of the different noise in the consecutive frames.

Purpose – the investigation of the ways of increasing the performance of the facial micro-expression analysis pipeline by using preliminary video image processing procedures.

It is shown that, as the preliminary stage of the micro-expression analysis pipeline, it is reasonable to perform the blurring of the original images to obtain the more stable results. The determined filtering parameters provide the MediaPipe framework a performance increase for the micro-expression analysis problems. It is shown that the video image blurring by the Gaussian filter with a size of 15×15 pixels allows to reduce the noise influence and to decrease the incorrect shifts of the facial landmarks from frame to frame induced by this noise.

The proposed procedure of preliminary video image processing can be used for increasing the facial micro-expression recognition performance in emotion recognition systems based on the video sequence analysis.

Melnik O.V., Sablina V.A., Burresi G., Savin A.V. Filter selection for image processing before the landmark detection stage for microexpression analysis. Biomedicine Radioengineering. 2021. V. 24. № 4. P. 40–48. DOI: 10.18127/j15604136-202104-06 (in Russian)

1. Zongping Deng, Ke Li, Qijun Zhao, and Hu Chen. Face landmark localization using a single deep network. In CCBR 2016. LNCS 9967. 2016. P. 68-76.
2. Sergeeva A.D., Sablina V.A. Eye Landmarks Detection Technology for Facial Micro-Expressions Analysis, 9th Mediterranean Conference on Embedded Computing (MECO) Proceedings. Budva, Montenegro. 2020. Р. 448–451.
3. Ekman P. Emotion in the Human Face, 2nd Edition. Malor Books. 2013. 456 p.
4. Yue Wu, Qiang Ji. Facial Landmark Detection: A Literature Survey. International Journal of Computer Vision. February 2019. V. 127.  Is. 2. P. 115-142.
5. Mel'nik O.V., Sablina V.A., Savin A.V., Borshchev A.B. Obnaruzhenie antropometricheskih tochek lica na osnove metodov glubokogo obucheniya s cel'yu raspoznavaniya emocij. Biomedicinskaya radioelektronika. 2020. T. 23. № 3. S. 45 52 (in Russian).
6. Lugaresi C., Jiuqiang Tang, Hadon Nash, Mc-Clanahan Ch., Esha Uboweja, Hays M., Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, Grundmann M. Mediapipe: A framework for building perception pipelines. Google Research. 2019. 9 p. [accessed 2021 April 30], https://arxiv.org/abs/1906.08172.
7. Sergeeva A.D., Savin A.V., Sablina V.A., Melnik O.V. Emotion Recognition from Micro-Expressions: Search for the Face and Eyes. 8th Mediterranean Conference on Embedded Computing (MECO) Proceedings. Budva, Montenegro. 2019. P. 632–635.
8. Kartynnik Yu., Ablavatski A., Grishchenko I., Grundmann M. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. CVPR Workshop on Computer Vision for Augmented and Virtual Reality 2019. IEEE, Long Beach, CA, 4 p. [accessed 2021 April 30], https://arxiv.org/abs/1907.06724.
9. Dokumentaciya po OpenCV. Sglazhivanie izobrazhenij [Elektronnyj resurs]. – URL: https://docs.opencv.org/master/d4/d13/tutorial_py_ filtering.html (дата обращения: 30.04.2021) (in Russian).
10. Davison A.K., Lansley C., Costen N., Tan K., Moi Hoon Yap SAMM: A Spontaneous Micro-Facial Movement Dataset. In IEEE Transactions on Affective Computing. 2018. V. 9. № 1. P. 116–129.