O.V. Melnik1, V.A. Sablina2, A.D. Chernenko3

1–3 Ryazan State Radio Engineering University named after V.F. Utkina (Ryazan, Russia)

Spatiotemporal feature descriptors find applications in fields related to the processing of image sequences that describe time-evolving processes. One such process is the manifestation of microexpressions on a person's face. Microexpressions last less than half a second and reflect a person's true emotions.

The main stages of the software pipeline for detecting microfacial movements include the detection and selection of anthropometric facial points, extraction of spatiotemporal features, and classification of descriptor features of selected areas. This work explores possible ways of implementing the spatiotemporal feature extraction stage. The theoretical features of the Histogram of Oriented Optical Flow (HOOF) descriptor and its modification, Fuzzy HOOF (FHOOF), are considered, and the results of using the FHOOF feature descriptor in the developed software pipeline for detecting microfacial movements are presented. The experiments conducted and comparative analysis of the results of microfacial movement detection based on the LBP-TOP and FHOOF feature descriptors are described.

Theoretical and experimental studies of the spatiotemporal feature descriptor, FHOOF, as a fuzzy version of the HOOF descriptor used for recognizing facial microexpressions, demonstrated its potential application in the software pipeline for detecting microfacial movements. Experiments showed that the FHOOF feature descriptor achieved better results with a microfacial movement detection accuracy of up to 82 % compared to the previously studied LBP-TOP feature descriptor with a detection accuracy of up to 73 %.

Melnik O.V., Sablina V.A., Chernenko A.D. An application of the FHOOF feature descriptor for micro-facial movement detection. Biomedicine Radioengineering. 2023. V. 26. № 3. Р. 61-70. DOI: https://doi.org/10.18127/j15604136-202303-08 (In Russian)

1. Ekman P. Emotion in the Human Face. 2nd Edition. Malor Books. 2013. 456 p.
2. Facial Action Coding System (FACS) – A Visual Guidebook [Elektronnyy resurs]. – URL: https://imotions.com/blog/facial-action-coding-system/ (data obrashcheniya: 01.05.2023).
3. Burresi G., Victoria A. Sablina V.A. Micro-Facial Movement Detection Using LBP-TOP Descriptors for Landmark Based Regions. 10th Mediterranean Conference on Embedded Computing (MECO) Proceedings. Budva. Montenegro. 2021. P. 401 404.
4. Chernenko A., Ashapkina M., Sablina V., Alpatov A. Physical Activity Set Selection for Emotional State Harmonization Based on Facial Micro-Expression Analysis. Proceedings of the 32nd International Conference on Computer Graphics and Vision “GraphiCon”. 2022. P. 682–691.
5. Sablina V.A. Tekhnologiya obnaruzheniya mikrolitsevykh dvizheniy dlya vyyavleniya istinnykh emotsiy cheloveka. Biotekhnicheskiye. meditsinskiye i ekologicheskiye sistemy. izmeritelnyye ustroystva i robototekhnicheskiye kompleksy – Biomedsistemy 2022: Sb. tr. XXXV Vseros. nauch.-tekhn. konf. stud., mol. uchenykh i spets., 7–9 dekabrya 2022 g. pod obshch. red. V.I. Zhuleva. Ryazan: IP Konyakhin A.V. (Book Jet). 2022. S. 11–17. (in Russian).
6. Chaudhry R., Ravichandran A., Hager G., Vidal R. Histograms of Oriented Optical Flow and Binet-Cauchy Kernels on Nonlinear Dynamical Systems for the Recognition of Human Actions. IEEE Conference on Computer Vision and Pattern Recognition. Miami. FL. 2009. P. 1932–1939.
7. Happy S.L., Routray A., Fuzzy Histogram of Optical Flow Orientations for Micro-Expression Recognition. IEEE Transactions on Affective Computing. 2017. V. 10. № 3. P. 394–406.
8. Sablina V.A., Belyayeva K.A. Otslezhivaniye tochek lazernogo podsveta na posledovatelnosti izobrazheniy kalibrovochnogo obyekta. Vestnik Ryazanskogo gosudarstvennogo radiotekhnicheskogo universiteta. 2015. Vyp. 54. Ch. 2. S. 39–44. (in Russian).
9. Hast A., Sablina V.A., Sintorn I. M., Kylberg G. A Fast Fourier based Feature Descriptor and a Cascade Nearest Neighbour Search with an Efficient Matching Pipeline for Mosaicing of Microscopy Images. International Journal “Pattern Recognition and Image Analysis. 2018. V. 28. № 2. P. 261–272.
10. Sablina V.A., Yakovlev N.V. Obzor prostranstvenno-vremennykh deskriptorov priznakov dlya analiza vyrazheniy litsa. Biotekhnicheskiye. meditsinskiye i ekologicheskiye sistemy. izmeritelnyye ustroystva i robototekhnicheskiye kompleksy. Biomedsistemy 2020: Sb. tr. XXXIII Vseros. nauch.-tekhn. konf. stud., mol. uchenykh i spets., 9 11 dekabrya 2020 g. pod obshch. red. V.I. Zhuleva. Ryazan: IP Konyakhin A.V. (Book Jet). 2020. S. 292–295. (in Russian).
11. Nikiforov M.B., Sablina V.A., Chernenko A.D. Primeneniye algoritmov prostranstvenno-vremennykh deskriptorov priznakov dlya analiza mikrovyrazheniy litsa. 78 ya Nauchno-tekhnicheskaya konferentsiya Sankt-Peterburgskogo NTO RES im. A.S. Popova. posvyashchennaya Dnyu radio. Sb. materialov. SPb.: SPbGETU «LETI». 2023. S. 365–370. (in Russian).
12. Davison A.K., Lansley C., Costen N., Tan K., Moi Hoon Yap. SAMM: A Spontaneous Micro-Facial Movement Dataset. IEEE Transactions on Affective Computing. 2018. V. 9. № 1. P. 116–129.
13. Hong X., Xu Y., Zhao G. LBP-TOP: A Tensor Unfolding Revisit. in ACCV 2016. Lecture Notes in Computer Science. Springer. Cham. 2017. V. 10116. P. 513–527.
14. Face landmarks detection task guide [Elektronnyy resurs]. – URL: https://developers.google.com/mediapipe/solutions/vision/ face_landmarker (data obrashcheniya: 30.04.2023).