Abstract
Humans use rich facial expressions to indicate unpleasant emotions, such as pain. Automatic pain intensity estimation is useful in a variety of applications in social and medical domains. However, the existing pain intensity estimation approaches are limited to either classifying the discrete intensity levels in pain or estimating the continuous pain intensities without considering the key-frame. The first approach suffers from abnormal fluctuations while estimating the pain intensity levels. Further, continuous pain estimation approaches suffer from low prediction capabilities. Hence, in this paper, we propose a deep hybrid network based approach to automatically estimate the continuous pain intensities by incorporating spatiotemporal information. Our approach consists of two key components, namely key-frame analyser and temporal analyser. We use one conventional and two recurrent convolutional neural networks to design key-frame and temporal analysers, respectively. Further, the evaluation on a benchmark dataset shows that our model can estimate the continuous emotions better than existing state-of-the-art methods.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
De Jeses, O., Hagan, M.T.: Backpropagation through time for a general class of recurrent network. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 2001, (Cat. No. 01CH37222), vol. 4, pp. 2638–2643. IEEE (2001)
Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., Pal, C.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 467–474. ACM (2015)
Ekman, R.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Cary (1997)
Florea, C., Florea, L., Vertan, C.: Learning pain from emotion: transferred hot data representation for pain intensity estimation. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8927, pp. 778–790. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16199-0_54
Frank, M.G., Ekman, P., Friesen, W.V.: Behavioral markers and recognizability of the smile of enjoyment. J. Pers. Soc. Psychol. 64(1), 83 (1993)
Hammal, Z., Cohn, J.F.: Automatic detection of pain intensity. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, pp. 47–52. ACM (2012)
Hong, X., Zhao, G., Zafeiriou, S., Pantic, M., Pietikäinen, M.: Capturing correlations of local features for image representation. Neurocomputing 184, 99–106 (2016)
Kaltwang, S., Rudovic, O., Pantic, M.: Continuous pain intensity estimation from facial expressions. In: Bebis, G., et al. (eds.) ISVC 2012. LNCS, vol. 7432, pp. 368–377. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33191-6_36
Kalkhoran, H.M., Fatemizadeh, E.: Pain level estimation in video sequences of face using incorporation of statistical features of frames. In: 2015 9th Iranian Conference on Machine Vision and Image Processing (MVIP), pp. 172–175. IEEE (2015)
Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3367–3375 (2015)
Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: Face and Gesture 2011, pp. 57–64. IEEE (2011)
Martinez, L., Rosalind Picard, D.: Personalized automatic estimation of self-reported pain intensity from facial expressions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 70–79 (2017)
Mendes, C.C.T., Frémont, V., Wolf, D.F.: Exploiting fully convolutional neural networks for fast road detection. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3174–3179. IEEE (2016)
Neshov, N., Manolova, A.: Pain detection from facial characteristics using supervised descent method. In: 2015 IEEE 8th International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), vol. 1, pp. 251–256. IEEE (2015)
Ondruska, P., Posner, I.: Deep tracking: seeing beyond seeing using recurrent neural networks. In: Thirtieth AAAI Conference on Artificial Intelligence, March 2016
Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. Pain 51(3), 297–306 (1992)
Prkachin, K.M., Solomon, P.E.: The structure, reliability and validity of pain expression: evidence from patients with shoulder pain. Pain 139(2), 267–274 (2008)
Rathee, N., Ganotra, D.: A novel approach for pain intensity detection based on facial feature deformations. J. Vis. Commun. Image Represent. 33, 247–254 (2015)
Roy, S.D., Bhowmik, M.K., Saha, P., Ghosh, A.K.: An approach for automatic pain detection through facial expression. Procedia Comput. Sci. 84, 99–106 (2016)
Rudovic, O., Pavlovic, V., Pantic, M.: Automatic pain intensity estimation with heteroscedastic conditional ordinal random fields. In: Bebis, G., et al. (eds.) ISVC 2013. LNCS, vol. 8034, pp. 234–243. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41939-3_23
Thuseethan, S., Rajasegarar, S., Yearwood, J.: Detecting micro-expression intensity changes from videos based on hybrid deep CNN. In: Yang, Q., Zhou, Z.-H., Gong, Z., Zhang, M.-L., Huang, S.-J. (eds.) PAKDD 2019. LNCS (LNAI), vol. 11441, pp. 387–399. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16142-2_30
Thuseethan, S., Rajasegarar, S., Yearwood, J.: Emotion intensity estimation from video frames using deep hybrid convolutional neural networks. In: 2019 International Joint Conference on Neural Networks (IJCNN). IEEE (2019)
Wang, F., et al.: Regularizing face verification nets for pain intensity regression. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 1087–1091. IEEE (2017)
Wu, Z., Wang, X., Jiang, Y.G., Ye, H., Xue, X.: Modeling spatial-temporal clues in a hybrid deep learning framework for video classification. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 461–470. ACM (2015)
Zhao, R., Gan, Q., Wang, S., Ji, Q.: Facial expression intensity estimation using ordinal information. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3466–3474 (2016)
Zhou, J., Hong, X., Su, F., Zhao, G.: Recurrent convolutional neural network regression for continuous pain intensity estimation in video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 84–92 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Thuseethan, S., Rajasegarar, S., Yearwood, J. (2019). Deep Hybrid Spatiotemporal Networks for Continuous Pain Intensity Estimation. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Lecture Notes in Computer Science(), vol 11955. Springer, Cham. https://doi.org/10.1007/978-3-030-36718-3_38
Download citation
DOI: https://doi.org/10.1007/978-3-030-36718-3_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36717-6
Online ISBN: 978-3-030-36718-3
eBook Packages: Computer ScienceComputer Science (R0)