Skip to main content
SpringerLink
Log in

Emotion Recognition Based on Wireless, Physiological and Audiovisual Signals: A Comprehensive Survey

  • Conference paper
  • First Online:
Proceedings of 2nd International Conference on Smart Computing and Cyber Security (SMARTCYBER 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 395))

Abstract

Recognizing emotions are one of the most widespread aspects of investigation among investigators today. Among the procedures that are exploited to determine one’s emotional situation (happy, sad, etc.), there are such as wireless, physiological and audiovisual signals. In this review, we attempted to summarize these techniques. Nevertheless, few articles have employed wireless signals for the recognition of emotions. The flow diagram of emotion recognition is identical for all three approaches, hence the methods are various. The surveys in this study are seldom as all the surveys we have endeavored to compare all of the techniques and discover the best technique. In the wireless signal community, the use of Radio Frequency (RF) signals for sensing rather than the use of a sensor body is exploited in physiological techniques. Furthermore, manipulate the voice and facial expression to hide or mask emotions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Salvendy G (1994) HCI international ‘93: 5th international conference on human-computer interaction. ACM SIGCHI Bull 26(4):76–77

    Google Scholar 

  2. Hosseini S, Naghibi-Sistani M (2011) Emotion recognition method using entropy analysis of EEG signals. Int J Image Graph Signal Process 3(5):30–36. https://doi.org/10.5815/ijigsp.2011.05.05

  3. Kwon O, Chan K, Hao J, Lee T (2003) Emotion recognition by speech signals. Institute for Neural Computation, University of California, San Diego, USA

    Google Scholar 

  4. Alhalaseh R, Alasasfeh S (2020) Machine-learning-based emotion recognition system using EEG signals. Computers 9(4):95. https://doi.org/10.3390/computers9040095

  5. Cowie R et al (2001) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18(1):32–80. https://doi.org/10.1109/79.911197

  6. Ben M, Lachiri Z (2017) Emotion classification in arousal valence model using MAHNOB-HCI database. Int J Adv Comput Sci Appl 8(3). https://doi.org/10.14569/ijacsa.2017.080344

  7. Hamidi M (2012) Emotion recognition from Persian speech with neural network. Int J Artific Intell Appl 3(5):107–112. https://doi.org/10.5121/ijaia.2012.3509

  8. Crookall D, Sandole DJD, Sandole-Staroste I (eds) (1987) Conflict management and problem solving: interpersonal to international applications. Frances Pinter, New York: New York University Press (25 Floral Str, London WC2E 9DS, UK; Washington Square, New York, NY 10003, USA, London. Simulat Games 20(1):107–108, 1989 (Book Reviews Miscellaneous Reviews). https://doi.org/10.1177/104687818902000150

  9. Barrett L (1998) Discrete emotions or dimensions? The role of valence focus and arousal focus. Cognit Emot 12(4):579–599. https://doi.org/10.1080/026999398379574

  10. Koelstra S et al (2012) DEAP: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31. https://doi.org/10.1109/t-affc.2011.15

  11. Yannakakis G, Isbister K, Paiva A, Karpouzis K (2014) Guest editorial: emotion in games. IEEE Trans Affect Comput 5(1):1–2. https://doi.org/10.1109/taffc.2014.2313816

  12. Kim J, Andre E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067–2083. https://doi.org/10.1109/tpami.2008.26

  13. Jerritta S, Murugappan M, Nagarajan R, Wan K (2011) Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th international colloquium on signal processing and its applications, pp 410–415. https://doi.org/10.1109/CSPA.2011.5759912

  14. Kahou S et al (2015) EmoNets: multimodal deep learning approaches for emotion recognition in video. J Multimod User Interf 10(2):99–111. https://doi.org/10.1007/s12193-015-0195-2

  15. Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37

    Article  Google Scholar 

  16. Human face processing: from recognition to emotion. Psychophysiology 50:S20–S21 (2013). https://doi.org/10.1111/psyp.12117

  17. Quintana DS, Guastella AJ, Outhred T, Hickie IB, Kemp AH (2012) Heart rate variability is associated with emotion recognition: direct evidence for a relationship between the autonomic nervous system and social cognition. Int J Psychophysiol 86(2):168–172

    Article  Google Scholar 

  18. Duan R-N, Zhu J-Y, Lu B-L (2013) Differential entropy feature for EEG-based emotion classification. In: 2013 6th international IEEE/EMBS conference on neural engineering (NER)

    Google Scholar 

  19. Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Autonom Mental Develop 7(3):162–175

    Google Scholar 

  20. Ghinea G, Timmerer C, Lin W, Gulliver SR (2014) Mulsemedia. ACM Trans Multimed Comput Commun Appl 11(1s):1–23

    Google Scholar 

  21. Covaci A, Zou L, Tal I, Muntean G-M, Ghinea G (2019) Is Multimedia multisensorial?—a review of mulsemedia systems. ACM Comput Surv 51(5):1–35

    Article  Google Scholar 

  22. Kamdar MR, Wu MJ (2015) Prism: a data-driven platform for monitoring mental health. In: Biocomputing 2016

    Google Scholar 

  23. Feng H, Golshan HM, Mahoor MH (2018) A wavelet-based approach to emotion classification using EDA signals. Expert Syst Appl 112:77–86

    Article  Google Scholar 

  24. Abdelnasser H, Youssef M, Harras KA (2015) WiGest: a ubiquitous WiFi-based gesture recognition system. In: 2015 IEEE conference on computer communications (INFOCOM), 2015

    Google Scholar 

  25. Sigg S, Scholz M, Shi S, Ji Y, Beigl M (2014) RF-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals. IEEE Trans Mobile Comput 13(4):907–920

    Google Scholar 

  26. Pu Q, Gupta S, Gollakota S, Patel S (2013) Whole-home gesture recognition using wireless signals. In: Proceedings of the 19th annual international conference on Mobile computing & networking—MobiCom ‘13

    Google Scholar 

  27. Raja M, Sigg S (2016) Applicability of RF-based methods for emotion recognition: a survey. In: 2016 IEEE international conference on pervasive computing and communication workshops (PerCom Workshops)

    Google Scholar 

  28. Zhao M, Adib F, Katabi D (2016) Emotion recognition using wireless signals. In: Proceedings of the 22nd annual international conference on mobile computing and networking

    Google Scholar 

  29. Kreibig SD (2010) Autonomic nervous system activity in emotion: a review. Biol Psychol 84(3):394–421

    Article  Google Scholar 

  30. Nussinovitch U, Elishkevitz KP, Katz K, Nussinovitch M, Segev S, Volovitz B, Nussinovitch N (2011) Reliability of ultra-short ECG indices for heart rate variability. Ann Noninvasive Electrocardiol 16(2):117–122

    Article  Google Scholar 

  31. Raina R, Battle A, Lee H, Packer B, Ng AY (2007) Self-taught learning. In: Proceedings of the 24th international conference on machine learning—ICML ‘07

    Google Scholar 

  32. Ohkura M, Hamano M, Watanabe H, Aoto T (2011) Measurement of Wakuwaku feeling of interactive systems using biological signals. Emotion Eng 327–343

    Google Scholar 

  33. Goenaga S, Navarro L, Quintero MCG, Pardo M (2020) Imitating human emotions with a NAO robot as interviewer playing the role of vocational tutor. Electronics 9(6), 971

    Google Scholar 

  34. Salzman CD, Fusi S (2010) Emotion, cognition, and mental state representation in amygdala and prefrontal cortex. Annu Rev Neurosci 33(1):173–202

    Article  Google Scholar 

  35. Torres EP, Torres EA, Hernández-Álvarez M, Yoo SG (2020) EEG-based BCI emotion recognition: a survey. MDPI, 07-Sep-2020 [Online]. https://www.mdpi.com/1424-8220/20/18/5083/htm. Accessed 30 Apr 2021

  36. Zheng W-L, Zhu J-Y, Lu B-L, Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2017.2712143

  37. Luo Q (2014) Speech emotion recognition in E-learning system by using general regression neural network. In: Future energy, environment and materials

    Google Scholar 

  38. Koolagudi SG, Rao KS (2012) Emotion recognition from speech: a review. Int J Speech Technol 15(2):99–117

    Article  Google Scholar 

  39. Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features, and methods. Speech Commun 48(9):1162–1181

    Article  Google Scholar 

  40. Wioleta S (2013) Using physiological signals for emotion recognition. In: 2013 6th international conference on human system interactions (HSI)

    Google Scholar 

  41. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175–1191

    Article  Google Scholar 

  42. Panoulas KJ, Hadjileontiadis LJ, Panas SM (2020) Brain-computer interface (BCI): types, processing perspectives and applications. In: Multimedia services in intelligent environments, pp 299–321

    Google Scholar 

  43. Ekman P (1992) Are there basic emotions? Psychol Rev 99(3):550–553

    Article  Google Scholar 

  44. Verma GK, Tiwary US (2016) Affect representation and recognition in 3D continuous valence–arousal–dominance space. Multimed Tools Appl 76(2):2159–2183

    Article  Google Scholar 

  45. Bălan O, Moise G, Moldoveanu A, Leordeanu M, Moldoveanu F (2019) Fear level classification based on emotional dimensions and machine learning techniques. Sensors 19(7):1738

    Article  Google Scholar 

  46. Zhao M, Adib F, Katabi D (2016) Emotion recognition using wireless signals. In: The 22nd annual international conference on mobile computing and networking (Mobicom’16)

    Google Scholar 

  47. Hyvärinen A, Oja E (1998) Independent component analysis by general nonlinear Hebbian-like learning rules. Signal Process 64(3):301–313

    Article  Google Scholar 

  48. Mehmood RM, Lee HJ (2015) Emotion classification of EEG brain signal using SVM and KNN. In: IEEE international conference on multimedia & expo workshops (ICMEW), pp 1–5. https://doi.org/10.1109/ICMEW.2015.7169786

  49. Henia WMB, Lachiri Z (2017) Emotion classification in arousal-valence dimension using discrete affective keywords tagging. In: 2017 international conference on engineering & MIS (ICEMIS), pp 1–6. https://doi.org/10.1109/ICEMIS.2017.8272991

  50. Yadava M, Kumar P, Saini R, Roy PP, Dogra DP (2017) Analysis of EEG signals and its application to neuromarketing. Multimed Tools Appl 76(18):19087–19111

    Google Scholar 

  51. Santamaria-Granados L, Munoz-Organero M, Ramirez-González G, Abdulhay E, Arunkumar N (2019) Using deep convolutional neural network for emotion detection on a physiological signals dataset (AMIGOS). IEEE Access 7:57–67. https://doi.org/10.1109/ACCESS.2018.2883213

    Article  Google Scholar 

  52. Katsigiannis S, Ramzan N (2018) DREAMER: a database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform 22(1):98–107

    Article  Google Scholar 

  53. Li Y, Zheng W, Cui Z, Zong Y, Ge S (2018) EEG emotion recognition based on graph regularized sparse linear regression. Neural Process Lett 49(2):555–571

    Article  Google Scholar 

  54. Tivatansakul S, Ohkura M (2016) Emotion recognition using ECG Signals with local pattern description methods. Int J Affect Eng 15(2):51–61

    Article  Google Scholar 

  55. Pandolfi E, Sacripante R, Cardini F (2016) Food-induced emotional resonance improves emotion recognition. Plos One 11(12)

    Google Scholar 

  56. Zhou G, Hansen JHL, Kaiser JF (2001) Nonlinear feature based classification of speech under stress. IEEE Trans Speech Audio Process 9(3):201–216

    Article  Google Scholar 

  57. Bhavan A, Chauhan P, Shah RR (2019) Bagged support vector machines for emotion recognition from speech. Knowledge-Based Syst 184:104886

    Google Scholar 

  58. De Silva LC, Miyasato T, Nakatsu R (1997) Facial emotion recognition using multi-modal information. In: Proceedings of ICICS, 1997 international conference on information, communications and signal processing. Theme: trends in information systems engineering and wireless multimedia communications (Cat., 1997), vol 1, pp 397–401. https://doi.org/10.1109/ICICS.1997.647126

  59. Ko B (2018) A brief review of facial emotion recognition based on visual information. Sensors 18(2):401

    Article  Google Scholar 

  60. Katabi D (2014) Tracking people and monitoring their vital signs using body radio reflections. In: Proceedings of the 2014 workshop on physical analytics—WPA ‘14

    Google Scholar 

  61. Kieser R, Reynisson P, Mulligan TJ (2005) Definition of signal-to-noise ratio and its critical role in split-beam measurements. ICES J Mar Sci 62(1):123–130

    Article  Google Scholar 

  62. Raja M, Sigg S (2017) RFexpress!—exploiting the wireless network edge for RF-based emotion sensing. In: 2017 22nd IEEE international conference on emerging technologies and factory automation (ETFA)

    Google Scholar 

  63. Xu T, Yin R, Shu L, Xu X (2019) Emotion recognition using frontal EEG in VR affective scenes. In: 2019 IEEE MTT-S international microwave biomedical conference (IMBioC)

    Google Scholar 

  64. Nie Y, Wu Y, Yang ZY, Sun G, Yang Y, Hong X (2017) Emotional evaluation based on SVM. In: Proceedings of the 2017 2nd international conference on automation, mechanical control and computational engineering (AMCCE 2017)

    Google Scholar 

  65. He C, Yao Y, Ye X (2016) An emotion recognition system based on physiological signals obtained by wearable sensors. In: Wearable sensors and robots, pp 15–25

    Google Scholar 

  66. Kaur B, Singh D, Roy PP (2016) A Novel framework of EEG-based user identification by analyzing music-listening behavior. Multimed Tools Appl 76(24):25581–25602

    Article  Google Scholar 

  67. Zhao L, Yang L, Shi H, Xia Y, Li F, Liu C (2017) Evaluation of consistency of HRV indices change among different emotions. In: 2017 Chinese Automation Congress (CAC)

    Google Scholar 

  68. Sznajder M, Lukowska M (2018) Python online and offline ECG QRS detector based on the pan-Tomkins algorithm. Zenodo, Tech Rep

    Google Scholar 

  69. Alva MY, Nachamai M, Paulose J (2015) A comprehensive survey on features and methods for speech emotion detection. In: 2015 IEEE international conference on electrical, computer and communication technologies (ICECCT)

    Google Scholar 

  70. Kim Y, Lee H, Provost EM (2013) Deep learning for robust feature generation in audiovisual emotion recognition. In: IEEE international conference on acoustics, speech and signal processing IEEE, 2013, pp 3687–3691

    Google Scholar 

  71. Kim J, Andre E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067–2083

    Article  Google Scholar 

  72. Petrantonakis PC, Hadjileontiadis LJ (2010) Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Trans Affect Comput 1(2):81–97

    Article  Google Scholar 

  73. Torres EP, Torres EA, Hernandez-Alvarez M, Yoo SG (2020) Emotion recognition related to stock trading using machine learning algorithms with feature selection. IEEE Access 8:199719–199732

    Article  Google Scholar 

  74. Emotion recognition using wearables: a systematic literature review—work-in-progress. IEEE Xplore [Online]. https://ieeexplore.ieee.org/document/9156096. Accessed 30 Apr 2021

  75. Chen S, Jiang K, Hu H, Kuang H, Yang J, Luo J, Chen X, Li Y (2021) Emotion recognition based on skin potential signals with a portable wireless device. Sensors 21(3):1018

    Article  Google Scholar 

  76. Lan Y-T, Liu W, Lu B-L (2020) Multimodal emotion recognition using deep generalized canonical correlation analysis with an attention mechanism. In: 2020 International Joint Conference on Neural Networks (IJCNN)

    Google Scholar 

  77. Sarkar P, Etemad A (2020) Self-supervised learning for ECG-based emotion recognition. In: ICASSP 2020–2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 3217–3221. https://doi.org/10.1109/ICASSP40776.2020.9053985

  78. Kahou SE, Bouthillier X, Lamblin P, Gulcehre C, Michalski V, Konda K, Jean S, Froumenty P, Dauphin Y, Boulanger-Lewandowski N, Ferrari RC, Mirza M, Warde-Farley D, Courville A, Vincent P, Memisevic R, Pal C, Bengio Y (2015) EmoNets: Multimodal deep learning approaches for emotion recognition in video. J Multimod User Interf 10(2), 99–111

    Google Scholar 

  79. Chauhan K, Sharma KK, Varma T (2021) Speech emotion recognition using convolution neural networks. In: 2021 international conference on artificial intelligence and smart systems (ICAIS)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, University of Science and Technology of China, Hefei, China

    Aisha Alabsi, Wei Gong & Ammar Hawbani

Authors
  1. Aisha Alabsi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ammar Hawbani
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computer Engineering, KIIT Deemed University, Bhubaneswar, India

    Prasant Kumar Pattnaik

  2. Department of Computer Information Engineering, Dongseo University, Busan, Korea (Republic of)

    Mangal Sain

  3. Smart Computing Department, Kyungdong University Global Campus, Gangwondo, Korea (Republic of)

    Ahmed A. Al-Absi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Alabsi, A., Gong, W., Hawbani, A. (2022). Emotion Recognition Based on Wireless, Physiological and Audiovisual Signals: A Comprehensive Survey. In: Pattnaik, P.K., Sain, M., Al-Absi, A.A. (eds) Proceedings of 2nd International Conference on Smart Computing and Cyber Security. SMARTCYBER 2021. Lecture Notes in Networks and Systems, vol 395. Springer, Singapore. https://doi.org/10.1007/978-981-16-9480-6_13

Download citation

Publish with us

Policies and ethics

Search

Navigation