Skip to main content
SpringerLink
Log in

A Short Survey of Elucidating the Emotion Recognition Methodologies Using Facial Images and EEG Signals

  • Conference paper
  • First Online:
Advanced Computational and Communication Paradigms (ICACCP 2023)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 535))

  • 238 Accesses

Abstract

Since the high demand for Human–Computer Interaction (HCI), developing an automated model for recognizing facial gestures or emotions becomes challenging. Some experts have used facial images as a constructive part of recognizing different emotions of humans. Simultaneously, emotions can also be classified by using some electrical signals such as Electrocardiogram (ECG), Electroencephalogram (EEG), Electromyogram (EMG), and so on. Among all these signals, EEG captures the major information of brain activities. Recently, along with face images, EEG signal plays a pivotal role in emotion recognition. Though these methods have been estimated the promising results, the real-world implications and diverse feature identification of human states are still in place to gain more attention. Considering these factors, this survey aims to review the literature for analyzing emotion recognition performance using EEG signals and facial images. It also demonstrates the techniques that are utilized in machine learning and deep learning as well. The drawback of such implemented models is discussed, leading to future development. Subsequently, the survey part is followed by exploring the chronological review of the emotion recognition work, dataset utilization, methodologies employed, and experimental analysis with divergent metrics and features, and challenges. Finally, the research challenging factors provoke to raise the novel effective system for emotion recognition using facial images and EEG signals.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Yang B, Cao J, Ni R, Zhang Y (2018) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  2. Said Y, Barr M (2021) Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimedia Tools Appl 80:25241–25253

    Article  Google Scholar 

  3. Liliana DY, Basaruddin T, Widyanto MR, Oriza IID (2019) Fuzzy emotion: a natural approach to automatic facial expression recognition from psychological perspective using fuzzy system. Cogn Process 20:391–403

    Google Scholar 

  4. Saurav S, Saini R, Singh S (2021) EmNet: a deep integrated convolutional neural network for facial emotion recognition in the wild. Appl Intell 51:5543–5570

    Article  Google Scholar 

  5. An Y, Xu N, Qu Z (2021) Leveraging spatial-temporal convolutional features for EEG-based emotion recognition. Biomed Signal Process Control 69

    Google Scholar 

  6. Liu Y, Fu G (2021) Emotion recognition by deeply learned multi-channel textual and EEG features. Futur Gener Comput Syst 119:1–6

    Article  Google Scholar 

  7. Alhussein M (2016) Automatic facial emotion recognition using weber local descriptor for e-Healthcare system. Clust Comput 17:99–108

    Article  Google Scholar 

  8. Topic A, Russo M (2021) Emotion recognition based on EEG feature maps through deep learning network". Eng Sci Technol Int J 24(6):1442–1454

    Google Scholar 

  9. Hu M, Wang H, Wang X, Yang J, Wang R (2019) Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks. J Vis Commun Image Represent 59:176–185

    Article  Google Scholar 

  10. Xiaohua W, Muzi P, Lijuan P, Min H, Chunhua J, Fuji R (2019) Two-level attention with two-stage multi-task learning for facial emotion recognition. J Vis Commun Image Represent 62:217–225

    Article  Google Scholar 

  11. Gu H, Chen Q, Xing X, Zhao J, Li X (2019) Facial emotion recognition in deaf children: evidence from event-related potentials and event-related spectral perturbation analysis. Neurosci Lett 703:198–204

    Article  Google Scholar 

  12. Zhang H, Jolfaei A, Alazab M (2019) A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access 7:159081–159089

    Article  Google Scholar 

  13. Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018

    Article  MathSciNet  Google Scholar 

  14. Mohan K, Seal A, Krejcar O, Yazidi A (2021) Facial expression recognition using local gravitational force descriptor-based deep convolution neural networks. IEEE Trans Instrum Measure 70, 1–12 (5003512)

    Google Scholar 

  15. Ullah Z, Qi L, Hasan A, Asim M (2022) Improved deep CNN-based two stream super resolution and hybrid deep model-based facial emotion recognition. Eng Appl Artif Intell 116

    Google Scholar 

  16. Devi DAS, Satyanarayana CH (2021) An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimedia Tools Appl 80:17543–17568

    Article  Google Scholar 

  17. Kumar RJR, Sundaram M, Arumugam N (2021) Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine. Vis Comput 37:2315–2329

    Article  Google Scholar 

  18. Mert A, Akan A (2018) Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Appl 21:81–89

    Article  MathSciNet  Google Scholar 

  19. Taran S, Bajaj V (2019) Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Comput Methods Programs Biomed 173:157–165

    Article  Google Scholar 

  20. Salankar N, Mishra P, Garg L (2021) Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomed Signal Process Control 65

    Google Scholar 

  21. Maheshwari D, Ghosh SK, Tripathy RK, Sharma M, Acharya UR (2021) Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals. Comput Biol Med 134

    Google Scholar 

  22. Gao Q, Wang C-H, Wang Z, Song X-L, Dong E-Z, Song Y (2020) EEG based emotion recognition using fusion feature extraction method. Multimedia Tools Appl 79:27057–27074

    Article  Google Scholar 

  23. Patel P, Raghunandan R, Annavarapu RN (2021) EEG-based human emotion recognition using entropy as a feature extraction measure. Brain Inf 8(20)

    Google Scholar 

  24. Iyer A, Das SS, Teotia R, Maheshwari S, Sharma RR (2022) CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings. Multimedia Tools Appl

    Google Scholar 

  25. Zhang J, Zhou Y, Liu Y (2020). EEG-based emotion recognition using an improved radial basis function neural network. J Ambient Intell Humanized Comput

    Google Scholar 

  26. Zhang H (2020) Expression-EEG based collaborative multimodal emotion recognition using deep autoencoder. IEEE Access 8:164130–164143

    Article  Google Scholar 

  27. Chao H, Dong L (2021) Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals. IEEE Sens J 21(2):2024–2034

    Article  Google Scholar 

  28. Wu D, Zhang J, Zhao Q (2020) Multimodal fused emotion recognition about expression-EEG interaction and collaboration using deep learning. IEEE Access 8:133180–133189

    Article  Google Scholar 

  29. Pane ES, Wibawa AD, Purnomo MH (2019) Improving the accuracy of EEG emotion recognition by combining valence lateralization and ensemble learning with tuning parameters. Cogn Process 20:405–417

    Article  Google Scholar 

  30. Pandey P, Seeja KR (2021) Subject independent emotion recognition system for people with facial deformity: an EEG based approach. J Ambient Intell Humaniz Comput 12:2311–2320

    Article  Google Scholar 

  31. Meshach WT, Hemajothi S, Anita EAM (2021) Real-time facial expression recognition for affect identification using multi-dimensional SVM. J Ambient Intell Humaniz Comput 12:6355–6365

    Article  Google Scholar 

  32. Alphonse AS, Dharma D (2018) Novel directional patterns and a generalized supervised dimension reduction system (GSDRS) for facial emotion recognition. Multimedia Tools Appl 77:9455–9488

    Article  Google Scholar 

  33. Momennezhad A (2018) EEG-based emotion recognition utilizing wavelet coefficients. Multimedia Tools Appl 77:27089–27106

    Article  Google Scholar 

  34. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2019) Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49(3):839–847

    Article  Google Scholar 

  35. Wang M, Huang Z, Li Y, Dong L, Pan H (2021) Maximum weight multi-modal information fusion algorithm of electroencephalographs and face images for emotion recognition. Comput Electr Eng 94

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering, Amity School of Engineering and Technology, Amity University Lucknow Campus, India

    Dilsheen Kaur & Anuradha Misra

  2. Information Technology, Indian Institute of Information Technology Allahabad, Prayagraj, India

    O. P. Vyas

Authors
  1. Dilsheen Kaur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anuradha Misra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. O. P. Vyas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dilsheen Kaur .

Editor information

Editors and Affiliations

  1. Department of Computer Applications, Sikkim Manipal Institute of Technology, Rangpo, Sikkim, India

    Samarjeet Borah

  2. Department of Electrical Engineering, IIT Delhi, Delhi, India

    Tapan K. Gandhi

  3. Department of Computer Science, University of Milan, Milano, Milano, Italy

    Vincenzo Piuri

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kaur, D., Misra, A., Vyas, O.P. (2023). A Short Survey of Elucidating the Emotion Recognition Methodologies Using Facial Images and EEG Signals. In: Borah, S., Gandhi, T.K., Piuri, V. (eds) Advanced Computational and Communication Paradigms . ICACCP 2023. Lecture Notes in Networks and Systems, vol 535. Springer, Singapore. https://doi.org/10.1007/978-981-99-4284-8_35

Download citation

Publish with us

Policies and ethics

Search

Navigation