Skip to main content
SpringerLink
Log in

Towards Machine Learning-Based Emotion Recognition from Multimodal Data

  • Conference paper
  • First Online:
Frontiers of ICT in Healthcare

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 519))

Abstract

Understanding human emotion is vital to communicate effectively with others, monitor patients, analyse behaviour, and keep an eye on those who are vulnerable. Emotion recognition is essential to achieve a complete human-machine interoperability experience. Artificial intelligence, mainly machine learning (ML), have been used in recent years to improve the model for recognising emotions from a single type of data. A multimodal system has been proposed that uses text, facial expressions, and speech signals to identify emotions in this work. The MobileNet architecture is used to predict emotion from facial expressions, and different ML classifiers are used to predict emotion from text and speech signals in the proposed model. The Facial Expression Recognition 2013 (FER2013) dataset has been used to recognise emotion from facial expressions, whilst the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset was used for both text and speech emotion recognition. The proposed ensemble technique consisting of random forest, extreme gradient boosting, and multi-layer perceptron achieves an accuracy of 70.67%, which is better than the unimodal approaches used.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 279.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Al Banna MH et al (2021) Attention-based bi-directional long-short term memory network for earthquake prediction. IEEE Access 9:56589–56603

    Article  Google Scholar 

  2. Al Nahian MJ, Ghosh T et al (2020) Towards artificial intelligence driven emotion aware fall monitoring framework suitable for elderly people with neurological disorder. In: Proceeding of Brain Information, pp 275–286

    Google Scholar 

  3. Al Nahian MJ et al (2021) Towards an accelerometer-based elderly fall detection system using cross-disciplinary time series features. IEEE Access 9:39413–31

    Article  Google Scholar 

  4. Basu S, Chakraborty J, Aftabuddin M (2017) Emotion recognition from speech using convolutional neural network with recurrent neural network architecture. In: Proceeding of ICCES, pp 333–336

    Google Scholar 

  5. Bertero D, Fung P (2017) A first look into a convolutional neural network for speech emotion detection. In: Proceeding of ICASSP, pp 5115–5119

    Google Scholar 

  6. Biswas M, Tania MH, Kaiser MS et al (2021) ACCU3RATE: a mobile health application rating scale based on user reviews. PloS one 16(12):e0258050

    Article  Google Scholar 

  7. Biswas M et al (2021) An xai based autism detection: the context behind the detection. In: Proceeding of Brain Information, pp 448–459

    Google Scholar 

  8. Choi WY, Song KY, Lee CW (2018) Convolutional attention networks for multimodal emotion recognition from speech and text data. In: Proceeding of challenge-HML, pp 28–34

    Google Scholar 

  9. Deepa B et al (2022) Pattern descriptors orientation and map firefly algorithm based brain pathology classification using hybridized machine learning algorithm. IEEE Access 10:3848–3863

    Article  Google Scholar 

  10. Fabietti M, Mahmud M, Lotfi A (2021) Anomaly detection in invasively recorded neuronal signals using deep neural network: effect of sampling frequency. In: Proceeding of AII, pp 79–91 (2021)

    Google Scholar 

  11. Fabietti M, Mahmud M, Lotfi A (2022) Channel-independent recreation of artefactual signals in chronically recorded local field potentials using machine learning. Brain Inform 9(1):1–17

    Article  Google Scholar 

  12. Fabietti M et al (2020) Artifact detection in chronically recorded local field potentials using long-short term memory neural network. Proceeding AICT 2020:1–6

    Google Scholar 

  13. Faria TH et al (2021) Smart city technologies for next generation healthcare. In: Data-driven mining, learning and analytics for secured smart cities, pp 253–274

    Google Scholar 

  14. Ghosh T et al (2021) Artificial intelligence and internet of things in screening and management of autism spectrum disorder. Sustain Cities Soc 74:103189

    Article  Google Scholar 

  15. Ghosh T et al (2021) An attention-based mood controlling framework for social media users. In: Proceeding of brain information, pp 245–256

    Google Scholar 

  16. Ghosh T et al (2021) A hybrid deep learning model to predict the impact of covid-19 on mental health form social media big data. Preprints 2021(2021060654)

    Google Scholar 

  17. Herzig J et al (2017) Emotion detection from text via ensemble classification using word embeddings. In: Proceeding ICTIR, pp 269–272

    Google Scholar 

  18. Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recognit Lett 120:69–74

    Article  Google Scholar 

  19. Jain N et al (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recognit. Lett. 115:101–106

    Article  Google Scholar 

  20. Kumar I et al (2022) Dense tissue pattern characterization using deep neural network. Cogn Comput 1–24 (2022) [ePub ahead of print]

    Google Scholar 

  21. Lalotra GS, Kumar V, Bhatt A, Chen T, Mahmud M (2022) Iretads: an intelligent real-time anomaly detection system for cloud communications using temporal data summarization and neural network. Secur Commun Netw 2022:9149164

    Article  Google Scholar 

  22. Mahmud F, Islam B, Hossain A, Goala PB (2018) Facial region segmentation based emotion recognition using k-nearest neighbors. In: Proceeding ICIET, pp 1–5 (2018)

    Google Scholar 

  23. Mahmud M, Kaiser MS, McGinnity TM, Hussain A (2021) Deep learning in mining biological data. Cognit Comput 13(1):1–33

    Article  Google Scholar 

  24. Mahmud M et al (2018) Applications of deep learning and reinforcement learning to biological data. IEEE Trans Neural Netw Learn Syst 29(6):2063–2079

    Article  Google Scholar 

  25. Mammoottil MJ, Kulangara LJ, Cherian AS, Mohandas P, Hasikin K, Mahmud M (2022) Detection of breast cancer from five-view thermal images using convolutional neural networks. J Healthc Eng 2022:4295221

    Article  Google Scholar 

  26. Nawar A, Toma NT, Al Mamun S et al (2021) Cross-content recommendation between movie and book using machine learning. In: Proceeding AICT, pp 1–6

    Google Scholar 

  27. Patwardhan AS (2017) Multimodal mixed emotion detection. In: Proceeding of ICCES, pp 139–143

    Google Scholar 

  28. Paul A et al (2022) Inverted bell-curve-based ensemble of deep learning models for detection of covid-19 from chest x-rays. Neural Comput Appl 1–15

    Google Scholar 

  29. Prakash N et al (2021) Deep transfer learning covid-19 detection and infection localization with superpixel based segmentation. Sustain Cities Soc 75:103252

    Article  Google Scholar 

  30. Satt A, Rozenberg S, Hoory R (2017) Efficient emotion recognition from speech using deep learning on spectrograms. In: Interspeech, pp 1089–1093

    Google Scholar 

  31. Satu M et al (2020) Towards improved detection of cognitive performance using bidirectional multilayer long-short term memory neural network. In: Proceeding of Brain Information, pp 297–306

    Google Scholar 

  32. Satu MS et al (2021) Tclustvid: a novel machine learning classification model to investigate topics and sentiment in covid-19 tweets. Knowl-Based Syst 226:107126

    Article  Google Scholar 

  33. Seal D, Roy UK, Basak R (2020) Sentence-level emotion detection from text based on semantic rules. In: Information and communication technology for sustainable development, pp 423–430

    Google Scholar 

  34. Sebastian J, Pierucci P et al (2019) Fusion techniques for utterance-level emotion recognition combining speech and transcripts. In: Interspeech, pp 51–55

    Google Scholar 

  35. Shrivastava K, Kumar S, Jain DK (2019) An effective approach for emotion detection in multimedia text data using sequence based convolutional neural network. Multimed Tools Appl 78(20):29607–29639

    Article  Google Scholar 

  36. Shu L, Xie J, Yang M, Li Z, Li Z, Liao D, Xu X, Yang X (2018) A review of emotion recognition using physiological signals. Sensors 18(7):2074

    Article  Google Scholar 

  37. Watkins J, Fabietti M, Mahmud M (2020) Sense: a student performance quantifier using sentiment analysis. In: Proceeding of IJCNN, pp 1–6

    Google Scholar 

Download references

Acknowledgements

MM and MSK are supported by the DIVERSASIA project (618615-EPP-1-2020-1-UKEPPKA2-CBHEJP) funded by the European Commission under the Erasmus+ programme.

Author information

Authors and Affiliations

  1. Department of Information and Communication Technology, Bangladesh University of Professionals, Dhaka, Bangladesh, India

    Md. Faiyaz Shahriar, Md. Safkat Azad Arnab, Munia Sarwat Khan & Safwon Sadif Rahman

  2. Department of Computer Science, Nottingham Trent University, Clifton, Nottingham, NG11 8NS, UK

    Mufti Mahmud

  3. MTIF, Nottingham Trent University, Clifton, Nottingham, NG11 8NS, UK

    Mufti Mahmud

  4. CIRC, Nottingham Trent University, Clifton, Nottingham, NG11 8NS, UK

    Mufti Mahmud

  5. IIT, Jahangirnagar University, Savar, 1342, Bangladesh, India

    M. Shamim Kaiser

Authors
  1. Md. Faiyaz Shahriar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Md. Safkat Azad Arnab
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Munia Sarwat Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Safwon Sadif Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mufti Mahmud
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. M. Shamim Kaiser
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mufti Mahmud .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, University of Kalyani, Kalyani, West Bengal, India

    Jyotsna Kumar Mandal

  2. Department of Computer Science and Engineering, School of Computational Science, Maulana Abul Kalam Azad University of Technology, Kolkata, West Bengal, India

    Debashis De

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shahriar, M.F., Arnab, M.S.A., Khan, M.S., Rahman, S.S., Mahmud, M., Kaiser, M.S. (2023). Towards Machine Learning-Based Emotion Recognition from Multimodal Data. In: Mandal, J.K., De, D. (eds) Frontiers of ICT in Healthcare . Lecture Notes in Networks and Systems, vol 519. Springer, Singapore. https://doi.org/10.1007/978-981-19-5191-6_9

Download citation

Publish with us

Policies and ethics

Search

Navigation