Skip to main content
SpringerLink
Log in

Transfer Learning and Recurrent Neural Networks for Automatic Arabic Sign Language Recognition

  • Conference paper
  • First Online:
The 8th International Conference on Advanced Machine Learning and Technologies and Applications (AMLTA2022) (AMLTA 2022)

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 113))

Abstract

Arabic Sign Language (ArSL) is the most utilized for hearing and speech impairments in Arab countries. The recognition system of ArSL could be an innovation to empower communication between the deaf and others. Recent advances in gesture recognition using deep learning and computer vision-based techniques have proved promising. Due to a lack of ArSL datasets, the ArSL dataset was created. The dataset was then expanded using augmentation methods. This paper aims to create an architecture based on both Transfer Learning (TL) models and Recurrent Neural network (RNN) models for recognizing ArSL. The extraction of spatial and temporal data was accomplished by combining TL and RNN models. Furthermore, the hybrid models outperformed current architectures when tested on both the original and augmented datasets. More overall, the highest recognition accuracy of 93.4% was attained.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.tensorflow.org/.

  2. 2.

    https://keras.io/.

References

  1. Bragg, D., et al.: Sign language recognition, generation, and translation: an interdisciplinary perspective. In: The 21st International ACM SIGACCESS Conference on Computers and Accessibility, pp. 16–31 (2019)

    Google Scholar 

  2. Xu, S., Liang, L., Ji, C.: Gesture recognition for human–machine interaction in table tennis video based on deep semantic understanding. Signal Process. Image Commun. 81, 115688 (2020)

    Article  Google Scholar 

  3. Wu, Z., Yao, T., Fu, Y., Jiang, Y.-G.: Deep learning for video classification and captioning. In: Frontiers of Multimedia Research, pp. 3–29 (2017)

    Google Scholar 

  4. Das, S., Chaudhary, A., Bremond, F., Thonnat, M.: Where to focus on for human action recognition? In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 71–80. IEEE (2019)

    Google Scholar 

  5. Tolentino, L.K.S., Juan, R.S., Thio-ac, A.C., Pamahoy, M.A.B., Forteza, J.R.R., Garcia, X.J.O.: Static sign language recognition using deep learning. Int. J. Mach. Learn. Comput. 9(6), 821–827 (2019)

    Article  Google Scholar 

  6. Jiang, X., Lu, M., Wang, S.-H.: An eight-layer convolutional neural network with stochastic pooling, batch normalization and dropout for fingerspelling recognition of Chinese sign language. Multimedia Tools Appl. 79(21), 15697–15715 (2020). https://doi.org/10.1007/s11042-019-08345-y

    Article  Google Scholar 

  7. Ameen, S., Vadera, S.: A convolutional neural network to classify American sign language fingerspelling from depth and colour images. Expert Syst. 34(3), e12197 (2017)

    Article  Google Scholar 

  8. Cayamcela, M.E.M., Lim, W.: Fine-tuning a pre-trained convolutional neural network model to translate American sign language in real-time. In: 2019 International Conference on Computing, Networking and Communications (ICNC), pp. 100–104. IEEE (2019)

    Google Scholar 

  9. Kamruzzaman, M.: Arabic sign language recognition and generating Arabic speech using convolutional neural network. Wirel. Commun. Mob. Comput. 2020 (2020)

    Google Scholar 

  10. Beena, M., Namboodiri, M.A., Dean, P.: Automatic sign language finger spelling using convolution neural network: analysis. Int. J. Pure Appl. Math. 117(20), 9–15 (2017)

    Google Scholar 

  11. Aly, S., Osman, B., Aly, W., Saber, M.: Arabic sign language fingerspelling recognition from depth and intensity images. In: 2016 12th International Computer Engineering Conference (ICENCO), pp. 99–104. IEEE (2016)

    Google Scholar 

  12. Shin, H., Kim, W.J., Jang, K.-A.: Korean sign language recognition based on image and convolution neural network. In: Proceedings of the 2nd International Conference on Image and Graphics Processing, pp. 52–55 (2019)

    Google Scholar 

  13. Rao, G.A., Syamala, K., Kishore, P., Sastry, A.: Deep convolutional neural networks for sign language recognition. In: 2018 Conference on Signal Processing and Communication Engineering Systems (SPACES), pp. 194–197. IEEE (2018)

    Google Scholar 

  14. ElBadawy, M., Elons, A., Shedeed, H.A., Tolba, M.: Arabic sign language recognition with 3D convolutional neural networks. In: 2017 Eighth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 66–71. IEEE (2017)

    Google Scholar 

  15. Ozcan, T., Basturk, A.: Transfer learning-based convolutional neural networks with heuristic optimization for hand gesture recognition. Neural Comput. Appl. 31(12), 8955–8970 (2019). https://doi.org/10.1007/s00521-019-04427-y

    Article  Google Scholar 

  16. Aktas, M., Gokberk, B., Akarun, L.: “Recognizing non-manual signs” in Turkish sign language. In: 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–6. IEEE (2019)

    Google Scholar 

  17. Ji, Y., Kim, S., Kim, Y.-J., Lee, K.-B.: Human-like sign-language learning method using deep learning. ETRI J. 40(4), 435–445 (2018)

    Article  Google Scholar 

  18. Vo, A.H., Pham, V.-H., Nguyen, B.T.: Deep learning for Vietnamese sign language recognition in video sequence. Int. J. Mach. Learn. Comput. 9(4), 440–445 (2019)

    Article  Google Scholar 

  19. Elboushaki, A., Hannane, R., Afdel, K., Koutti, L.: MultiD-CNN: a multi-dimensional feature learning approach based on deep convolutional networks for gesture recognition in RGB-D image sequences. Expert Syst. Appl. 139, 112829 (2020)

    Article  Google Scholar 

  20. Liao, Y., Xiong, P., Min, W., Min, W., Lu, J.: Dynamic sign language recognition based on video sequence with BLSTM-3D residual networks. IEEE Access 7, 38044–38054 (2019)

    Article  Google Scholar 

  21. Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109(1), 43–76 (2020)

    Article  Google Scholar 

  22. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  23. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  24. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  25. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  26. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

    Google Scholar 

  27. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  28. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  29. Cui, Z., Ke, R., Pu, Z., Wang, Y.: Stacked bidirectional and unidirectional LSTM recurrent neural network for forecasting network-wide traffic state with missing values. Transp. Res. Part C Emerg. Technol. 118, 102674 (2020)

    Article  Google Scholar 

  30. Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014)

  31. Lynn, H.M., Pan, S.B., Kim, P.: A deep bidirectional GRU network model for biometric electrocardiogram classification based on recurrent neural networks. IEEE Access 7, 145395–145405 (2019)

    Article  Google Scholar 

  32. Wen, Q., et al.: Time series data augmentation for deep learning: a survey. arXiv preprint arXiv:2002.12478 (2020)

  33. Liang, H., et al.: DARTS+: improved differentiable architecture search with early stopping. arXiv preprint arXiv:1909.06035 (2019)

  34. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  35. Postalcıoğlu, S.: Performance analysis of different optimizers for deep learning-based image recognition. Int. J. Pattern Recognit. Artif. Intell. 34(02), 2051003 (2020)

    Article  Google Scholar 

  36. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to express their gratitude to the signers who assisted in the creation of the dataset.

Author information

Authors and Affiliations

  1. Faculty of Computers and Artificial Intelligence, Cairo University, Cairo, Egypt

    Elsayed Mahmoud, Khaled Wassif & Hanaa Bayomi

Authors
  1. Elsayed Mahmoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khaled Wassif
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanaa Bayomi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elsayed Mahmoud .

Editor information

Editors and Affiliations

  1. Faculty of Computer and AI, Cairo University, Giza, Egypt

    Aboul Ella Hassanien

  2. Port Said University, Port Fouad, Egypt

    Rawya Y. Rizk

  3. Department of Computer Science, VŠB-TUO, Ostrava-Poruba, Czech Republic

    Václav Snášel

  4. Faculty of Engineering, Port Said University, Port Fouad, Egypt

    Rehab F. Abdel-Kader

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mahmoud, E., Wassif, K., Bayomi, H. (2022). Transfer Learning and Recurrent Neural Networks for Automatic Arabic Sign Language Recognition. In: Hassanien, A.E., Rizk, R.Y., Snášel, V., Abdel-Kader, R.F. (eds) The 8th International Conference on Advanced Machine Learning and Technologies and Applications (AMLTA2022). AMLTA 2022. Lecture Notes on Data Engineering and Communications Technologies, vol 113. Springer, Cham. https://doi.org/10.1007/978-3-031-03918-8_5

Download citation

Publish with us

Policies and ethics

Search

Navigation