Abstract
Multi-dimensional Long Short-Term Memory networks (MDLSTMs) are now a state-of-the-art technology that provide a very good performance on different machine learning tasks due to their ability to model any n dimensional pattern using n recurrent connections with n forget gates. For this reason, we are going to focus on the handwritten Arabic word recognition, in which we will need to use only two dimensional MDLSTM for 2D input images then try to improve the accuracy of this baseline recognition system. Such several deep neural networks, the vanishing gradient problem can affect the performance of this MDLSTM-based recognition system. To solve this problem, Rectified Linear Units (ReLUs) are added with different modes, to draw out the best MDLSTM topology for the offline Arabic handwriting recognition system. Proposed systems are evaluated on a large database IFN/ENIT. According to the experimental results and compared to the baseline system, the best tested architecture gives a 5.57% reduction in the label error rate.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Pechwitz, M., Maddouri, S.S., Märgner, V., et al.: IFN/ENIT-database of handwritten Arabic words. In: Proceedings of CIFED. Citeseer, pp. 127–136 (2002)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Graves, A., Liwicki, M., Bunke, H., Schmidhuber, J., Fernández, S.: Unconstrained on-line handwriting recognition with recurrent neural networks. In: Advances in Neural Information Processing Systems, pp. 577–584 (2008)
Maalej, R., Tagougui, N., Kherallah, M.: Online Arabic handwriting recognition with dropout applied in deep recurrent neural networks. In: 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pp. 417–421. IEEE (2016)
Graves, A., Schmidhuber, J.: Offline handwriting recognition with multidimensional recurrent neural networks. In: Advances in Neural Information Processing Systems, pp. 545–552 (2009)
Graves, A., Fernández, S., Gomez, F., et al.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376. ACM (2006)
Maalej, R., Kherallah, M.: Improving MDLSTM for offline Arabic handwriting recognition using dropout at different positions. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 431–438. Springer, Cham (2016)
Märgner, V., El Abed, H.: ICDAR 2009-Arabic handwriting recognition competition, vol. 14, pp. 3–13 (2011)
Maalej, R., Tagougui, N., Kherallah, M.: Recognition of handwritten Arabic words with dropout applied in MDLSTM. In: International Conference Image Analysis and Recognition, pp. 746–752. Springer (2016)
Maalej, R., Kherallah, M.: Maxout into MDLSTM for offline Arabic handwriting recognition. In: International Conference on Neural Information Processing, ICONIP 2019, pp. 534–545. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36718-3_45
Goodfellow, I.J., Warde-Farley, D., Mirza, M., et al.: Maxout networks. arXiv Prepr arXiv:13024389 (2013)
Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International Conference on Machine Learning, pp. 1310–1318 (2013)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5, 157–166 (1994)
Dahl, G.E., Sainath, T.N., Hinton, G.E.: Improving deep neural networks for LVCSR using rectified linear units and dropout. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8609–8613 (2013)
Cai, M., Liu, J.: Maxout neurons for deep convolutional and LSTM neural networks in speech recognition. Speech Commun. 77, 53–64 (2016)
Li, X., Wu, X.: Improving long short-term memory networks using maxout units for large vocabulary speech recognition. In: ICASSP-2015, pp. 4600–4604 (2015)
Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)
Russakovsky, O., Deng, J., Su, H., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015)
Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Frinken, V., Uchida, S.: Deep BLSTM neural networks for unconstrained continuous handwritten text recognition. In: 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 911–915. IEEE (2015)
Luo, Y., Liu, Y., Zhang, Y., et al.: Maxout neurons based deep bidirectional LSTM for acoustic modeling. In: International Conference on Robotics and Biomimetics (ROBIO), pp. 1599–1604. IEEE (2017)
Maalej, R., Kherallah, M.: Improving the DBLSTM for on-line Arabic handwriting recognition. Multimed. Tools Appl. 79, 17969–17990 (2020). https://doi.org/10.1007/s11042-020-08740-w
Hinton, G.E., Srivastava, N., Krizhevsky, A., et al.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv Prepr arXiv:12070580 (2012)
Jayech, K., Mahjoub, M., Ben, Amara N.: Arabic handwritten word recognition based on dynamic Bayesian network. Int. Arab J. Inf. Technol. 13(6B), 1024–1031 (2016)
Amrouch, M., Rabi, M., Es-Saady, Y.: Convolutional feature learning and CNN based HMM for Arabic handwriting recognition, pp. 265–274 (2018)
Maalej, R., Kherallah, M.: Convolutional neural network and BLSTM for offline Arabic handwriting recognition. In: International Arab Conference on Information Technology (ACIT), pp. 1–6. IEEE (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Maalej, R., Kherallah, M. (2021). ReLU to Enhance MDLSTM for Offline Arabic Handwriting Recognition. In: Abraham, A., Siarry, P., Ma, K., Kaklauskas, A. (eds) Intelligent Systems Design and Applications. ISDA 2019. Advances in Intelligent Systems and Computing, vol 1181. Springer, Cham. https://doi.org/10.1007/978-3-030-49342-4_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-49342-4_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49341-7
Online ISBN: 978-3-030-49342-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)