Abstract
There are numerous use cases of machine translation (MT) systems. Therefore, it has become very important to evaluate the performance of MT which can help researchers design a robust and reliable machine translation system. Although there are number of automatic MT evaluation metrics available nowadays, most of these fail to produce correct score. In this paper, we shall describe the most recent framework in MT industry that is Neural Machine Translation, and our approach is to test the performances of most popular translators—Google Translate and Microsoft’s Bing translator. We use language pairs English and Bengali for our detail analysis. Experiments are performed to translate English to Bengali. Bengali is a resource poor and one of the most widely spoken language in Indian subcontinent. Unlike glass box approach, where the performance of the MT systems are tried to enhance by adjusting various hyper parameters, we will be using black box approach, i.e., evaluating the performance of the already built systems. Beside performance analysis of the two translators, our main focus is to evaluate the performance of one of the very popular automatic evaluators Bilingual Under Study (BLEU) and some other automatic evaluation metrics in our work. This paper aims to measure the performance of BLEU and other automatic metrics during English to Bengali translation process by conducting survey with the help of questionnaires distributed among twenty people having moderate to high linguistic expertise on both the languages. Their responses are collected, and mean score is calculated which we have considered as human score (human judgement). BLEU and scores generated by other automatic metrics are used and compared their performance with human-generated score. A correlation is measured between human score and other metrics with Pearson correlation coefficient. Finally, some important observations are reported for this language pairs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
B.J. Dorr, P.W. Jordan, J.W. Benoit, A survey of current paradigms in machine translation. Adv. Comput. 49, 1–68, Elsevier (1999)
R. Ananthakrishnan, P. Bhattacharyya, M. Sasikumar, R.M. Shah, Some Issues in Automatic Evaluation of English-Hindi MT: More Blues for BLEU
C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, J. Schroeder, (Meta-) evaluation of machine translation, in Proceedings of the Second Workshop on Statistical Machine Translation (Pragua, Association for Computational linguistic, 2007), pp. 138–158
P.F. Brown, S.A. Della, V.J. Pietra, D. Pietra, R.L. Mercer, The Mathematics of statistical machine translation: parameter estimation. Association Comput. Linguist. 19(2), 1993 (1993)
P. Koehn, Statistical Machine Translation (Cambridge publisher, 2010)
Z. Wang, J. Shawe-Taylor, S. Szedmak, Kernel regression based machine translation, in Proceedings of NAACL HLT 2007, Companion Volume (Rochester, NY, Association for Computational Linguistics, 2007), pp. 185–188
Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin, A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)
L. Zhou, W. Hu, J. Zhang, C. Zong, Neural System Combination for Machine Translation. arXiv:1704.06393v1 [cs.CL] 21 Apr 2017
M.-T. Luong, H. Pham, C.D. Manning, Effective approaches to attention-based neural machine translation, in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, Lisbon, Portugal, 17–21 September 2015). c©2015
M. Wang, J. Xie, Z. Tan, J. Su, D. Xiong, C. Bian, Neural machine translation with decoding-history enhanced attention, in Proceedings of the 27th International Conference on Computational Linguistics (Santa Fe, New Mexico, USA, 20–26 Aug 2018), pp. 1464–1473
D. Bahdanau, K. Hyun, C.Y. Bengio, Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473v7 [cs.CL] 19 May 2016
K. Cho, B. van Merrienboer, D. Bahdanau, Y. Bengio, On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. arXiv:1409.1259v2 [cs.CL] 7 Oct 2014
X. Shi, I. Padhi, K. Knight, Does string based neural MT learn source syntax?, in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (Association for Computational Linguistics, Austin, Texas, 1–5 Nov 2016), pp. 1526–1534. c©2016
Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144v2 [cs.CL] 8 Oct 2016
R. Sennrichand, B. Haddow, Linguistic Input Features Improve Neural Machine Translation (School of Informatics, University of Edinburgh)
A. Das, P. Yerra, K. Kumar, S. Sarkar, A study of attention-based neural machine translation models on Indian languages, in Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (Osaka, Japan, 11–17 Dec 2016)
A. Kulesza, S.M. Shieber, A learning approach to improving sentence-level MT evaluation, in Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation (Baltimore, MD, 4–6 Oct 2004
F. Guzmán, S. Joty, L. Màrquez, P. Nakov, Machine Translation Evaluation with Neural Networks (Elsevier, Computer Speech and Language, 2017).
M. Fomicheva, L. Specia, Taking MT evaluation metrics extremes: beyond correlation with human judgment. Comput. Linguist. 45 (2019)
T. Dasgupta, M. Sinha, A. Basu, Resource creation and development of an English-Bengali back transliteration system. Int. J. Knowl. Intell. Eng. Syst. 19, 35–46 (2015)
R. Balyan, N. Chatterjee, Factor-based evaluation for English to Hindi MT outputs. Lang Res. Eval. 52, 969–996 (2018)
K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: a method for automatic evaluation of machine translation, in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) (Philadelphia, July 2002), pp. 311–318
M.A.A. Mumin, M.H. Seddiqui, M.Z. Iqbal, M.J. Islam, SUPara0.8M: A Balanced English-Bengali Parallel Corpus, IEEE Dataport, 2018 (2020)
J.P. Turian, L. Shen, I. Dan Melamed, Evaluation of Machine Translation and it’s Evaluation (2003)
N. Cancedda, M. Dymetman, G. Foster, C. Goutte, A statistical machine translation primer, in Learning Machine Translation (PHI Learning Pvt. Ltd., 2010)
E. Matusov, G. Leusch, H. Ney, Learning to combine machine translation systems, in Learning Machine Translation (PHI learning Pvt. Ltd., 2010)
H. Echizen’ya, K. Araki, E. Hovy, Word embedding-based automatic MT evaluation metric using word position information, in Proceedings of NAACL-HLT 2019, pp. 1874–1883 (2019)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Datta, G., Joshi, N., Gupta, K. (2021). Empirical Analysis of Performance of MT Systems and Its Metrics for English to Bengali: A Black Box-Based Approach. In: Paprzycki, M., Thampi, S.M., Mitra, S., Trajkovic, L., El-Alfy, ES.M. (eds) Intelligent Systems, Technologies and Applications. Advances in Intelligent Systems and Computing, vol 1353. Springer, Singapore. https://doi.org/10.1007/978-981-16-0730-1_24
Download citation
DOI: https://doi.org/10.1007/978-981-16-0730-1_24
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-0729-5
Online ISBN: 978-981-16-0730-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)