Abstract
In the field of artificial intelligence (A), there has been a remarkable breakthrough with the emergence of large language models (LLMs) that are fine-tuned to follow human instructions. One such model is OpenAI's ChatGPT (Chat Generative Pre-trained Transformer), which has proven to be a highly capable tool for various tasks including question answering, code debugging, and dialogue generation. However, while these models are touted for their multilingual proficiency, their ability to accurately analyze sentiment, particularly in the Arabic language, has not been extensively investigated. Recognizing this limitation, we aim to address this gap by conducting a comprehensive evaluation of ChatGPT’ sentiment analysis capabilities specifically for Arabic text. We investigate the impact of utilizing the ChatGPT variants for Arabic sentiment analysis (ASA) and propose a new active labeling methods for ChatGPT. We evaluate the performance of four machine learning (ML) techniques, including Naive Bayes (NB), K-Nearest Neighbors (K-NN), Support Vector Machine (SVM), and Random Forest (FR), using accuracy, recall, precision, and F-score measure. We also compare six methods of labeling the data for ASA, manual labeling by humans, labeling using ChatGPT by Assistant-Poe, labeling using ChatGPT by Bing-Edge, labeling using ChatGPT by Assistant-Poe with humans, labeling using ChatGPT by Bing-Edge with humans, and labeling using ChatGPT by Assistant-Poe with Bing-Edge. Our experimental results show that the NB technique performed the best, achieving an accuracy of 91.22%, recall of 89.62%, precision of 88.90%, and F-score of 89.26% by using multiple Bing-Edge models for ASA. Moreover, utilizing our proposed active labeling method with ChatGPT achieved higher accuracy compared to other labeling methods. Our study suggests that the NB technique with multiple Bing-Edge models and our proposed active labeling method are effective approaches for ASA using ChatGPT. Our study contributes to the advancement of sentiment analysis in Arabic text and offers valuable insights into effective approaches for this task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kadaoui, K., et al.: TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties. arXiv preprint arXiv:2308.03051 (2023)
Ray, P.P.: ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber-Phys. Syst. (2023)
Al-Shalabi, A.A., Al-Gaphari, G., Salah, A.H., Alqasemi, F.: Investigating the impact of utilizing the K-nearest neighbor and levenshtein distance algorithms for Arabic sentiment analysis on mobile applications. JAST 1(2) (2023)
Al-Hagree, S., Al-Gaphari, G.: Arabic sentiment analysis based machine learning for measuring user satisfaction with banking services’ mobile applications: comparative study. In: 2022 2nd International Conference on Emerging Smart Technologies and Applications (eSmarTA), pp. 1–4. IEEE (2022)
Al-Hagree, S., Al-Gaphari, G.: Arabic sentiment analysis on mobile applications using levenshtein distance algorithm and naive bayes. In: 2022 2nd International Conference on Emerging Smart Technologies and Applications (eSmarTA), pp. 1–6. IEEE ,(2022)
Praveen, S.V., Vajrobol, V.: Understanding the perceptions of healthcare researchers regarding ChatGPT: a study based on bidirectional encoder representation from transformers (BERT) sentiment analysis and topic modeling. Ann. Biomed. Eng. 1–3 (2023)
Wang, Z., Xie, Q., Ding, Z., Feng, Y., Xia, R.: Is ChatGPT a good sentiment analyzer? A preliminary study. arXiv preprint arXiv:2304.04339 (2023)
Susnjak, T.: Applying bert and chatgpt for sentiment analysis of lyme disease in scientific literature. arXiv preprint arXiv:2302.06474 (2023)
Zhu, Y., Zhang, P., Haq, E.U., Hui, P., Tyson, G.: Can chatgpt reproduce human-generated labels? a study of social computing tasks. arXiv preprint arXiv:2304.10145 (2023)
Ubani, S., Polat, S.O., Nielsen, R.: Zero shot data Aug: generating and augmenting training data with ChatGPT. arXiv preprint arXiv:2304.14334 (2023)
Zhang, B., Yang, H., Liu, X.Y.: Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models. arXiv preprint arXiv:2306.12659 (2023)
Erfina, A., Nurul, M.R.: Implementation of Naive Bayes classification algorithm for Twitter user sentiment analysis on ChatGPT using Python programming language. Data Metadata 2, 45 (2023)
Koonchanok, R., Pan, Y., Jang, H.: Tracking public attitudes toward ChatGPT on Twitter using sentiment analysis and topic modeling. arXiv preprint arXiv:2306.12951 (2023)
Karanouh, M.: Mapping ChatGPT in mainstream media: early quantitative insights through sentiment analysis and word frequency analysis. arXiv preprint arXiv:2305.18340 (2023)
Liao, W., et al.: Differentiate chatgpt-generated and human-written medical texts. arXiv preprint arXiv:2304.11567 (2023)
Roumeliotis, K.I., Tselikas, N.D.: ChatGPT and open-AI models: a preliminary review. Future Internet 15(6), 192 (2023)
Al-Helali, B.: A new imputation method based on genetic programming and weighted KNN for symbolic regression with incomplete data. Soft. Comput. 25(8), 5993–6012 (2021)
Hadwan, M., Al-Hagery, M.A., Al-Sanabani, M., Al-Hagree, S.: Soft Bigram distance for names matching. PeerJ Comput. Sci. 7, e465 (2021)
Kwon, S.Y., Bhatia, G., Nagoud, E.M.B., Abdul-Mageed, M.: ChatGPT for arabic grammatical error correction. arXiv preprint arXiv:2308.04492 (2023)
Mujahid, M., Kanwal, K., Rustam, F., Aljadani, W., Ashraf, I.: Arabic ChatGPT tweets classification using RoBERTa and BERT ensemble model. ACM Trans. Asian Low-Resource Lang. Inform. Process. (2023)
Tawkat Islam Khondaker, M., Waheed, A., Moatez Billah Nagoudi, E., Abdul-Mageed, M.: GPTAraEval: a comprehensive evaluation of ChatGPT on arabic NLP. arXiv e-prints, arXiv-2305 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Al-Gaphari, G., AL-Hagree, S., Al-Helali, B. (2024). Investigating the Impact of Utilizing the ChatGPT for Arabic Sentiment Analysis. In: Saeed, F., Mohammed, F., Fazea, Y. (eds) Advances in Intelligent Computing Techniques and Applications. IRICT 2023. Lecture Notes on Data Engineering and Communications Technologies, vol 210. Springer, Cham. https://doi.org/10.1007/978-3-031-59711-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-59711-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-59710-7
Online ISBN: 978-3-031-59711-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)