Skip to main content
SpringerLink
Log in

A Review of Tools and Techniques for Preprocessing of Textual Data

  • Conference paper
  • First Online:
Computational Methods and Data Engineering

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1227))

Abstract

With the high availability of computing facilities, a huge amount of data is available in electronic form. Processing of huge data is required to discover new facts and knowledge. But dealing with huge datasets is challenging because real-world data is generally incomplete, inconsistent, contains errors or outliers. More than 80% of the data is unstructured or semi-structured. The data is prepared by data preprocessing. Data preprocessing has become an essential step in data mining. Data Preprocessing takes 80% of the total efforts of any data mining project and it directly affects the quality of data mining. The selection of the right technique and tool for data preprocessing helps to enhance the speed of data mining process. This paper discusses different preprocessing techniques, different tools available for text preprocessing, carries out their comparison and briefs the challenges faced such as knowledge of sentence structure of a language to perform tokenization, difficulty in constructing domain-specific stop words list, over stemming and under stemming etc.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ramírez-Gallego S, Krawczyk B, García S, Woźniak M, Herrera F (2017) A survey on data preprocessing for data stream mining: current status and future directions. Neurocomputing 239:39–57. https://doi.org/10.1016/j.neucom.2017.01.078

    Article  Google Scholar 

  2. N, Y., S, M (2016) A review on text mining in data mining. Int. J. Soft Comput 7:01–08. https://doi.org/10.5121/ijsc.2016.7301

  3. Gera M, Goel S (2015) Data mining—techniques, methods and algorithms: a review on tools and their validity. Int J Comput Appl 113:22–29. https://doi.org/10.5120/19926-2042

    Article  Google Scholar 

  4. Talib R, Kashif M, Ayesha S, Fatima F (2016) Text mining: techniques, applications and issues. Int J Adv Comput Sci Appl 7:414–418. https://doi.org/10.14569/ijacsa.2016.071153

  5. Mining, U., Chandrama, W., Devale, P.P.R., Murumkar, P.R.: Survey on Data Preprocessing Method of Web 5:3521–3524 (2014)

    Google Scholar 

  6. Dr.S.Kannan, V.G.: Preprocessing Techniques for Text Mining. J. Emerg. Technol. Web Intell. (2016)

    Google Scholar 

  7. Srividhya, V., Anitha, R.: Evaluating Preprocessing Techniques in Text Categorization. Int. J. Comput. Sci. Appl. 49–51 (2010)

    Google Scholar 

  8. Lourdusamy, R., Abraham, S.: A Survey on Text Pre-processing Techniques and Tools. Int. J. Comput. Sci. Eng. 06, 148–157 (2019). https://doi.org/10.26438/ijcse/v6si3.148157

  9. Katariya NP, Chaudhari MS (2015) Text preprocessing for text mining using side information. Int Comput Sci Mob Appl 3:3–7

    Google Scholar 

  10. Kadhim AI (2018) An Evaluation of Preprocessing Techniques for Text Classification. 16:22–32

    Google Scholar 

  11. El Haddaoui, B., Chiheb, R., Faizi, R., El Afia, A.: Toward a Sentiment Analysis Framework for Social Media. 1–6 (2018). https://doi.org/10.1145/3230905.3230919

  12. Putra, S.J., Khalil, I., Gunawan, M.N., Amin, R., Sutabri, T.: A Hybrid Model for Social Media Sentiment Analysis for Indonesian Text. 297–301 (2019). https://doi.org/10.1145/3282373.3282850

  13. Camacho-collados, J.: On the Role of Text Preprocessing in Neural Network Architectures : An Evaluation Study on Text Categorization and Sentiment Analysis. 40–46 (2018)

    Google Scholar 

  14. Orellana, G., Arias, B., Orellana, M., Saquicela, V., Baculima, F., Piedra, N.: A study on the impact of pre-processing techniques in Spanish and english text classification over short and large text documents. Proceedings—3rd International Conference on Information Systems and Computer Science, INCISCOS 2018, Dec pp 277–283 (2018). https://doi.org/10.1109/INCISCOS.2018.00047

  15. Harvey S (2009) A study of interscholastic soccer players perceptions of learning with game sense. Asian J Exerc Sport Sci 6:1–11. https://doi.org/10.15439/2018KM46

  16. Effrosynidis D, Symeonidis S, Arampatzis A (1999) Conference on research and advanced technology for digital libraries. Interlend Doc Supply 27:300–302. https://doi.org/10.1108/ilds.1999.12227cab.016

  17. Wankhede S, Patil R, Sonawane S, Save PA (2018) Data preprocessing for efficient sentimental analysis. In: Proceedings international conference on inventive communication and computational technologies ICICCT 2018, pp 723–726. https://doi.org/10.1109/ICICCT.2018.8473277

  18. Roy D, Mitra M, Ganguly D (2018) To clean or not to clean: document preprocessing and reproducibility. J Data Inf Qual 10. https://doi.org/10.1145/3242180

  19. Saxena D, Saritha SK, Prasad KNSS V (2017) Survey on feature extraction methods in object. Int J Comput Appl 166:11–17

    Google Scholar 

  20. Waykole RN, Thakare AD (2018) A review of feature extraction methods for text. Int J Adv Eng Res 351–354

    Google Scholar 

  21. Zin HM, Mustapha N, Murad MAA, Sharef NM (2017) The effects of pre-processing strategies in sentiment analysis of online movie reviews. In: AIP conference proceedings, pp 1–8. https://doi.org/10.1063/1.5005422

  22. Haddi E, Liu X, Shi Y (2013) The role of text pre-processing in sentiment analysis. Procedia Comput Sci 17:26–32. https://doi.org/10.1016/j.procs.2013.05.005

    Article  Google Scholar 

  23. Ghalehtaki RA, Khotanlou H, Esmaeilpour M (2014) Evaluating preprocessing by turing machine in text categorization. In: 2014 Iranian conference on intelligent systems ICIS 2014. https://doi.org/10.1109/IranianCIS.2014.6802540

  24. Dos Santos FL, Ladeira M (2014) The role of text pre-processing in opinion mining on a social media language dataset. In: Proceedings—2014 Brazilian conference on intelligent system BRACIS 2014, pp 50–54. https://doi.org/10.1109/BRACIS.2014.20

  25. Krouska A, Troussas C, Virvou M (2016) The effect of preprocessing techniques on Twitter sentiment analysis. In: IISA 2016—7th international conference on information, intelligence, systems and applications (2016). https://doi.org/10.1109/IISA.2016.7785373

  26. Geetharamani R, Kumar MN, Balasubramanian L (2017) Identification of emotions in text articles through data pre-processing and data mining techniques. In: Proceedings 2016 international conference on advanced communication, control & computing technologies ICACCCT 2016, pp 611–615. https://doi.org/10.1109/ICACCCT.2016.7831713

  27. Angiani G, Ferrari L, Fontanini T, Fornacciari P, Iotti E, Magliani F, Manicardi S (2016) A comparison between preprocessing techniques for sentiment analysis in Twitter. In: Proceedings 2nd international workshop on knowledge discovery on the web, KDWeb 2016, pp 1–11 (2016). https://doi.org/10.1007/978-3-319-67008-9_31

  28. Kaur A, Chopra D (2016) Comparison of text mining tools. In: 2016 5th international conference on reliability, Infocom technologies and optimization (Trends Futur Dir 186–192. https://doi.org/10.1109/ICRITO.2016.7784950

  29. Wilson M, Tchantchaleishvili V (2013) The importance of Free and Open Source Software and Open Standards in Modern Scientific Publishing. Publications 1:49–55. https://doi.org/10.3390/publications1020049

    Article  Google Scholar 

  30. Dan L, Lihua L, Zhaoxin Z (2013) Research of text categorization on Weka. In: Proceedings 2013 3rd international conference on intelligent system design and engineering applications ISDEA 2013, pp 1129–1131 (2013). https://doi.org/10.1109/ISDEA.2012.266

  31. Kalra V, Aggarwal R (2018) Importance of text data preprocessing & implementation in RapidMiner. Proc First Int Conf Inf Technol Knowl Manag 14:71–75. https://doi.org/10.15439/2017km46

  32. Berthold MR, Cebron N, Dill F, Gabriel TR, Kotter T, Meinl T, Ohl P, Sieb C, Thiel K, Wiswedel B (2007) Knime. Web. 1–8. https://doi.org/10.1007/978-3-540-78246-9

  33. Hofmann M, Chisholm A, Chisholm H, Berthold M (2016) Text mining and visualization: case studies using open-source tools

    Google Scholar 

  34. Welbers K, Van Atteveldt W, Benoit K (2017) Text analysis in R. Commun Methods Meas 11:245–265. https://doi.org/10.1080/19312458.2017.1387238

    Article  Google Scholar 

  35. Orange3 Text Mining Documentation (2018)

    Google Scholar 

  36. Project Jupyter: Project Jupyter | Home. http://jupyter.org/, (2017)

  37. Rangra K, Bansal KL (2014) Comparative study of data mining tools. Int J Adv. Res Comput Sci Softw Eng 4:216–223. https://doi.org/10.1016/j.nuclphysa.2007.03.042

    Article  Google Scholar 

  38. Chauhan N, Gautam N (2015) Parametric comparison of data mining tools 291–298

    Google Scholar 

  39. Solanki H (2013) Comparative study of data mining tools and analysis with unified data mining theory. Int J Comput Appl 75:975–8887. https://doi.org/10.5120/13195-0862

    Article  Google Scholar 

  40. Patel PS, Desai SG (2015) A comparative study on data mining tools. Int J Adv Trends Comput Sci Eng 4:28–30

    Google Scholar 

  41. Bisht P, Negi N, Mishra P, Chauhan P (2018) A comparative study on various data mining tools for intrusion detection 9:1–8

    Google Scholar 

  42. Singh DK (2017) Comparative study of various open source data mining tools 356–358

    Google Scholar 

  43. Ranjan R, Agarwal R, Venkatesan S (2017) Detailed analysis of data mining tools. Int J Eng Res 6:785–789. https://doi.org/10.17577/ijertv6is050459

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Applications, Panjab University, Chandigarh, India

    Abhinav Kathuria, Anu Gupta & R. K. Singla

Authors
  1. Abhinav Kathuria
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anu Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. K. Singla
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abhinav Kathuria .

Editor information

Editors and Affiliations

  1. School of Computer Science, University of Petroleum and Energy Studies, Dehradun, Uttarakhand, India

    Vijendra Singh

  2. Department of Electrical and Computer Engineering, University of Dayton, Dayton, OH, USA

    Vijayan K. Asari

  3. Department of Computer Science and Engineering, SRM University Delhi-NCR, Sonepat, Haryana, India

    Sanjay Kumar

  4. Department of Computer Science and Engineering, Chandigarh College of Engineering and Technology, Chandigarh, Punjab, India

    R. B. Patel

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kathuria, A., Gupta, A., Singla, R.K. (2021). A Review of Tools and Techniques for Preprocessing of Textual Data. In: Singh, V., Asari, V., Kumar, S., Patel, R. (eds) Computational Methods and Data Engineering. Advances in Intelligent Systems and Computing, vol 1227. Springer, Singapore. https://doi.org/10.1007/978-981-15-6876-3_31

Download citation

Publish with us

Policies and ethics

Search

Navigation