Abstract
Maintaining the data quality of any database is always a critical aspect. To maintain the data quality, first of all, there is always a need to remove the redundant data significantly, when multiple data sources are integrated. In this paper, a new and effective data cleaning algorithm is presented aiming to remove duplicate data in text documents having different data types such as numbers, characters, words, and special symbols. The algorithm takes text data as input and removes the duplicate data through checking. Experiment results depict that the algorithm is simple and effective to deal with duplicate data and works fairly well.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
He L, Zhang Z, Tan Y, Liao M (2011) An efficient data cleaning algorithm based on attributes selection. In: 2011 IEEE 6th International Conference on Computer Sciences and Convergence Information Technology (ICCIT), pp 375–379
Beskales G, Soliman MA, Ilyas IF, Ben-David S, Kim Y (2010) ProbClean: a probabilistic duplicate detection system. In: 2010 IEEE 26th International Conference on Data Engineering (ICDE), pp 1193–1196
Maddodi S, Attigeri GV, Karunakar AK (2010) Data deduplication techniques and analysis. In: 2010 3rd International Conference on Emerging Trends in Engineering and Technology (ICETET), pp 664–668
Ali K, Warraich MA (2010) A framework to implement data cleaning in enterprise data warehouse for robust data quality. In: 2010 IEEE International Conference on Information and Emerging Technologies (ICIET), pp 1–6
Higazy A, El Tobely T, Yousef AH, Sarhan A (2013) Web-based Arabic/English duplicate record detection with nested blocking technique. In: 2013 8th International Conference on Computer Engineering & Systems (ICCES), pp 313–318
Zhang J (2010) An efficient and effective duplication detection method in large database applications. In: 2010 IEEE 4th International Conference on Network and System Security (NSS), pp 494–501
Lwin T, Nyunt TTS (2010) An efficient duplicate detection system for XML documents. In: 2010 second IEEE International Conference on Computer Engineering and Applications (ICCEA), vol 2, pp 178–182
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Jha, A., Somwanshi, D., Bundele, M. (2020). An Effective Duplicate Removal Algorithm for Text Documents. In: Mathur, G., Sharma, H., Bundele, M., Dey, N., Paprzycki, M. (eds) International Conference on Artificial Intelligence: Advances and Applications 2019. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-15-1059-5_30
Download citation
DOI: https://doi.org/10.1007/978-981-15-1059-5_30
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1058-8
Online ISBN: 978-981-15-1059-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)