Data Cleaning in Machine Learning: Improving Real Life Decisions and Challenges

Abstract

Data in real world is generally impure, it may contain outliers, incomplete, duplicated or obsolete values. Unclean data can direct to false conclusions and incorrect decisions. Thus it is crucial to clean the data efficiently. It is extensively known that the data quality affects machine learning (ML) model performances, data scientists spend considerable time on data cleaning before model training. During this work we shed light on some of the most important aspects of data cleaning efforts. We focused on a very important problem which is outlier detection. Regarding outlier detection we classified its approaches into three categories: statistics-based, distance-based, and Model-Based techniques. We discussed the previous techniques also mentioning the different sub-techniques of each, likewise the advantages and disadvantages of every technique. We also showed how contextual outlier identification and subspace outlier detection techniques may be used to overcome the “curse of dimensionality” in highdimensional outlier detection. The next task was to benchmark the major algorithms of the outlier detection and selecting the best algorithms depending on the dataset. Finally we proposed a new algorithms based on mainly two techniques k-Means clustering and Isolation forest in order to improve results of the outlier detection algorithms.