Keywords

1 Introduction

The analysis of microscopic images plays a huge role in the fields of medicine and computer science. Several research issues are involved in the analysis of microscopic images which are the first steps in detecting and diagnosing diseases like malaria, leukemia, and anemia. There have been many researchers conducted on identification of diseases which involves algorithms based on image processing, computer vision, machine learning and deep learning. Following are the diseases that will be identified and classified in the proposed work.

“Leukemia”, which refers to the cancer of the blood and bone marrow (where blood cells are produced) is one such disease where there is a rapid growth of abnormal WBCs. The diagnosis of leukemia is based on the fact that the white blood cell count is increased with immature last cells (lymphoid or myeloid) and the neutrophils and platelets decrease. In “Anemia”, this disease is diagnosed based on the shape of the RBCs. People with anemia have red blood cells that have an unusual shape and might look larger or smaller than normal. In the case of “Lymphoma”, which is a cancer affecting lymph nodes, comes in 3 types i.e., CLL (chronic lymphocytic leukemia), FL (follicular lymphoma) and MCL (mantle cell lymphoma). In this disease, the lymphocytes, which are a type of WBCs that are present in the lymph, become abnormally cancerous and grow uncontrollably. As these cells travel through the lymphatic system, they settle in the lymph nodes and cause them to swell. And in “Myeloma”, which is a plasma cell related disease, there is an abnormal growth of the plasma cells in the bone marrow and based on that this disease is diagnosed. Lastly, there is “Malaria” which is a disease caused by a bite from a malaria-carrying mosquito. The malaria parasite enters the bloodstream, multiplies, infects, and destroys the RBCs and based on the number of RBCs present, this disease is detected.

In order to detect these diseases, hematologists have to thoroughly examine blood smears under microscope for proper classification of diseased cells and identify the disease which can be quite time consuming and inaccurate. This approach will help the hematologists by reducing the time for diagnosing the diseases with existing technologies using chemical and electro voltaic methods, this will also reduce the expense of machines and human resource as this is a integrated system for 7 types of diseases. Therefore, this research aims to develop an image processing and deep learning-based system to detect these diseases in an efficient and more accurate way. The model will classify about 7 diseases and the classification model is validated based on the accuracy of the model, precision, recall and f1 score of the confusion matrix obtained from the model.

2 Literature Work

In the past, there have been many works related to classification and detection of hematologic diseases.

Tiancheng Xia et al. detected CBCs (complete blood cells) by using deep neural networks through YOLOv3 algorithm and applied the detector to the blood samples of COVID 19 patients where blood cell clots are its typical symptom. Their precision and recall values of test results were low which led to a lesser model performance accuracy [1]. Asem H. and Ashraf Y. used data mining techniques through decision tree, rule induction and Naive Bayes on a test blood dataset which detected blood diseases of adult and children hematology and tumor. The accuracy of data mining techniques was quite less. Maximum accuracy to detect tumor was achieved by Naïve Bayes which was only 56% and for adult and children hematology, rule induction technique achieved accuracy in the range of 57–67%. Hence, the use of data mining algorithms was ruled out for the proposed system [2]. Angelo Galiano et al. developed a home health assistance web-based communication system to manage medical services measuring data from patients at home which would be automatically transferred to the hospital [3]. Also, in [4], another health care monitoring system was developed which was an IoT based biotelemetry system to analyze ECG signals of patients. Both of the above works have built a hardware-based system to diagnose medical symptoms which are relatively expensive and requires high maintenance. Same is the case with [5] in which Rao et al. have developed a hardware-based image processing system which works on image segmentation and localization of the iris of the eye. Lei Zhao et al. did a study based on the blood quality control materials for analyzing hematologic diseases. The equipment stated in this work is a hematology analyzer which is used to monitor the blood quality and analyze its components. This system makes use of many chemicals to store blood and even temperature conditions are to be considered carefully for storage [6]. The system proposed in this paper does not require to store huge amounts of blood for detection, only a microscopic image of a drop of blood smear is required to perform detection and classification which makes it more efficient. Guclu Ongun et al. developed an automatic differential blood count system using active contour models and for classification used K-nearest neighbor, learning vector quantization, multilayer perceptron and support vector machine [7]. Wei Yu et al. developed an automatic cell recognition system using deep neural networks to classify different types of leukocytes. The overall accuracy achieved by this system was about 88.5% which is relatively lesser than the accuracy of the proposed system [8]. Pooja and Nagaraj used ensemble learning methods like stacking, bagging, voting, Adaboost and Bayesian boosting and applied classifiers such as Decision tree, artificial neural network, Naive Bayes and K-nearest neighbor to detect anemia from red blood cells [9]. Preeti and Virani devised a methodology for leukemia detection using image processing techniques and image segmentation based on K means clustering, Marker controlled watershed algorithm and HSV color-based segmentation then used SVM for further classification of leukemia types [10]. Both of the above works have implemented ensemble algorithms and classifiers to detect the diseases but were able to obtain only 80–90% accuracy. Deep learning algorithms are known to produce greater accuracy and classification performance as compared to other machine learning algorithms. Subhash et al. proposed an acute lymphoblastic leukemia detection system that used OpenCV and skimage for image processing to extract features from blood images and used classifiers such as CNN, FNN, SVM and KNN [11]. Prashanth built an automatic microscopic blood smear RBC classification using Principal Component Analysis (PCA) and SVM classifier [12,13,14,15,16,17,18] were based on detection and classification of myeloma cells in microscopic images which was done using various deep learning algorithms like CNN, ANN, Mask R-CNN, along with classification algorithms based on SVM, Random Forest, etc. Hend Mohamed et al. developed an approach to automatically detect WBC cancer diseases by dividing the disease categories based on similar symptoms, extracting features from them and applying Random Forest classifier to classify the diseases [19,20,21,—22] discusses approaches to detect lymphoma cells using a computer vision method, an immunophenotyping approach and a segmentation-based approach called DenseX-Net respectively. The problem with the DenseX-Net approach is that it faced problems with detecting lymphoma images where it behaved badly on boundary delineation wherever the boundary of lymphoma cell was blurred. Also, it misdiagnosed certain images since its maximum accuracy was 72.84%. Hence, for the proposed system, image processing methods were implemented to detect the boundaries of infected cells thoroughly and extract required features and the infected cells were labeled properly in the images for better classification. [23,24,25,26,27,28,29] discusses various approaches for classification of types of blood cells. Deep learning approaches such as Inception Recurrent Residual Convolutional Neural Network (IRRCNN), Faster R-CNN were used, some other classification algorithms based on Otsu and Naïve Bayes were used and some image processing techniques such as masking, and morphological operations were implemented. [30,31,32] explains approaches to count different types of cells from microscopic images based on computer vision and machine learning algorithms.

Most of the previous works are based on deep neural networks or a combination of more than one image processing and machine learning based algorithms to detect only one particular type of blood disease. The performance accuracy of most of the literature works was found to be less than the accuracy of the proposed system. Deep learning algorithms are found to achieve better accuracies as compared to other approaches for classification. Some of the works mentioned above have used deep learning methods such as YOLO, Faster and Mask R-CNN and IRRCNN. Even [33, 34] have developed image detection and classification systems based on CNN and have proved that neural networks give the best accuracy. More training images should be added to enhance the classification performance. But the above approaches have worked only on classification of one or few diseases and the sample size of training images taken were less.

Hence, the proposed system has been built using a deep learning approach to classify and detect 7 types of blood diseases with a large dataset which contains ample number of images for each disease class. The model has achieved a great performance accuracy of 98.6%. Moreover, the implementation of preprocessing techniques on image datasets has contributed to the enhancement of the classification performance of the model which led to achieve more accuracy than the approaches mentioned in the literature work. Further sections will explain the dataset and methodology used for the system along with the results and performance parameters that were used to validate the accuracy of the model.

3 Methodology

In the proposed methodology, an approach based on Image processing and Deep learning was implemented to detect and classify 7 classes of diseases i.e., Leukemia, Anemia, CLL, FL, MCL, Myeloma and Malaria from the healthy blood smear images. The approach is divided into 2 sections namely, Image Preprocessing and Classification. The first section will explain the application of image processing operations on datasets of Leukemia and Anemia based on Feature Extraction and Ni-black Thresholding. For the remaining 5 diseases, ready-made preprocessed datasets were available on the net so the preprocessing for the same was not required. In the second section, a deep learning model was built to classify the diseases which is based on a neural network. VGG16 was used as the neural network for the model along with pretrained weights of ImageNet. These sections will be explained in depth in the following Proposed Work segment of the methodology. Following is the flowchart of the proposed methodology (Fig. 1).

Fig. 1
A flowchart of the methodology starts from blood samples to microscopic image acquisition to neutral network to classification. The arrow between M I A and network is labeled lymphoma, malaria. M I A branches into 2: leukemia to image pre-processing to neutral network; and anemia to image pre-processing to grayscale conversion to neutral network.

Flowchart of proposed methodology

Full size image
  1. A.

    Dataset

Images dataset of anemia and Leukemia was taken from google images. Anemia dataset consists of 20 images and Leukemia dataset consists of 40 images. Images taken from google was not pre-processed, for Anemia detection the Ni-Black pre-processing is used to get better results. For Leukemia the feature extraction method is used, after applying preprocessing on both the dataset it is finally used in model. Pixel size of both the dataset is resolved to (224 × 224,3) because the model accepts the (224 × 224,3) resolution [35, 36].

Image datasets for each of the disease classes were taken from Kaggle which were already preprocessed. Three datasets of Lymphoma, Myeloma, and Malaria were used.

Lymphoma: The Lymphoma dataset consists of three classes of diseases (CLL, FL, MCL), totaling 5400 images [37]

Myeloma: For Myeloma, a total of 298 images were taken. The Bone marrow photos were collected in a microscope of a pixel size of 2040 × 1536 pixels and then preprocessed [38,39,40].

Malaria: The Malaria dataset consists of two classes those are infected and non-infected, combining the total number of images are 27,558 [41].

  1. B.

    Proposed Work

    1. 1.

      Image Preprocessing

      In this section, image processing algorithms were used on two different datasets of Leukemia and Anemia. The two main algorithms used are:

      1. 1.1

        Feature Extraction: This algorithm was used to detect Leukemia from blood sample images of Leukemic patients. The images were first converted to grayscale from its BGR form. Then a feature extraction method was applied on the grayscale images to extract features which is helpful for distinguishing the resulting values from the standard ones. This feature extraction was done by segmenting blood cells and cell nuclei into binary equivalent images. The features were based on its color, geometrical features like symmetry and concavity, texture features like entropy and homogeneity and statistical features like skewness and mean gradient matrix.

      2. 1.2

        Ni-black Thresholding: This algorithm was used to detect Anemia from microscopic blood images of Anemic patients. The niblack thresholding method aims to achieve better results specially for microscopic images, it separates the object of our interest from the rest of the image for further processing. [42] compared different binarization methods like Niblack, Bernsen, Sauvola and Otsu for counting of WBCs and RBCs. In their experiments over different samples with different conditions showed that Niblack is the most reliable method. It maintains disjoint components which is necessary for avoiding over or under segmentation. The images were first converted to grayscale from its BGR form since it is a crucial step for applying Niblack Thresholding. The gray level of pixels belonging to RBCs are entirely different from the gray levels of pixels belonging to the other blood contents. The niblack thresholding method is applied on the images by segmenting the foreground region to isolate the normal and abnormal RBCs from the background region. It provides us with the binary image from the grayscale one, effectively separating the normal and abnormal cells from the rest of the contents of blood. It is done by calculating the local mean and standard deviation of the pixel value within a confined size of a window in the grayscale image. The thresholding value for each window is computed based on the mean m, standard deviation σ of the pixels in that window.

        $$T=m+k\times \alpha$$

        k is -0.2 suggested by Ni-black. In this way, the images are binarized by showing sickle cells darker than the rest of the contents.

  2. 2.

    Classification

In this section, a deep neural network-based approach was used to perform the classification of the diseases. A neural network model called VGG16 was used. VGG16 is Very Deep Convolutional Networks for Large-Scale Image Recognition with 93% using ImageNet weight, in 2014 it was considered as the best architecture for image classification. It is a convolutional neural network (CNN) that is 16 layers deep. A pre-trained version of the network which was trained on more than a million images is loaded and this pretrained weight was taken from the ImageNet database. This weight can classify images up to 1000 object categories. Hence, this network was chosen for its outstanding classification property. Following is the architecture of VGG16 which shows a schematic arrangement of all the layers i.e., convolutional, pooling, and dense (fully connected) layers (Fig. 2).

Fig. 2
The layer arrangement in V G G 16. 1 layer of max pooling is in between the 3 layers of convolution 1 to 5 with the following measurements: 224 into 224 into 64; 112 into 112 into 128; 56 into 56 into 256; 14 into 14 into 512. The max pooling layer after conv 5 is measured 7 into 7 into 512. 3 layers of fully connected 6 to 8 are arranged after it.

Source: https://medium.com/mlearning-ai/an-overview-of-vgg16-and-nin-models-96e4bf398484)

VGG16 architecture (

Full size image

The model consists of 6 layers of which there are 3 dense layers, 2 dropout layers and a flatten layer. Firstly, a flatten layer was added, then a dense layer was added with an activation function of ‘LeakyReLU’ and its learning rate as 0.3. This is followed by a dropout layer of rate 0.5 and again a dense layer. Then another dropout layer of rate 0.3 followed by a dense layer added on 8 classes (7 disease classes and 1 healthy image class) with an activation function of ‘SoftMax’. Then, a loss function of ‘categorical_crossentropy’ was used with an optimizer function of ‘adam’. For training of the model, 100 epochs were set for a batch size of 10. Tensor board was used to extract the graph results of loss and accuracy over epoch of the model. This research tried other loss such as ‘root mean square’ but it gave 31.3% so this was dropped and ‘adam’ was choosen.

4 Results

In this paper, a novel problem has been solved which involves including more than two diseases into a CNN based classifier. A high accuracy has been achieved which is specifically 98.6% and were able to predict diseases correctly. For acquiring a dataset of two diseases, image pre-processing techniques was implemented and for the rest of diseases which directly fed the data into the model. Following results are divided into 2 sections, (4.1) Image Preprocessing, and (4.2) Classification.

4.1 Image Preprocessing

This section provides results for the preprocessing of disease classes of Leukemia and Anemia.

Preprocessing of Leukemia: (Fig. 3)

Fig. 3
2 microscopic images of blood samples. The image on the left displays the unaffected blood cells represented as blue spots. The image on the right displays the blood cells affected with leukemia represented with green covering outside the blue spots.

a Original image b LEUKEMIA detected image

Full size image

The image above shows the original microscopic image of blood smear containing leukemia affected cells. The image below shows the output after preprocessing the image and extracting features. The leukemia affected cells are highlighted as shown in the figure below.

Preprocessing of Anemia: (Fig. 4)

Fig. 4
2 microscopic images of blood samples. The image on the left represents the blood cells as red circles with some having white spots. The image on the right represents the blood cells as plain black circles and black circles with white spots.

a Original blood image b Ni-black thresholding

Full size image

The image above shows the original microscopic image of blood smear containing anemia affected cells. The image below shows the output after preprocessing the image using Ni-black Thresholding. The anemic cells are threshold from the rest of the blood contents as shown in the figure below.

4.2 Classification

This section shows the results obtained from the classification model built using Neural networks. The section is further divided into 2 parts–(a) Disease Identification, and (b) Performance Parameters for the Classification Model.

  1. (a)

    Disease Identification

Following results show the prediction outputs of all the disease classes performed by the model after applying the neural network by training all the 7 disease classes of image datasets.

Leukemia detection: (Fig. 5)

Fig. 5
A microscopic image of leukemia-affected blood sample. It has red dots in the background and a few blue dots with a green outline highlighted in the front. The title above the image reads I D, 5; label, leukemia.

Prediction of leukemi

Full size image

Anemia detection: (Fig. 6)

Fig. 6
A microscopic image of the anemia-affected blood sample. The image has blood cells represented as black circles with white spots. The title above the image reads I D, 1; label, anemia.

Prediction of anemia

Full size image

MCL Lymphoma detection: (Fig. 7)

Fig. 7
A microscopic image of a blood sample affected with M C L lymphoma. The image has tiny black dots with a few red and yellow dots scattered in between them. The title above the image reads I D, 6; label, M C L.

Prediction of MCL lymphoma

Full size image

CLL Lymphoma detection: (Fig. 8)

Fig. 8
A microscopic image of a blood sample affected with C L L lymphoma. The densely packed blood cells are represented as black dots with a few yellow and red spots.

Prediction of CCL lymphoma

Full size image

FL Lymphoma detection: (Fig. 9)

Fig. 9
A microscopic image of a blood sample affected with F L lymphoma. The densely packed blood cells represented as red dots have tiny black and yellow dots scattered all over them.

Prediction of FL lymphoma

Full size image

Myeloma detection: (Fig. 10)

Fig. 10
A microscopic image of a blood sample affected with myeloma. The image has yellow dots scattered in the background with 6 blue dots in the front. The title above the image reads I D, 8; label, Myeloma.

Prediction of myeloma

Full size image

Malaria detection: (Fig. 11)

Fig. 11
A microscopic image of a blood sample affected with malaria. The image has a large orange circle with a blood-like spot in the center.

Prediction of malaria

Full size image

4.3 Performance Parameters for the Classification Model:

Following parameters which include the classification matrix and graphs of Epoch versus Loss and Epoch versus Accuracy are shown and explained.

  1. 1.

    Classification Matrix

The image below shows the classification matrix of the model. It is observed that most of the precision, recall and f1 scores for each disease class is 1.00 which means that the model has excellent performance measures based on the number of positive predictions. Precision as 1.00 proves that the model was able to make 100% correct positive predictions. Recall as 1.00 proves that the model was able to classify 100% positive cases from the whole dataset of that particular disease class (Table 1).

Table 1 Classification matrix of model
Full size table

For example, Anemia has a precision and recall of 1.00 which means all the Anemia images were correctly classified and detected from the rest of the image data so there were 0 wrong predictions for Anemia detection whereas in the case of Healthy image classification, the precision score is 0.59 which implies there were 59% correct predictions of healthy images from the data and the remaining 41% were wrong predictions.

And finally, an F1 score takes both precision and recall into account to ultimately measure the accuracy for each disease classification. Following is the formula to calculate F1 score:

$${F}_{1}=2\times \left(\frac{precision\times recall}{precision+recall}\right)$$

As for the overall performance of the model, an accuracy of 98.6% was achieved with a loss of 0.47, which is pretty good for a multi class classification.

  1. 1.1

    Epoch versus Loss

  2. 1.2

    Epoch versus Accuracy

In the above graphs X-axis indicates the epoch and Y-axis indicates loss in Fig. 12a and accuracy in Fig. 12b. From this it can be seen that at the epoch range between 90 and 100, an optimized value of accuracy at 98.6% and loss value at 0.47 is obtained.

Fig. 12
2 graphs labeled a and b. The decreasing graph on the left plots epoch versus loss. Red line, (8, 1.2), (50, 0.38), (100, 0.2). Blue line, (0, 1.2), (50, 0.42), (100, 0.3). The increasing graph on the right plots epoch versus accuracy. Red line, (0, 0.4), (50, 0.87), (100, 0.9). Blue line, (0, 0.48), (50, 0.92), (100, 0.9). Values are estimated.

a Epoch versus Loss graph b Epoch versus Accuracy graph

Full size image

5 Conclusion

In this paper, an automated system has been developed to detect various hematologic diseases such as leukemia, anemia, lymphoma, myeloma, and malaria by using deep learning and image processing techniques. An accuracy of 98.6% with a minimum loss of 0.47 was achieved by the model and was able to classify 7 classes of these diseases from the healthy type. It can be concluded that an automated system saves time and is cheaper than manual testing methods. It can help medical practitioners to a great extent. This research can be further extended to other diseases and a real time hardware model can be developed which could be used in the public health sector as an effective tool.