1 Introduction

Cancer is one of the leading causes of mortality worldwide. In these years, the number of deaths due to cancer continues to increase, and in 2016 it became approximately 30% of the cause of death in Japan. In the field of histopathology, the evaluation of the disease status is based on cell nuclei information of the tissue images, and appropriate treatments are required for each cancer and grade. Therefore, it is necessary to extract the region of cell nuclei accurately. However, cells have complex structures (see Fig. 1), and they come from various situations [1]. For that reason, manual analysis is difficult for pathologists. Digital pathology and microscopy images play a significant role in decision making for disease diagnosis, since they can provide extensive information for Computer-Aided Diagnosis (CAD). It realizes a quantitative analysis of digital images with a high throughput processing rate.

Currently, there are several review papers on automated pathological image analysis. The review paper published by Gurcan et al. [2] summarized the CAD system technologies for image analysis, which covers preprocessing, nuclei segmentation and gland segmentation, feature extraction, and classification. Irshad et al. [1] provided a survey on the methods for nuclei detection, segmentation and classification in hematoxylin and eosin (H&E) and immunohistochemistry (IHC) stained histopathology images. Xing et al. have proposed the review on robust nuclei/cells detection and segmentation in digital pathology and microscopy images [3]. According to [2], despite a lot of automated methods for nuclei segmentation have been proposed, it is difficult for pathologists to find a good one to use since there are plenty of competing segmentation methods with different capbilities.

In this survey, we focus on automatic computational methods for nuclei/cells segmentation and review some of the important techniques published in the past years. Also, we also surveyed commonly used modality for staining, preprocessing and some basic methods. The organization of this paper is as follows. Section 2 introduces the imaging modalities in histopathology and emphasizes the challenges in computational nuclei segmentation. Section 3 provides the methods of preprocessing and segmentation in histopathology. Section 4 addresses the challenges and clarifies the gap in nuclei segmentation. We show future directions and problems in nuclei segmentation and conclude with a discussion. A list of symbols and notation commonly used in this survey paper is shown in Table 1.

Table 1 Abbreviations used in the survey
Full size table
Fig. 1
figure 1

Different types of nuclei. a LN, b EN, c EN (cancer), d EN (mitosis)

Full size image

2 Basics of Digital Pathology and Challenges

Fig. 2
figure 2

Examples of hematoxylin and eosin and IHC staining images. a H&E. b IHC

Full size image

2.1 Staining and Image Modality in Digital Pathology

In digital pathology, the microscopic examination of surgical specimen or biopsy is conducted onto glass slides to study cancer expression, genetic progression, and cellular morphology for cancer diagnosis and prognosis. For tissue constituents visualization under a microscope, tissue images are dyed with one or more stains.

H&E staining is one of the staining methods used worldwide in the field of pathology. H&E staining has been used by pathologists for a long time and is still widely used for visualization of tissue features under a microscope. Hematoxylin stains nuclei in dark blue color, and eosin stains other structures (cytoplasm, stroma, etc.) with a pink color (Fig. 2a), [1]. Nuclei are diagnostically meaningful to indicate benign or malignant cancer.

Immunohistochemical (IHC) is a type of the staining methods, which is usually used as a cancer diagnosis to determine whether the obtained tissue is benign or malignant. Figure 2b shows an example of IHC staining images. IHC can help to determine the region of a tumor and we can select an appropriate treatment by investigating the appearance of specific proteins.

Papanicolaou staining is a typical staining method for morphological observation. This method is also an important method like H&E staining. Papanicolaou staining method is almost the same as H&E staining for the cell nuclei, but the cytoplasm can be represented by three kinds of colors. However, this method requires many steps to acquire results. It means that the accuracy of the obtained result may or may not depend on facilities. The normalization of staining is required in the method. They are basic staining methods to study for cancer diagnosis and prognosis under a microscope.

2.2 Challenges in Nuclei Segmentation

There are various types of nuclei, and it depends on several factors such as nuclide, malignant tumor, their life cycles, etc. In the case of white blood cells, there are several classes, e.g., Neutrophils, Eosinophils, Lymphocytes. For instance, lymphocytes are essential for immune cells that occupy most of the white blood cells. The variety of blood cells makes nuclei segmentation very difficult.

Automated nuclei segmentation is one of the well-studied topics in the field of digital pathology, and many methods have already been proposed. However, these proposed methods are not perfect and cannot be applied to all types of histopathology images due to variability caused by factors in slide preparation (dyes concentration, damages of the given tissue sample, etc.) and image acquisition (presence of digital noise, specific features of the slide scanner, etc.). Furthermore, nuclei in the given image are sometimes overlapped each other and distributed non-uniformity. All of them affect not only nuclei segmentation but also prognostic prediction and diagnosis. All image processing approaches have to overcome these issues, and also should be robust methods to keep accuracy high in any situation.

3 Methods for Nuclei Segmentation

Nuclei segmentation is an important step in cancer diagnosis and grading, prognostic prediction. Since tissue properties are different in each disease stage, the information of nuclei is critical for evaluating disease progress and its severity. Therefore, various approaches are required at each step in the processing. Nuclei segmentation mainly consists of two important steps, preprocessing and segmentation. We next introduce the standard preprocessing methods that are used in nuclei segmentation methods for histopathology images. We further review salient nuclei segmentation methods that were published in the recent years.

3.1 Preprocessing

To extract expected information from given images, preprocessing is necessary and usually performed. In the histopathological image processing, the structure and status of cells differ depending on each disease. There are many types of preprocessing methods, and some of them consist of the combination of basic techniques.

3.1.1 Denoising

In this process, filter techniques are often used to remove the noise in the images. Depending on the purpose, it is possible to obtain the appropriate results by emphasizing the feature (expansion/contraction, etc.) and blurring to reduce the noise component (averaging/median, etc.). For instance, a median filter was used in [4, 5]. A median filter can remove noise without blurring contours of desired objects. Understanding feature(s) of each filtering technique is essential in preprocessing.

3.1.2 Color Processing

In image processing, color plays an important role. The outline of what is drawn in the image can be recognized by color. Images give different impressions only by changing the color space. The color information serves as an index for dividing an area and is a useful feature for distinguishing objects in the image. By converting the color system of the given image, it is possible to make it easy to obtain the intended information. It is also possible to extract the area for analysis and diagnosis from the brightness gradient by extracting pixel values. Generally, RGB color space can be converted into several color space.

CIE L*a*b* color system is designed to approximate human vision. In general, the human sense tends to be proportional to the logarithm of the stimulus value, not proportional to the stimulus value itself. The sense of brightness is no exception. In [6], they transform the histopathology images from the RGB to the CIELAB color space, which is based on human eye perception, to exploit this property. In the three L*a*b*, defined by CIE, the a* component, which divides the color axis between magenta and green, was used for discrimination. A double-threshold was used on the a* channel to discriminate between connective tissue, background and myocytes [6].

Conversion of a color image is quite a common way in digital image processing. HSV (Hue, Saturation, Value) transform is widely used in image analysis methods for segmentation [7]. HSV color system is more suitable to convey color information compared to RGB color model, and not vulnerable to illumination. Hue information in the HSV model is a significant feature for cell segmentation because cell nuclei and cytoplasm will absorb different coloring agents, [4]. There are some studies using not only one of the HSV color factors [7, 8] but also all factors. There is also HSI color space, which is similar to HSV color space [9].

3.1.3 Seeds Detection

In segmentation, detection, and classification of cells and nuclei, they are not necessarily independent. Therefore, it is difficult to directly segment cells from microscopic images when they are in contact with each other. In carrying out various treatments, separating contacting cells are required. This process needs the identification of seed points where there is one marker per cell. For instance,

Al-Kofahi et al. identified the seed point of cells with the multiscale Laplacian of Gaussian (LoG) filter [10]. LoG filter [11] consists of Gaussian and Laplacian filters. After smoothing the image with the Gaussian filter(s) to reduce noise, the boundary is extracted with Laplacian filter. Since Laplacian filter acts as a quadratic differential operation(s), noise is easily emphasized, but it is reduced by smoothing the image beforehand with Gaussian filters.

Parvin et al. [12] detected cell seed points by a method using repetitive voting technique. Xin Qi et al. [13] mentioned the views on the problems of these methods and proposed a new method based on parallel seed detection and repulsive level sets.

3.1.4 Color Normalization

Image processing specialists often compare some digital images that were captured under different light condition. In such cases, it is required to convert the mean and standard variation of the image brightness value. Such techniques are usually called color normalization, and many color normalization techniques have been proposed. For instance, histograms or quantile normalization are used for color normalization. In the case of color image, distributions of three color channels are often normalized separately. By adjusting the unevenness of brightness with these techniques, the performance of the subsequent processing can be improved. Most segmentation and detection methods use the RGB color system, but it is not a perceptual color model. For that reason, other color models such as HSV, CIE L*a*b*, and Luv are usually used.

3.1.5 Region of Interest (ROI) Detection

The region of Interest (ROI) means the area selected for filter or other operations. The ROI is defined and expressed by a binary mask. This is an image with the same size as the target image, The pixel values that correspond to ROI are set to 1, and other regions are set to 0. In some cases, the regions are defined based on the area that consists of polygons enclosed by consecutive pixels, or by intensity ranges.

In some frameworks, noise reduction and ROI detection are performed simultaneously. For example, the ROIs were selected by excluding regions with little content and noise at the preprocessing step [14]. For nuclei level feature computation, noise reduction is succeeded by ROI detection to determine the nuclei region [15].

Thresholding is common for ROI detection. Lee et al. [16] normalized the ROI corresponding to the nuclei region from the microscopic image before identifying it. After normalization, using the Otsu thresholding method [17], ROI was detected from the image by combining global thresholding and local thresholding.

3.2 Segmentation Methods

3.2.1 Thresholding

Table 2 Summary of thresholding based nuclei segmentation methods for histopathological images
Full size table

Thresholding is the easiest method for image segmentation. The basic idea of thresholding is to replace each pixel in the image with a white pixel if the pixel value exceeds a certain threshold and replace it with a black pixel if not. There are roughly two methods for determining the threshold T to be used. The first method is a global threshold, and the other is a local threshold approach. A global threshold value can be determined by using computational methods like the Otsu method. A histogram of the entire image is obtained, and then a threshold value is determined for the whole image from the result of the histogram. In the local (adaptive) threshold approach, the average of pixel values and the standard deviation around the target pixel to be binarized, and the threshold value is determined considering them. After this, the given image is converted into a binary image, and we can extract desired objects easily from the obtained binary image.

Thresholding approach works well and is the most efficiently conducted in HSV color space. There are variety of variants on nuclei segmentation using thresholding [8, 18,19,20,21,22,23,24, 26,27,28,29,30,31,32,33].

For instance, Lu et al. introduced the segmentation method by using the local (adaptive) thresholding in cutaneous images [18]. Gadgil et al. used the same approach as the preprocessing for the nuclei segmentation [20]. Thus, thresholding is very important as not only merely segmentation but also preprocessing for applicative segmentation. Table 2 provides the comparison of different thresholding methods for nuclei segmentation methods along with the results obtained. We highlight the prominent preprocessing steps utilized as well.

3.2.2 Watershed

Table 3 Summary of watershed based nuclei segmentation methods for histopathological images
Full size table

Watershed is one of the segmentation methods and used in many papers [23, 28, 30, 34,35,36,37,38,39,40,41]. As the first step of this approach, specific pixels called makers are determined, and then the markers are expanded gradually like flooding. In Watershed, the surrounding regions of makers are called catchment basin, and each pixel value is treated as a local topography. Catchment basins are separated topographically from adjacent catchment basins by maximum altitude lines called watershed lines. It allows classifying every point of a topographic surface as either belonging to the catchment basin associated with one of the local minimum or to the watershed line. Details about watershed approach can be found in [42]. The Watershed transformation is usually computerized on the gradient image instead of the intensity image.

Segmentation using the watershed transform works well if the position of the front object and the background can be identified or marked. A marker-controlled watershed is used in [34, 35]. This algorithm considers the input image as a topographic surface and simulates its flooding from specific seed points or markers. Table 3 provides the details of watershed based approaches. The highest accuracy was reported by Rajyalakshmi et al. [38], wherein a modified the marker-controlled watershed obtained the segmentation accuracy of 95.79 (normal), 95.56 (invasive) %.

3.2.3 Morphological Operations

Table 4 Summary of mathematical morphology filters based nuclei segmentation methods for histopathological images
Full size table

Morphology is a set-theoretic approach that considers an image as the elements of a set and process images as geometrical shapes. This approach is defined by the operation(s) between the processing target images and the structuring elements. The two basic morphological operations are Erosion and Dilation. The effect of these operations is to shrink (enlarge) the boundaries of foreground pixels. Two other major operations in morphological approach are Opening and Closing. The opening operation is an erosion of an image followed by dilation, and it eliminates small objects and lines. Closing is a dilation of an image followed by erosion, and it fills small holes in the image. White and Black Top-Hat Transforms are two other operations derived from morphology. They allow extracting small elements and details from given images. The white top-hat transform is defined as the difference between the original image and the opening images. The Black Top-Hat Transform is defined as the difference between the original image and the closing images. There are methods that exploit morphological operations along with other other techniques such as thresholding [22, 43], level sets [7, 48], k-means clustering [44, 45], and PCA [45], Hough transform [47]. Table 4 provides an overview of these morphological techniques in nuclei segmentation. As can be seen, highest accuracy was obtained by a hybrid method that uses anisotropic diffusion filtering, and morphological toggle filter [46] on two different datasets.

3.2.4 Active Contour Models (ACMs) and Level Sets

Table 5 Summary of active contours, level sets based nuclei segmentation methods for histopathological images
Full size table

Active contour models (ACMs) or deformable models and these are widely used in image segmentation [49,50,51,52,53,54]. In the approaches, a segmentation boundary is expressed as deformable splines, and an energy function, which is defined by gradient information, determines the shapes by seeking to minimize the energy function [62]. In case of nuclei segmentation, the contour points that yield the minimum energy level form the boundary of nuclei. The energy function is often defined to penalize discontinuity in the curve shape and gray-level discontinuity along the contour [63].

There is two main form of ACMs. An explicit parametric representation of the contour, which is called Snakes [64], is robust to image noise and boundary gaps as it constrains the extracted boundaries to be smooth. It is, however, difficult for Snakes to separate or segmentation in the case of multiple objects. Alternatively, the ACMs called Level Set is specifically designed to handle topological changes [13, 48, 55,56,57,58,59,60,61], but they are not robust to boundary haps and have other deficiencies as well [65]. The basic idea is to determine level curves from a potential function. These are iterative schemes thereby making computationally expensive to apply on large-scale histopathological images e.g. whole-slides. However, as can be seen in Table 5 active contours with level set implementations have been applied to nuclei segmentations.

3.2.5 Graph Cuts

Graph-based image segmentation approaches model one image as a weighted graph, in which each node associates a pixel or superpixels in the image and each edge weight between two nodes corresponds to the similarity between neighboring pixels of superpixels. According to a specific criterion, the graph is partitioned into multiple sets, each representing an object segment in the image [3]. Max-flow/min-cut algorithms have been widely applied to image segmentation in the field of computer visions and medical image analysis.

In graph Cuts algorithm, graphs are created from input images, and graph division is realized using max-flow/min-cut algorithm [40, 66, 67]. The graph consists of nodes corresponding to each pixel of the image, and terminals called source and sink. The edge connecting each node is called n-link, and the edge connecting source (S) and sink (T) terminals from each node is called t-link. The input object and background are called seed and the graph is divided into the object (nuclei), and the background by using max-flow/min-cut algorithms for the created graph. Details on graph cuts algorithm can be found in [68]. Table 6 provides a summary graph cut based methods for nuclei segmentation.

Table 6 Summary of graph cut based nuclei segmentation methods for histopathological images
Full size table

3.2.6 K-Means Clustering

K-means clustering is one of the basic clustering techniques.and also, it is an iterative method used for partitioning the given image to K clusters (regions). Many techniques using K-means approach has been proposed like [9, 31, 41, 44, 69,70,71,72,73,74]. The basic algorithm of k-means is as follows.

  1. a)

    Pick K cluster centers, either randomly or based on some heuristic.

  2. b)

    Assign cluster label to each pixel in the image that minimizes the distance between the pixel and the cluster center.

  3. c)

    Recompute the cluster centers by averaging all the pixels in the cluster.

  4. d)

    Repeat steps b) and c) until convergence is attained or no pixel changes its cluster.

In basic k-means clustering techniques, Euclidean distance is usually used for the distance between the cluster center and the data when the cluster is comparatively isotropy. As more highly technique, fuzzified k-means clustering method “Fuzzy c-means” is sometimes used. The fuzzy c-means method uses possibilities that shows how the clusters overlap each other for clustering. The nuclei segmentation using k-means clustering method are proposed in [75, 76]. The segmentation method using superpixel algorithm is also proposed, and these are called simple linear iterative clustering (SLIC) [77, 78]. For instance, Cheikh et al. [77] applied the SLIC approach, that clusters pixels in the five-dimensional color and image plane space to efficiently generate compact, nearly uniform small regions. The result of their approach using the SLIC algorithm was better than watershed-based methods. The differences are typically based on the pixel value, texture, and location, or a weighted combination of these factors. Its robustness depends mainly on the initialization of clusters [3]. Table 7 details the k-means clustering based approaches applied to nuclei segmentation.

Table 7 Summary of k-means clustering based nuclei segmentation methods for histopathological images
Full size table

3.2.7 Segmentation with Deep Learning Techniques

Deep learning is based on artificial neural networks (ANN) which is a system imitating the mechanism of human nerve cells (neurons). Although the performance of deep learning approaches heavily depends on the number of learning data, the algorithm of deep learning is generally smart compared to other methods. In these years, this approach is often used for automatic classification and segmentation, Fig. 3. The algorithm can classify and segment by learning features from mass data, and several methods have already been established. These methods are introduced in literature [79].

Deep Belief Network (DBN) [80], Fig. 4, was developed as one of the methods of deep learning. After this, various architectures such as Stacked Auto Encoder (SAE) and Convolutional Neural Network (CNN) appeared. CNN is the common and worldwide neural network method for image processing. CNN has obtained great success in various tasks of computer vision and biomedical image analyses. Generally, when we use traditional and typical neural networks and handle two-dimensional (matrix) data like an image, the data structure should be converted into one-dimensional (vector) data for learning. On the other hand, since CNN can handle the two-dimensional data as learning data, it is expected that CNN works well the data that has information of positional relationship(s). Figure 3 illustrates an example of the basic architecture of CNN. CNN consists of convolution layer(s) and pooling layer(s). The convolution layer compresses the features in small area of the given image, and the pooling layer generates new feature maps obtained by the convolutional operation.

Fig. 3
figure 3

An example of convolutional neural network (CNN) architecture used in [81]

Full size image
Fig. 4
figure 4

Deep belief network (DBN) architecture used in [80]

Full size image

In [82,83,84,85,86,87], the authors used CNN approach for nuclei segmentation and classification. For instance, Kumar et al. [82] presented a detection method of boundary using three-class CNN (CNN3) for touching nuclei. In the literature, they used The Cancer Genome Atlas (TCGA) to make experimental materials [88]. They carried out color normalization as preprocessing. By adding the third class, it is possible to find inter-nuclear boundaries irrespective of the configuration of the crowded nuclei.

Fig. 5
figure 5

CNN design used in [84], which was based on the U-Net model

Full size image
Table 8 Summary of machine learning (deep learning) based nuclei segmentation methods for histopathological images
Full size table

Isaksson et al. [84] proposed semantic nuclei segmentation method using CNN. Figure 5 shows their network design. Due to the limitation of the number of annotated data, they chose to expand it artificially using data augmentation. The data augmentation is used for rotation, enlargement, and reduction when the experimental data is not enough. In [85], Fu et al. also used data augmentation for their experiment. Consequently, Isaksson et al. [84] got a segmentation accuracy of \(78\%\).

Recently, segmentation and classification using deep learning approaches are used widely to obtain state-of-the-art for disease analysis and diagnosis in various biomedical imaging domains. As can be seen in Table 8, CNN models obtain good nuclei segmentations across datasets in H&E stained images with consistently high accuracy. However, the interpretability of such deep CNN models is not easy and despite the adaptability across datasets, deep learning models still require fair bit of training that in turn requires high computational cost. Nevertheless, using deep learning models and especially CNN models are useful in avoiding adhoc filtering, thresholding and feature selection steps in the previous nuclei segmentation techniques surveyed. We also note the lack of large-scale histopathological datasets dedicated to nuclei segmentations benchmarking, and current deep learning models require domain related training data in achieving higher accuracy.

4 Discussion and Conclusion

Since late 1990’s, research projects on nuclei segmentation have been reported, but the number of the published papers is not many because the digital equipment was accessible in the limited medical/clinical areas. Thanks to the recent developments in digital pathology, a lot of methods for cancer detection and grading approaches have been proposed. Accordingly, the number of published papers on nuclei segmentation methods has been increasing. However, most of the studies use their in-house datasets as experimental materials. Of course, there are various image modalities, staining method, and it is impossible to do in the same condition. Therefore, evaluation criteria depends on pathologists. Due to this reason, evaluating the obtained results may or may not be correct. To evaluate the performance of these methods under the same condition, we need public datasets for histopathological image processing and analysis. Currently, there are several public datasets for nuclei/cell detection and segmentation approaches, but they cannot cover all situation completely. More datasets will be required to expand the field of histopathological image processing and analysis.

We should also address the development of a more robust method that can be used with different staining, scanner, lighting conditions, and magnifications. Segmentation methods such as thresholding, region growing, and watershed approaches can locate cell nuclei but do not work well when they try to segment the touching and overlapping nuclei. Deep learning based on neural networks are well-suited to tackle nuclei segmentation under complex conditions. Due to the lack of availability of large-scale labeled data in the histopathological community, crowd-sourced or generative adversarial networks (GANs) based approaches can be employed. We envisage top performing deep learning models, in particular, methods based on the already successful CNNs can further be expanded to include domain-specific priors for improving the histopathological nuclei segmentation.

It is still one of the hot topics for nuclei segmentation to deal with overlapped and clustered nuclei. Several methods for the challenges have been proposed so far, but the problem has not been completely solved yet. The methods such as thresholding [25, 32, 33], marker-controlled watershed [34, 35], level set [13, 59], graph cut [67] approaches have been developed to separate overlapping, touching and clustered nuclei. By using these methods, cell nuclei that are slightly touching/overlapping can be separated, but they sometimes do not work well when there are many touching/overlapping nuclei in the given image. Also, these methods are not suitable for large scale nuclei segmentation, for example in whole-slide images. For instance, when we use marker-controlled watershed, true nuclei markers are needed. ACMs, clustering and graph-based methods may need plenty of computational costs when applying to WSI. Therefore, the demand for establishing a robust nuclei segmentation method, which is adaptable to WSI and a large number of overlapping and touching nuclei, will be grown in the field of image analysis for digital pathology and microscopy images.