A Novel Cell Orientation Congruence Descriptor for Superpixel Based Epithelium Segmentation in Endometrial Histology Images

Abstract

Recurrent miscarriage can be caused by an abnormally high number of Uterine Natural Killer (UNK) cells in human female uterus lining. Recently a diagnosis protocol has been developed based on the ratio of UNK cells to stromal cells in endometrial biopsy slides immunohistochemically stained with Haematoxylin for all cells and CD56 as a marker for the UNK cells. The counting of UNK cells and stromal cells is an essential process in the protocol. However, the cell counts must not include epithelial cells from glandular structures and UNK cells from epithelium. In this paper, we propose a novel superpixel based epithelium segmentation algorithm based on the observation that neighbouring epithelial cells packed at the boundary of glandular structures or background tend to have similar local orientations. Our main contribution is a novel cell orientation congruence descriptor in a machine learning framework to differentiate between epithelial and non-epithelial cells.

1 Introduction

Uterine Natural Killer (UNK) cells normally make up no more than $5\,\%$ of all cells in the womb lining and it has recently been shown [1] that an over-presence of UNK cells leads to recurrent miscarriage. Thus UNK testing plays a significant role in clinical diagnosis of recurrent miscarriages. The diagnosis protocol devised by Quenby et al. in [1] calculates the ratio of UNK cells to stromal cells in histology images of endometrial tissue slides stained with Haematoxylin and CD56, which stains UNK cells brown when used with DAB staining. The task is challenging in that the epithelial cells should not be counted in calculating the ratio, which means that epithelium from glands or luminal epithelium from tissue boundary should be excluded from the counting process. The problems of the detection of UNK and stromal cells, and localisation of luminal epithelium from tissue boundaries were addressed in [2]. However, [2] does not solve the segmentation problem of epithelium from glands. In this paper, we present a method designed for segmenting both glandular epithelium and luminal epithelium (examples are shown in Fig. 1(a)) from tissue boundary.

Existing methods of segmenting glandular structure are mainly based on texture and structure. Farjam et al. [3] proposed a variance filter which produces different texture features on lumen and cell regions and the segmentation is accomplished by clustering the texture features. This method is only capable of segmenting lumen regions in our case, due to epithelium and cell regions having similar texture features. Naik et al. [4] used a Bayesian classifier to detect potential lumens and then initialised level set curve on the boundaries of detected luminal area to finalise the segmentation. The drawback of this method is that level set curve is not capable of approximating epithelium boundaries with complex shape and texture. Nguyen et al. [5] first label nuclei, cytoplasm and lumen by colour space analysis and utilise the constitution of these glandular components to achieve the segmentation. Demir et al. [6] constructed an object graph of a set of circular objects decomposed from the image to identify cell and lumen. Then cell objects are used to form the boundary of glandular structures. Recently, Sirinukunwattana et al. [7] proposed a novel Random Polygons Model (RPM) using epithelial cells as the vertices of a polygon to approximate boundaries of glandular at cost of relatively high computational complexity. A major limitation of such structure-based methods is that they rely on prior knowledge of the glandular structures. In our case, epithelium is characterized by strong inhomogeneity, i.e., discontinuity or multiple layers of epithelial cells.

Epithelium is formed by layers of epithelial cells and identification of these cells is a primary task of the segmentation. Generally, there are not significant distinctive features, in terms of colour and morphology, between epithelial cells and stromal cells. However, we observe that epithelial cells normally surround lumen or are located at the boundary of the background region in a locally and neatly oriented manner. Based on this observation, we propose a novel cell orientation congruence (COC) descriptor using a cell and its neighbours’ orientations, which can be used to accurately identify epithelial regions. A major advantage of the proposed algorithm over the above methods is that unlike these methods, our algorithm is not restricted to the detection and segmentation of a closed epithelial structure such as a gland, but it is also capable of segmenting luminal epithelium from the tissue boundary.

Fig. 1.

The cyan window in (a) shows an image region used in (c); ground truth is shown in orange in (b); lumen segmentation is shown in yellow in (c); in (c) and (d), blue dots depict all cells, green dots depict epithelial cells after the classification using the proposed COC descriptor, black bar represents the orientation of a cell, red grids mark superpixels, potential epithelial superpixels are shown in red; and final epithelial segmentation is shown in green in (c) and (e) (Color figure online).

2 Materials and Methods

Endometrial biopsies were collected in a clinic at University Hospitals Coventry and Warwickshire NHS Trust from patients suffering from recurrent pregnancy loss or recurrent IVF treatment failure. The biopsy tissue slides are stained with Haematoxylin and CD56 is used as a marker for the UNK cells. More details about the slides preparation can be found in [1]. The image data in our experiment are image regions manually cropped from the high power fields (HPFs) of digitised images of endometrial biopsy slides at $40\times$ resolution. The image regions are saved in the JPEG format with a resolution of $1,700\times 900$ pixels ($0.25\,\upmu \text {m}$/pixel).

Figure 1(c) shows a block diagram with the intermediate result of each main step of the proposed method. We first identify lumen and background regions to localise the potential epithelial regions. Second, we detect cells in the potential epithelial regions and compute their cell orientation congruence (COC) descriptors. Third, we perform a two-stage classification process. The first stage is to distinguish epithelial and stomal cells in the potential epithelial regions using their COC descriptors and the second stage is to label true epithelial regions using the epithelial cells. At last, the labelled epithelial regions form the final epithelium segmentation. Experimental results show that the accuracy of the proposed method is competitive compared with 3 state-of-the-art methods.

2.1 Localisation of Potential Epithelial Superpixels

Segmenting epithelial cells is the way of achieving epithelium segmentation. It is difficult to directly segment epithelial cells due to them having very similar stained colour and similar morphological appearances as stromal cells in our case. However, epithelium normally covers the exterior of lumen region or is often located near the non-tissue (background) region. Thus, segmentation of lumen and background regions can be used to locate potential epithelial regions. A sample input image is shown Fig. 1(a).

We first separate the input image into the two underlying stain channels, Haematoxylin and DAB (CD56), using a colour deconvolution method proposed in [8]. The lumen and background regions are segmented on the Haematoxylin channel using a lumen segmentation method proposed in [3], which is based on the observation that lumen and cell regions have distinct local standard deviations. The result of lumen segmentation on an image region from the input image is shown in Fig. 1(c).

Next, we decompose the input image into small image patches, the so-called superpixels using the Simple Linear Iterative Clustering (SLIC) algorithm proposed in [9]. In our cases, the superpixel is a small homogeneous region depicting lumen, background or cell. Superpixels generated on an image region from the input image are shown in Fig. 1(c). We classify superpixels into two types: cell superpixels, which depict either epithelial cell or stromal cell regions, and lumen superpixel, which depict either lumen or background region. We classify a superpixel as lumen superpixel if more than half of its region overlaps with the mask of the lumen mask obtained from the lumen segmentation step, otherwise it is categorised as a cell superpixel. Next, we define that a cell superpixel is a first level potential epithelial superpixel if it immediately connects to lumen superpixels. However, thick epithelium is formed by multiple layers of epithelial cells, a layer of superpixels which immediately connects to lumen superpixels is not enough to represent the epithelium in general. To retrieve as much potential epithelial superpixels as possible, we define that a cell superpixel is a second level potential epithelial superpixel if it immediately connects to the first level epithelial potential superpixels. In addition, we remove a potential epithelial superpixel if more than half of its area is segmented (Otsu thresholding [10]) as non-cell in the Haematoxylin channel, which more likely depicts a background region. The result of potential epithelial superpixel localisation for on an image region from the input image is shown in Fig. 1(c).

2.2 Computation of Cell Orientation Congruency (COC) Descriptor

The Haematoxylin channel of the input image is normalised with zero mean and unit standard deviation in pixel intensities by following the approach introduced in [11]. Let us denote the normalised Haematoxylin channel as N(i, j), where (i, j) denotes pixel coordinates. The directional gradients in x and y directions of N(i, j) are calculated and the local orientation O(i, j) of N(i, j) is estimated as follows,

$$\begin{aligned} G_{xy}(i,j) = \big ((G_{x} \cdot G_{y}) \star \varPhi \big )(i,j)\end{aligned}$$

(1)

$$\begin{aligned} G_{xx}(i,j) = \big (G^2_{x} \star \varPhi \big )(i,j)\end{aligned}$$

(2)

$$\begin{aligned} G_{yy}(i,j) = \big (G^2_{y} \star \varPhi \big )(i,j)\end{aligned}$$

(3)

$$\begin{aligned} O(i,j) = \frac{\pi }{2} + \tan ^{-1}\left( \frac{G_{xy}(i,j)}{G_{xx}(i,j)-G_{yy}(i,j)}\right) \end{aligned}$$

(4)

where $G_{x}(i,j)$ and $G_{y}(i,j)$ are gradient images of N(i, j) in x and y direction, respectively, $\varPhi$ is a 2D Gaussian filter of window size w and standard deviation $\sigma$, and $\star$ represents the image convolution operation. O(i, j) is the least squared estimation of the local orientation of N(i, j).

Fig. 2.

An illustration of the cell orientation congruence (COC) descriptor. Black arrow represent the orientation vector of a cell.

Li et al. in [2] demonstrated that their cell detection method based on phase symmetry attains high accuracy in endometrial histology images. Thus, we employ the detection method in [2] to detect cells in N(i, j). The orientation of a cell with its nucleus centred at the spatial coordinates C is statistically represented by the local orientations using its surrounding pixels as follows:

• 1. Pixels in a circular neighbourhood with radius $\alpha$ centred at C are sampled, we use $\alpha =7$ pixels in practice.

• 2. The local orientation, in radians, of the sampled pixels are rescaled to $[0,\pi ]$. To more precisely approximate the cell orientation, we quantise the sampled pixels into seven overlapping orientation bins: $[0,\pi /4]$, $[\pi /8,3\pi /8]$, $[\pi /4,\pi /2]$, $[3\pi /8,5\pi /8]$, $[\pi /2,3\pi /4]$, $[5\pi /8,7\pi /8]$, $[3\pi /4,\pi ]$.

• 3. The mean of the orientation bin containing the highest number of local orientations is then used to represent the orientation of C, denoted as $O_c$.

Next, the neighbouring cell detections in a circular neighbourhood with radius $\lambda$ centred at C are sampled, we use $\lambda =120$ pixels in practice. The circular neighbourhood is uniformly divided into 16 half overlapping orientation bins of width $\pi /4$. The first bin is from $[O_c,O_c+\pi /4]$, the second bin is $[O_c+\pi /8,O_c+3\pi /8]$, and so on. An illustration of the overlapping orientation bins is shown in Fig. 2. The cell orientation congruence descriptor of C is then constructed using the orientations of its neighbouring detections as follows,

$$\begin{aligned} S_{d} = \sum _{i=1}^{N_d}\varOmega _{i}\cos (\varTheta _{i})\end{aligned}$$

(5)

$$\begin{aligned} \varOmega _{i} = \frac{\omega _i}{\sum _{i=1}^{N_d}\omega _i}, \omega _i = e^{\frac{-D_i^2}{2\sigma ^2}} \end{aligned}$$

(6)

where $S_{d}$ is the weighted orientation congruence of the d-th orientation bin, $N_d$ is the total number of sampled neighbouring detection in the d-th orientation bin of C, $\varTheta _{i}$ is the angle between the orientation vectors of C and the i-th neighbouring detection in the d-th orientation bin, $\varOmega _{i}$ is the normalised weight used to indicate the importance of the orientation difference. $D_i$ is the Euclidean distance between C and the i-th neighbouring detection in the d-th orientation bin, and $\sigma =70$ is used in our case. The cell orientation congruence (COC) descriptor is expressed by a 16 dimensional vector as given below,

$$\begin{aligned} COC = [S_{1},S_{2},...,S_{d},...,S_{16}]^T \end{aligned}$$

(7)

2.3 Labelling of Epithelial Superpixels

We perform a two-stage classification to distinguish the epithelial and stomal superpixels. First, the cell detections located within potential epithelial superpixels are considered as potential epithelial cell detections. We compute the cell orientation congruence (COC) descriptors of all potential epithelial cells. Then we employ the random forests classifier using the descriptors to classify the potential epithelial cells. These epithelial cells are marked as either false or true. The classification results of epithelial cells are shown in Fig. 1(d).

Next, a potential epithelial superpixel which contains any false epithelial cell or no cell is initially classified as a non-epithelial superpixel, otherwise it is classified as an epithelial superpixel. A false epithelial superpixel may contain a true epithelial cell, which is also likely to be an epithelial superpixel. A refinement process for the non-epithelial superpixels is performed as follows:

• 1. Let us denote a non-epithelial superpixel as P, we define that a potential epithelial superpixel which immediately connects to P as level 1 neighbourhood of P, denoted as $H_1^i$, and a set of level 1 neighbours is denoted as $\left\{ H_1^i\right\}$;

• 2. We define that a potential epithelial superpixel which immediately connects to $H_1^i$ and also does not belong to $\left\{ H_1^i\right\}$ as level 2 neighbourhood of P, denoted as $H_2^j$, and a set of level 2 neighbours is denoted as $\left\{ H_2^j\right\}$;

• 3. We count the number of true epithelial cells and the number of false epithelial cells within P, $\left\{ H_1^i\right\}$ and $\left\{ H_2^j\right\}$, denoted as $NC_{true}$ and $NC_{false}$ respectively. Then P is corrected to be an epithelial superpixel if $NC_{true}\ge NC_{false}$, otherwise it remains the same.

The epithelial superpixels after the refinement process are merged to form the final epithelium segmentation. The final segmentation of a sample image in Fig. 1(a) is shown in Fig. 1(e). The ground truth is shown in Fig. 1(b).

Table 1. Segmentation accuracies of the proposed method and the other methods on 30 unseen sample images. The Dice scores are reported by the averages $\pm$ standard deviations. The best results are in bold.

3 Results

We calculate the Dice scores of the segmentation results by a gland segmentation accuracy measures introduced in [7] to compare their performance of the proposed method with 3 state-of-the-art methods: [4, 5, 7]. We trained these methods using 5 sample images and the segmentations using these methods are performed on 30 unseen sample images. Since the algorithms in [4, 5, 7] are proposed for only segmenting glandular structures, so we remove luminal epithelium (which come from tissue boundaries rather than glands) in the ground truth while calculating the Dice scores. Table 1 shows the segmentation accuracies of the proposed method and the other methods on 30 unseen sample images. The results show that our method offers superior segmentation accuracy compared with [4, 5, 7].

4 Conclusions

In conclusion, we proposed a superpixel based epithelium segmentation method using a novel cell orientation congruence descriptor. The descriptor is used to discriminate between epithelial and stromal cells based on the observation that the epithelial cells in normal endometrium are packed such that their orientation is more or less similar to their neighbouring epithelial cells. The results show that our method attains a good accuracy which is ready to be employed in practice. In future work, we plan to extend the design of the descriptor, e.g., multi-scale nature of the descriptor, and to conduct a large-scale validation of our method.