Keywords

1 Introduction

Application of image analysis and machine learning algorithms and tools to cell biology is a very active research field aimed to provide fast and objective methods for analyzing the large amount of images produced by modern high-throughput screening platforms available in biological research laboratories [3, 20, 22, 25].

In this work we describe first results related to the development of a multi-component software framework devoted to define automated morphological analysis and classification of Embryonic Stem Cells (ESCs) colonies. The colony-forming (CF) assay is widely used for monitoring the quality of ESC cultures as it currently offers the most sensitive and specific method to quantify the frequency of undifferentiated cells present in a culture. Moreover, it provides a reliable tool also for evaluating quantitative changes in pluripotent cell numbers, following manipulations that may affect the self-renewal and differentiation properties of the treated cells. In a clonogenic assay, under specific supportive conditions, pluripotent mouse embryonic stem cells (mESCs) form tridimensional round-shaped (domed) colonies. After the exposure to conditions that affect their metastability promoting an Epiblast-like phenotype [4] or induce differentiation, cells loose the ability to grow tridimensionally and form irregular and flattened (flat) colonies. The 4-day CF assay, by detecting the ability of mESCs to form domed or flat colonies, allows the composition of test cell populations to be quantified at the single cell level. However, to achieve acceptable statistical accuracy, a high number of cell colonies is required. In this regard, manual counting and classification are tedious, time-consuming, resource-intensive and subjective (operator-dependent). Therefore, the development of a reliable automated colony counter and classifier for such clonogenic assays would reduce time and resources required, while allowing greater statistical accuracy, standardization and reproducibility, thus offering the possibility for greater throughput over extended periods. To this aim, we proposed an experimental software tool which is able to automatically discriminate and quantify domed colonies, raising from undifferentiated self-renewing mESCs, and flat colonies, derived from undifferentiated Epiblast-like or differentiating cells. It is a multi-component framework interfacing different general-purpose software modules, implementing highly accurate algorithms for image pre-processing (spanning from region of interest identification to background removal), segmentation, and feature-based classification. Trained on untreated reference samples as well as on samples treated with reference compounds, the prototypal version of the software has been tested on a dataset of 40 microscopy images of single wells containing cells grown in different conditions. Comparison of the first results from automated versus manual colony segmentation and classification on randomly chosen images proved that the proposed tool is promising to be used as blind tool to support reliable analysis of molecular screening.

The paper is organized as follows. In Sect. 2 we describe the software components, both in terms of functionality and of main models and algorithms. In Sect. 3 we present first results and provide a quantitative performance analysis referring to usual performance metrics for segmentation and classification problems. Conclusions are drawn in Sect. 4.

2 Software System Components

According to the typical image analysis pipeline for High-Content Analysis [22], the proposed steps for automatically discriminating domed and flat colonies in microscopy images can be summarized as follows:

  1. 1.

    Pre-processing, to reduce image artifacts caused by imperfections in the image acquisition process.

  2. 2.

    Segmentation, to separate the cell colonies in each well image.

  3. 3.

    Feature computation, to provide numerical descriptors of each segmented colony.

  4. 4.

    Classification, to finally provide the discrimination and quantification of domed and flat colonies, based on the most discriminating features.

2.1 Pre-Processing

Input images are initially pre-processed, in order to allow an easier and more accurate segmentation of ESC colonies.

First of all, the well area is extracted by the whole image, in order to focus on the actual region of interest (ROI) in all subsequent steps, as exemplified in Fig. 1. This allows us to reduce not only the computational complexity of the entire procedure, but also the potentially misleading influence of non interesting image details (e.g., shadows and dark areas around the well, as shown in Fig. 1-a).

Fig. 1
figure 1

Extraction of the ROI: (a) original image; (b) circular ROI of the well; (c) obtained sub-image, which will be the input for subsequent steps (non interesting pixels shown in black)

Full size image

Relying on the strong intensity discontinuities (edges) provided by the well borders, the circular ROI is accurately obtained through the Hough Transform [10], a powerful tool for the detection of parametric curves in images. It implements a voting process that maps image edge points into manifolds in an appropriately defined parameter space; peaks in this space correspond to the parameters of detected curves. Specifically, the Circle Hough Transform is designed to determine the parameters of a circle when a number of points that fall on its perimeter are known. A circle with radius r and center (x 0, y 0) can be described with the parametric equations

$$\displaystyle{\left \{\begin{array}{l} x = x_{0} + r\cos \theta \\ y = y_{0} + r\sin \theta \end{array}.\right.}$$

When the angle \(\theta\) sweeps through the full 360 degree range, the points (x, y) trace the perimeter of the circle. An image edge point (x, y) is mapped to the 3D parameter space (x 0, y 0, r), voting for all the circles that it could lie on. These votes are accumulated in a 3D array, whose maxima provide the parameters of most prominent circles. In the case of our single-well images, the accumulator maximum provides the well center pixel coordinates and the radius.

The next pre-processing step estimates and then removes the well background, in order to avoid the influence of eventual uneven illumination, as exemplified in Fig. 2.

Fig. 2
figure 2

Background estimation and removal: (a) input image (green color band of the image in Fig. 1-c); (b) estimated background; (c) obtained image, with background removed

Full size image

The goal is achieved through mathematical morphology operations [21], that, based on set theory, provide a tool to extract image components useful for the representation and description of region shape, and for pre- and post-processing of images.

Specifically, we relied on closing top-hat filtering, an operation that extracts small elements and details from a given image. Let f be a grayscale image, and let b be a grayscale structuring element. The closing top-hat transform of f (sometimes called the bottom-hat transform or the black top-hat transform) is given by:

$$\displaystyle{T_{b}(f) = f\bullet b - f,}$$

where • indicates the closing operation. It returns an image containing the objects that are smaller than the structuring element and are darker than their surroundings. In the case of our images, morphological closing f•b using a structuring element b having size greater than any expected cell colony removes all the colonies, thus providing a faithful approximation of the well background (see Fig. 2-b). The subsequent subtraction of the original image by the estimated background provides a uniformly illuminated image of the cell colonies, as shown in the normalized result reported in Fig. 2-c.

2.2 Segmentation

In order to partition each well image into its constituent objects (ESC colonies and background), we devised two different approaches. In the first approach, the image resulting from pre-processing is binarized through Otsu’s method [17] (see Fig. 3-b) and then refined through binary morphological operations. Specifically, refinement involves morphological closing for removing small holes, hole-filling for removing internal holes, and removal of very small objects, i.e., small cell colonies or generic particles of no biological interest (see Fig. 3-c).

Fig. 3
figure 3

First approach to ESC colonies segmentation: (a) input image (the pre-processed image of Fig. 2-c); (b) binarized image; (c) refined binarized image

Full size image

A second approach is based on a well-known variational model, for which we recently proposed efficient numerical solvers [9] with the final aim to develop software modules for modern high-performance environments [8]. In the following we briefly describe the model and the main features of the numerical approach and discuss the subsequent refinement of variational-based segmentation for application to the cell identification problem.

The image segmentation problem can be mathematically formulated in terms of a variational model, i.e., in terms of an energy minimization criterion. We look for a piecewise smooth function u which approximates the original image function f, with u discontinuous across a closed set K that is included in the image domain Ω and represented by a suitable function z. In more details, let \(\varOmega \subset \mathfrak{R}^{2}\) be a bounded open set and \(f \in L^{\infty }(\varOmega )\) the observed gray-level image. The problem can be described in terms of the minimization of the following functional:

$$\displaystyle{ E_{\epsilon }(u,z) =\int _{\varOmega }(u - f)^{2}dxdy +\beta \int _{\varOmega }z^{2}\vert \nabla u\vert ^{2}dxdy +\alpha \int _{\varOmega }\left (\epsilon \vert \nabla z\vert ^{2} + \frac{(z - 1)^{2}} {4\epsilon } \right )dxdy,}$$

where \(u \in C^{1}(\varOmega \setminus K)\), 0 ≤ z ≤ 1 is a function which controls | ∇u | and gives an approximate representation of the set K, α and β are positive coefficients and ε is a positive sufficiently small parameter. The choice of the parameter set, especially of ε, affects the accuracy in the detection of K and it is a main drawback in the numerical solution of 1. Our approach for the solution of 1, known as the Ambrosio-Tortorelli model or phase-field approximation of the Mumford-Shah model [1], is based on the classical approach of Calculus of Variations which consists in writing the corresponding Euler-Lagrange equations, representing the necessary conditions for the minimizer. Euler-Lagrange equations for 1 are a non-linear system of elliptic partial differential equations coupled with Neumann conditions, which can be solved by finite-difference discretization. We proposed to apply a second-order finite-difference scheme and, starting from a block form of the resulting algebraic equations, we applied a first-order alternating minimization scheme based on the non-linear Gauss-Seidel method, accelerated by inner linear iterations. Our numerical results and comparisons with standard numerical techniques show the efficiency and the robustness of our solution approach for increasing sizes of cell colony images. Details on the numerical scheme and on the above results are discussed in [9]. Here, we only point out that results discussed in the following have been obtained, starting from the preprocessed image obtained by the green band of the original RGB image, when the parameters in 1 were set as follows: α = 1, β = 2 and \(\epsilon = 10^{-3}\).

The obtained function z (see Fig. 4-a), also known as the edge set of the image, represents the set K where the piecewise smooth function u (also known as the restored image) is discontinuous and allow us to identify the borders of the cell colonies. Indeed, starting from the edge set, we applied subsequent refinements, based on binarization (see Fig. 4-b), hole-filling for removing internal holes, and finally removal of very small objects, analogous to those of the previously described approach, which leads to the cell colony segmentation (see Fig. 4-c).

Fig. 4
figure 4

Second approach to ESC colonies segmentation: (a) edge set of the preprocessed image of Fig. 2-c; (b) binarized version of the edge set; (c) refined binarized image

Full size image

2.3 Features Computation

In order to classify the ESC colonies and be able to differentiate between domed and flat ones, we designed a general approach to feature selection and classification, based only on distinctive feature estimates that can be computed by the available images.

The most distinctive features that allow the biologist to discriminate the two kinds of colonies concern their shape: domed colonies appear compact and rounded, as opposed to flat colonies that spread more or less throughout the well (see Fig. 5)

Fig. 5
figure 5

Examples of ESC colonies extracted from the image in Fig. 1: (a) a domed colony (size about 4 % of the original image); (b) a flat colony (size about 7 % of the original image)

Full size image

This prior knowledge is exploited by using geometric models locally estimated for each colony in a well. Among the considered shape features, the most significative are those reported in Table 1. The Area A(X) provides the number of pixels of the object (colony) X and the Perimeter P(X) is computed as the sum of the distances between consecutive pairs of boundary pixels of X. Solidity S(X) is given by the ratio of the area A(X) of the object X and the area A(CH(X)) of its convex hull [5]. Solidity ranges in [0,1], producing low values when the shape of the object shows many concavities (as in the flat colonies) and high values when the shape of the object shows few or zero concavities (as in the domed colonies). The irregularity of a contour is expressed through Compactness C(X) [7], given by the ratio between the area A(X) of the object and P 2(X)∕4π, the area of a circle having the same perimeter P(X) as the object. Indeed, this feature, ranging in [0, 1], provides low values for scarcely compact shapes (as in the flat colonies) and high values for compact shapes (as in the domed colonies), reaching its maximum for the most compact shape: the circle. Other adopted shape features are: Eccentricity, that is the eccentricity of the minimum area ellipse including the colony image; Min and Max axis length, that are the minimum and the maximum axis length of the ellipse containing the colony image, respectively; Nsegments, that gives the ratio of the number of contour segments and the area of a colony image.

Table 1 Some of the adopted shape (S) and texture (T) features and their values for the ESC colonies of Fig. 5
Full size table

Besides shape estimates, textures have also been considered. In an image, the intensity variations which define a texture are mostly related to physical variations in the scene (such as pebbles on the ground). It is very difficult to model these variations and no precise definition of texture is present in computer vision literature [6]. For this reason, textures are usually characterized by intensity value variations in the two-dimensional space of an image. The adopted texture features of each object can be described in terms of the gray-level co-occurrence matrix (GLCM) [13], that allows us to capture the spatial dependence of gray-level values which contribute to the perception of texture, by showing how often different combinations of pixel brightness values occur in an image. The GLCM matrix is a square matrix M of dimension n, where n is the number of different gray levels in the image. Each element M(i, j) is generated by counting the number of times a pixel in X with gray value i is adjacent to a pixel in X having gray value j. Each element M(i, j) is then normalized so that the sum of all elements of M is equal to 1, and can thus be considered as the probability of occurrence of adjacent pixel pairs having gray level values i and j in the image. Among the considered texture features, the most significant are those reported in Table 1. The Entropy H(X) measures the randomness of the gray-level distribution in the colony image X, while the Contrast C N(X) is a measure of the intensity contrast between a pixel and its neighbors over X, assuming null value for a constant image. The Energy E(X) yields the smallest value when all entries in M are equal; it is 1 for a constant image. The Correlation C R(X) measures the dependency of gray levels on those of neighboring pixels. Further texture features that have been considered are the Homogeneity, that measures the closeness of the distribution of elements in the GLCM to the GLCM diagonal, and Centropy and Bentropy, that measure the entropy of color and grayscale colony image, respectively.

2.4 Classification

In machine learning, supervised classification methods aim at inferring a classification rule from a class-labeled set of examples described by a set of features. The inferred rule is then used for predicting the class of further unlabeled data. Therefore, a classifier can be considered as a mapping from a feature space to a set of classes. There exist several classifiers in literature, since there are several ways of building up such a mapping. For example, the mapping can be described by a set of induction rules, like in the tree-based classifiers, or the classifiers may be expressed as a linear separator in the original feature space, like for the perceptron or the support vector machines (SVM) classifiers. The choice of a good classifier is not a simple task, as it depends on several choices [14], including the complexity of the classification rule, the size of the training set, and the number of features.

For the classification of ESC colonies into domed and flat ones, we trained a set of binary classifiers from a set of labeled images described by the selected features (see Sect. 2.3). The classifiers have been chosen based on the criterion that the classification rule can be expressed either by a set of induction rules or by an easy interpretable mathematical function, where the importance of each feature in the classification rule is easily interpretable. This suggests which features are more relevant for labeling an image.

Specifically, we adopted three classifiers based on decision trees, that return induction rules for deciding the label of an instance: CART [2], J48 [19] and Adaboost [11] using as weak learners CART classifiers (Adaboost+CART). Among the classifiers searching for a linear separator in the feature space between instances with different labels, we used two versions of the SVM classifier: the SVM linear [24] and the SVM linear with hinge loss [18]. Finally, we adopted the Naïve Bayes classifier [15], a probabilistic algorithm which assigns a probability to each instance for each label, according to the highest label probability. All the above classifiers are available through the open source Weka software [12], that allows testing several classifiers in a very user friendly way.

3 Experimental Results

The proposed software framework has been tested on images of a CF assay done by plating mESCs in 96-well plates at the density of 500 cells∕cm2 in medium for propagation of undifferentiated cells (with Lif and serum). After cell adhesion (4 h), cells were treated with reference compounds controlling their metastability or differentiation, L-Proline [4] and Retinoic Acid [23], respectively, added at three different concentrations and allowed to grow for 4 days. Single well bright field images were acquired under a Leica MZ16FA stereo microscope.

Randomly chosen images of treated and untreated wells have been manually annotated, in order to provide the ground-truths for objective evaluation of the software components.

3.1 Performance Metrics

Validation of all the system modules has been performed in terms of different metrics frequently adopted in the literature [16], namely Precision and Recall

$$\displaystyle{Recall = \frac{TP} {TP + FN},\;\;\;\;Precision = \frac{TP} {TP + FP},}$$

where TP, FN, and FP indicate the total number of true positives, false negatives, and false positives, respectively. Recall, also known as detection rate or sensitivity, gives the percentage of detected true positives as compared to the total number of true positives in the ground truth. Precision, also known as positive prediction, gives the percentage of detected true positives as compared to the total number of items detected by the method. Using the above mentioned metrics, generally a method is considered good if it reaches high Recall values, without sacrificing Precision. A further metric F 1, also known as F-score or F-measure, given by the weighted harmonic mean of Precision and Recall

$$\displaystyle{F_{1} = \frac{2 {\ast} Recall {\ast} Precision} {Recall + Precision},}$$

provides a single measure that can be used to “rank” different methods.

For validating the classification accuracy, we adopted also the False Positive Rate (FPR)

$$\displaystyle{FPR = \frac{FP} {FP + TN},}$$

where TN indicates the total number of true negatives.

3.2 Evaluation of the Segmentation Step

For the segmentation step, Recall, Precision, and F 1 metrics have been adopted both as pixel-based and as object-based measures, depending on wether true positives (resp. false negatives) are intended as true positive (resp. false negative) pixels or objects (colonies), respectively. While the pixel-based measures provide hints on the fine-grain segmentation accuracy, the object-based measures provide hints on the colony counting ability of the proposed modules. Average segmentation accuracy of the two segmentation approaches in term of the above metrics is reported in Table 2 and some details of the achieved results are reported in Fig. 6.

Table 2 Average segmentation accuracy of the two segmentation approaches
Full size table

The higher pixel-based accuracy values of the second approach (fourth column of Table 2) indicate a better ability of the variational model-based segmentation to precisely detect the colony contours (e.g., compare Figs. 6-g and 6-h). However, this ability leads also to fuse different colonies that are very close (e.g., see Fig. 6-d), resulting in lower object-based accuracy values (fifth column of Table 2).

Fig. 6
figure 6

Evaluation of the segmentation step: details of the original image of Fig. 1 (first column), of the ground truth (second column), and of corresponding results obtained by the first and second segmentation approaches (third and fourth column, respectively). Green pixels superimposed on the original image indicate pixels included into the segmentation

Full size image

Both the approaches report very few false positives and false negatives. Examples of false positives, e.g., segmented colonies that have no interest for the biologist, are reported in Figs. 6-g and 6-h, where the detected small colony has been excluded by the biologist in the ground truth reported in Fig. 6-f.

Most of the false negatives are linked to adjacent colonies, barely distinguishable by an untrained human eye, that are segmented as a single colony. An example is given in the third row of Fig. 6, where the adjacent colonies of the original image (Fig. 6-i) have been manually segmented in the ground truth as two separate colonies (Fig. 6-j) based only on the biologist experience. The two approaches, instead, both provide a single segmented colony (Figs. 6-k and 6-l). This analysis of false positives and false negatives suggests that features other than contrast or edges, able to better describe the discrimination ability of the trained eye, could help avoiding the few segmentation errors reported. Hints could be provided by the subsequent steps of feature computation and feature-based classification.

Although the segmentation results are quite accurate for both the proposed approaches, a suitable combination of them, together with a feedback from the subsequent steps of feature computation and classification, could help further improving segmentation accuracy.

3.3 First Classification Results

Each of the selected classifiers (see Sect. 2.4) has been trained using the selected features (see Sect. 2.3) of the colonies included into the constructed ground truths. The statistical validation of results has been obtained using a 10-fold cross validation. This procedure consists in splitting the training set in ten parts; at each step, one of those parts is used for testing and the remaining nine are used for training. Therefore, the performance metrics have been evaluated as the average among the one hundred repetitions of the ten-fold cross validation.

Table 3 10-fold cross validation of classification results
Full size table

Table 3 reports the classification performance values for each classifier, suggesting that all the six classifiers are very accurate in predicting the label of an instance. In the case of SVM with hinge loss, the number of false positives is reduced up to 1 %. This is exemplified in Fig. 7, where we compare the manual classification provided by the biologist with the automatic classification computed using the SVM with hinge loss for the well of Fig. 1.

Fig. 7
figure 7

Example of classification results: (a) manual classification; (b) automatic classification using SVM with hinge loss

Full size image

Table 4 reports the weights (one for each feature) of the normalized equation of the hyperplane that separates domed vs. flat colonies in the feature space. Positive weights are associated to domed colonies. The higher the absolute value of the weights, the more the feature is relevant for discriminating the classes. Also the other classifiersFootnote 1 confirmed that, as expected, Solidity and Compactness shape features are the most discriminant features for our binary classification. Among texture features, Correlation is the most relevant.

Table 4 SVM with hinge loss: shape (S) and texture (T) features ordered by the absolute value of (normalised) weights
Full size table

4 Concluding Remarks

In this paper we present a novel software framework for the segmentation and classification of microscopic images obtained by CF assays of mESCs and suitable for high-throughput applications. Our main aims were to use general-purpose software components for image processing and machine learning, eventually developed in different projects, to support biological experiments within a multidisciplinary context. First results show that our approach is promising to be used as blind tool to support reliable analysis of molecular high-throughput screening. Future works will include comparisons in terms of efficiency, reliability and flexibility with some existing application-specific software tools and the development of a user-friendly software interface for biologists without deep knowledge in computer programming.