1 Introduction

Pollen grains that are produced by higher plants (angiosperms) contain proteins that may provoke allergic reactions in patients with hay fever or asthma who have been sensitized by previous exposure to these so-called allergens. For diagnostic purposes and adequate treatment, it is relevant to know when the allergenic pollen grains are present in the air. Therefore, a large number of pollen-counting stations has been established across Europe. Currently, pollen recognition and counting is performed by qualified researchers through microscopic examination of pollen slides. Although highly reliable, this method is labor intensive and requires extensive training of the researchers involved. As a result, it is not performed daily. An automated pollen recognition system would make this procedure less labor intensive and enable us to present patients with a prompt daily report.

So far, different systems have been proposed in the literature, some of which require advanced microscope equipment, such as confocal laser scanning microscopes (Ronneberger, Schultz, & Burkhardt, 2002). Unfortunately, such advanced requirement hampers a common application in daily practice. Other projects aimed at pollen identification use standard optical microscopes. Li and Flenley (1999) applied neural networks to identify four pollen types that were relevant in paleopalynology, based on texture analysis. However, their method is not applicable to airborne pollen types. Boucher et al. (Bonton et al., 2001; Boucher et al., 2002) developed a semi-automatic recognition system for Cupressaceae, Olea, Poaceae and Urticaceae pollen types. They used a set of global shape and specific pollen features (e.g., cytoplasm, pore, reticulum), which were extracted from a sequence of 100 two-dimensional (2D) pollen images. Such a sequence of 2D images gives an effective way of representing three-dimensional (3D) pollen grains. However, the algorithms used for detecting pores are not applicable to all possible orientations of a pollen grain. Cernadas, Formella, & Rodríguez-Damiá (2004) described a method combining statistical gray-level and shape features to recognize the Urticaceae family. Since they did not use 3D or a sequence of 2D images as pollen representation, specific and structure oriented features were missing.

In this study, we performed a feasibility study on the automated recognition of pollen by using a conventional optical microscope to be used for daily counting. Therefore, we aimed to develop an automated feature extraction and classification process, which would operate on a sequence of 2D pollen images. Methods were developed to automatically extract shape and statistical gray-level and specific pore/colpus features by utilizing image processing and pattern recognition software tools. Similar approaches have been used by Boucher et al. (2002) and Cernadas et al. (2004) However, we focused in more detail on the pore/colpus structures of the pollen. Besides shape and statistical gray-level features, the structure of pores and colpi typically reflect the palynological knowledge that is used by biologists during pollen recognition (Weber, 1998). To investigate the feasibility of such a system, we focused on three purified pollen types: Dactylis glomerata (grass), Betula verrucosa (birch) and Artemisia vulgaris (mugwort), which are the most important allergenic pollen with overlapping flowering seasons in The Netherlands.

2 Materials and methods

Purified pollen of grass, birch and mugwort were obtained from HAL Allergy (Haarlem, The Netherlands). The pollen were dispersed in phosphate-buffered saline pH 7.4 on a microscope slide and stained with safranin (2 μg/100 ml of coloring medium). To mimic the aerobiological sampling procedure using a Burkard pollen sampler, we inserted a cellulose strip, covered with Vaseline, into the slide. Following this sampling procedure, we prepared two types of slides. One contained each pollen type separately (single-type slides); the other contained a mixture of the three pollen types (mixed-type slides).

Pollen images were acquired with a digital camera (C-3030 Olympus, Japan) connected to a microscope (BX 41 Olympus, amplification 40×, set-up with Köhler illumination). The resolution of the image was set at 1,536 × 2,048 pixels, in order to capture the detailed structures inside the pollen.

A sequence of five images with an interval of approximately 8 μm was taken at successive focal planes for each microscopic field, among which the central image revealed the most information (e.g., shape, texture and pore/colpus). Therefore, mainly this central image was used for segmentation and feature extraction. It was found automatically, by selecting the image with the highest variance.

The images were analyzed with a computer program developed in Matlab 7.0. (The Mathworks, USA) and toolboxes for image processing and pattern recognition, developed at the Delft University of Technology. The analysis consists of five steps, described in the next sections: (1) image preprocessing; (2) automatic pollen segmentation; (3) feature extraction; (4) classification and (5) validation.

2.1 Image preprocessing

Images were restored from shading effects, which were caused by the digital camera and microscope. This restoration was based on two test images taken without any objects under the microscope; one with the microscope lamp turned off and the other with the lamp turned on. Subsequently, the image was restored through linear transformation of both test images, according to Young, Gerbrands, and Van Vliet (1998a).

2.2 Automatic pollen segmentation

Since several pollen appear in one image, each pollen was isolated and copied into smaller images, and the RGB color images were converted into gray level (green band) images. The segmentation method was applied only to the central image of the sequence. Afterwards, the position of each pollen grain in the image was recorded. This position helped us to locate the same pollen in the four remaining images of the sequence.

Images were segmented by a thresholding method, with the combination of two automatic threshold selection criteria: the ‘triangle’ and the ‘isodata’ criteria (Young, Gerbrands, & Van Vliet, 1998b). The ‘triangle’ algorithm was applied first to achieve a coarse segmentation, and then the ‘isodata’ algorithm was applied to obtain a refined segmentation. In some cases the texture inside the pollen, which contained gray level comparable to the background, was excluded erroneously. Therefore a hole-filling technique was applied afterwards.

From a total of 135 and 80 images taken from single-type and mixed-type slides, respectively, we obtained 254 isolated pollen grain images, including 79 grass, 79 birch and 96 mugwort pollen.

2.3 Feature extraction

The features used in this study can be divided into three categories: shape features, statistical gray-level features and pore/colpus features, which are discussed in the next sections.

2.3.1 Shape features

Shape features represent the basic characteristics of a pollen grain. From the palynological literature (Weber, 1998) it is known that grass pollen grains (diameter varying between 30 μm and 40 μm) are usually larger than those of birch (diameter varying between 18 μm and 28 μm) and mugwort (diameter varying between 18 μm and 24 μm). Furthermore, grass pollen are more circular, while birch and mugwort pollen can have a triangular shape, depending on their orientation. Using this knowledge, we chose 19 shape features (Table 1) and extracted them from the binary central image (Costa and Cesar 2001a).

Table 1 Shape features
Full size table

2.3.2 Statistical gray-level features

Statistical gray-level features (Young, Gerbrands, & Van Vliet 1998c) (Table 2) were chosen, based on the fact that grass pollen grains contain much more texture than birch and mugwort pollen grains, and that birch pollen grains are more red than grass and mugwort pollen grains, if the same staining method is used. Features were extracted from the central image, for this image reveals the most significant pollen texture in comparison with that in the remaining images of the sequence. Since statistical gray-level features are sensitive to environmental brightness, we first normalized the gray-level images by setting the mean gray value of the background to a fixed value.

Table 2 Statistical gray-level features
Full size table

2.3.3 Pore and colpus features

The number and shape of pores or colpi are very informative during visual interpretation of the images in daily practice. Consequently, our goal was to automatically detect these different types of pores and colpi in the monoporate grass pollen, the triporate birch pollen and the tricolporate mugwort pollen. However, the appearance of pore and colpus structures varies with the orientation of the pollen grain in a slide, which complicates the automatic recognition of these structures. Therefore, we discriminated six different appearances of pores and colpi (Fig. 1): (1) pores appearing as circles within the pollen, as seen in grass and birch (Fig. 1a and d, respectively); (2) pores appearing as ellipses close to the boundary of grass and birch (Fig. 1b and e, respectively); (3) pores completely on the boundary, as seen in grass (Fig. 1c); (4) pores with an oncus, as seen in birch in polar view (Fig. 1f); (5) mugwort colpi in equatorial view (Fig. 1g) and (6) mugwort colpi in polar view (Fig. 1h). Based on these pore/colpus appearances, three pore/colpus detectors were designed to detect the following situations: (i) circular pore detector—pores appear as a circle inside the pollen boundary (Fig. 1a, d), (ii) birch pore detector—birch pores appearing on the pollen boundary (Fig. 1f); and (iii) mugwort colpi detector—mugwort colpi in polar view (Fig. 1h). These three detectors are described in more detail in the following three sections.

Fig. 1
figure 1

General appearances of pores/colpi of grass (a, b, c), birch (d, e, f) and mugwort pollen (g, h). Grass and birch pollen with pores appearing as a circle inside the pollen grain (a and d); pores appearing as an ellipse close to the edge (b and e); pores precisely on the pollen boundary (c and f); mugwort with the colpi from equatorial view (g); mugwort colpi in polar view (h)

Full size image

As these detectors may repeatedly detect the same pore/colpus at the different depths of focus in an image sequence, the detected pore/colpus positions from all five depths of focus were clustered to obtain the ultimate pore/colpi positions, using a distance-based clustering algorithm (Webb, 2002a). In this manner the number of pores/colpi according to a certain detector was calculated and used as a separate feature (Table 3). In order to obtain the total number of pores/colpi of a pollen grain, as a human observer uses to distinguish pollen types, we combined the results of the separate detectors into a single estimate of this total number.

Table 3 Pore/colpus features
Full size table
2.3.3.1 Circular pore detector

Since pores inside the boundary of grass and birch pollen have a circular appearance, these pores can be detected with the Hough transformation, based on gradient direction with a range of radii (Costa & Cesar, 2001b; Hough, 1962; Illingworth & Kittler, 1988). The gradient direction indicates the likely direction where the circle center is expected to be, which considerably reduces computation time, and the range of radii makes the detector adaptable to the pore size. Subsequently, the position where most lines coincide indicates the circle center; in our case it refers to the pore position. Figure 2 shows the original pollen image and the corresponding Hough-transformed image. The position of the pore is determined by applying a threshold to the Hough-transformed image.

Fig. 2
figure 2

Principle of circular pore detector. a Example of an original image for a grass pollen grain. b The corresponding Hough-transformed image

Full size image

We optimized the circular pore detector by varying its thresholds and evaluating the resulting success rate of the detection in a free-response receiver operating characteristics (FROC) curve (Egan, 1975). Three sets of thresholds were then selected with three criteria: (1) the highest true-negative rate; (2) the highest true-positive rate and (3) the best trade-off for both rates. Detectors with these three sets of thresholds were all included in the feature set (Table 3), and the threshold that met the best detection rate was chosen during the feature selection step.

Additionally, the maximum value in the Hough-transformed image was considered to be a separate feature.

2.3.3.2 Birch pore detector

Since birch pores, which appear at the boundary of the pollen, usually have a very specific oncus pattern and a regular shape, these pores can be detected with a template matching method (Pratt, 1978), as illustrated in Fig. 3. First, the gray-level image is transformed into polar coordinates with a 120° overlap (Fig. 3a). This overlap avoids insufficient matching of pores on the edge of the polar image. A typical pore image is chosen as a template that is matched along the polar image (Fig. 3a). At each position of the image, the similarity to the template is indicated by their cross-correlation value. Figure 3b shows the original polar image and the corresponding cross-correlation image after template matching. Finally, the position of the pores is determined by applying a threshold to the cross-correlation image.

Fig. 3
figure 3

a Principle of birch pore detector. b An example of an original birch pollen polar image and the image after template matching, showing cross-correlation values of each position

Full size image

The threshold was optimized in the same way as the circular pore detector described in the previous section. As a separate feature for classification, the maximum value in the cross-correlation image was calculated (Table 3).

2.3.3.3 Mugwort colpi detector

In polar view, mugwort colpi are always uniformly distributed along the pollen boundary. Therefore, we developed an algorithm that examines the regularity of the gray levels along the cell wall. First, a polar image of the pollen wall is created (Fig. 4a, b). Subsequently, the cell wall is aligned (Fig. 4c) by adjusting the vertical position of each column in the image, according to the cross-correlation with consecutive columns. Finally, the row located on the cell wall is identified by selecting the row with minimum mean gray value (Fig. 4d). The intensity profile (i.e., the gray levels along the cell wall) then shows a pattern typical for mugwort, with high gray-level intensities for cell wall parts interrupted by very low intensity parts where colpi are present (Fig. 4e). The frequency spectrum of this intensity profile then reveals high peaks at frequencies that are very specific for mugwort in comparison with those for grass and birch. Finally, the position of the colpi can be detected by applying a gray intensity threshold to the intensity profile.

Fig. 4
figure 4

Principle of mugwort colpi detector. a An example of an original mugwort pollen image. b The corresponding polar image. c Column-aligned polar image. d Row of interest (ROI), shown as the white line. e Gray-level intensity profile of the ROI

Full size image

Since it was easy to determine a threshold that met both high true-positive rate and high true-negative rate, the results from such threshold were directly included in the feature set. Furthermore, the height of the peak in the frequency spectrum is calculated as a separate feature in the feature set (Table 3).

2.4 Classification

In total, 47 features were included in a feature set. We applied pattern-recognition algorithms to this feature set in order to obtain classifiers, so as to recognize the three types of pollen grains. First, we reduced the number of features (by so-called forward selection), by selecting the smallest number of features that gave the most discriminative power (Kittler, 1975). Afterwards, we applied the following two types of classifiers:

  1. (i)

    Classifiers based on Gaussian density estimation: quadratic normal classifier (QNC); linear normal classifier (LNC); nearest mean classifier (NMC) (McLachlan, 1992; Webb, 2002b)

  2. (ii)

    K nearest neighbor classifier (KNNC) (Dasarathy, 1991).

2.5 Validation

In obtaining a gold standard for the validation of the automatic classification, the mixed-type slides were classified by one of the authors (L.dW.), a biologist experienced in identifying pollen from microscope images. All images from the single-type slides were already classified, as they contained by definition only the prescribed type of pollen.

All images were processed by the image analysis program, described above, and the resulting (reduced) set of features for each detected pollen grain was then used to classify each individual grain. The classification error was calculated through a cross-validation process, in which the classification is carried out five times with random division into training and test set. Subsequently, the average error of these five classification results is calculated (Lachenbruch & Mickey, 1968; Stone, 1974). We determined the best type of classifier by evaluating the performance of each type of classifier.

To investigate the importance of the pore related features, we performed the same experiment to classify the pollen images, using two feature sets; with and without pore features.

3 Results

The performances of the different classifiers on the feature set, as obtained from cross-validation, are shown in Fig. 5. The best type of classifier was the LNC method (Fig. 5a), which resulted in the smallest mean error.

Fig. 5
figure 5

Classification results. a Performance of different classifiers on the feature set. The x-axis shows the number of selected discriminative features and the y-axis shows the average classification error from the fivefold cross-validation. b Performance of LNC on the feature set. The x-axis shows the number of selected discriminative features and the y-axis shows the average error and its standard deviation from the fivefold cross-validation

Full size image

The lowest mean error rate and smallest standard deviation of the errors were obtained when 12 features were used by the LNC classifier (Fig. 5b), which means that 12 among the 47 features were most discriminative for recognition (Table 4). Based on the above results, the LNC classifier with 12 features was chosen as the optimal model for pollen recognition.

Table 4 Twelve most discriminative features for pollen recognition
Full size table

On average, this model reached an overall success rate of 97.2%. In all cases grass and birch pollen were recognized correctly, whereas mugwort was recognized correctly in 93.8% of the cases (Table 5). The six misclassified mugwort pollen are shown in Fig. 6.

Table 5 Confusion matrix in recognizing grass, birch and mugwort pollen
Full size table
Fig. 6
figure 6

The six misclassified mugwort pollen grains

Full size image

The comparison of classification results between the feature sets with and without pore features showed that a decrease in the classification error from 5.1% to 2.8% was obtained by the inclusion of pore features.

4 Discussion

In this pilot study, we investigated the feasibility of automated recognition of grass, birch and mugwort pollen grains. Pollen grains were collected, sampled and recorded in sequences of 2D images at different focus levels. Subsequently, features were extracted from those sequences of pollen images. In addition to some conventional feature sets (binary shape features and statistical gray-level features), we focused on the specific pore/colpus structure of the pollen grains and designed software to automatically detect these structures. After feature extraction, 12 features were chosen that gave the best discriminative performance and the best classifier (LNC), with a correct recognition score of 97.2%. All grass and birch pollen were recognized correctly, while 6.2% of the mugwort pollen were misclassified as birch pollen.

The segmentation of the pollen grains is a process that provides standardized pollen images for feature measurements. In this feasibility study, a simple thresholding method was sufficient, since we used purified pollen images without dirt or fungal spores. In practice, however, where samples are collected from a pollen sampler, this approach will not be adequate. Therefore, in future research, one may either improve the segmentation method using color images, or maintain this simple thresholding method but add an extra class of non-pollen objects to be identified.

Detection of the pore/colpus structure was the key issue in this study. We distinguished six appearances of a pore (Fig. 1), for which we designed three detectors for circular pore appearances in grass and birch (Fig. 1a, d), for birch pores in polar view (Fig. 1f) and for mugwort colpi in polar view (Fig. 1h). An elliptical appearance of a pore, located close to the boundary (Fig. 1b, e), could not, therefore, be detected by our system. A more complex Hough transformation would then be needed. Also, the detection of pores that appear on the grass pollen boundary (Fig. 1c) or the mugwort colpus in equatorial view (Fig. 1g) needs further study.

Feature reduction is an important process in reducing the complexity of the classification system, and it provides the most powerful features for discrimination. The 12 best features were dominated by shape and pore/colpus features rather than statistical gray-level features.

Features derived from the intermediate results of pore detection (by the birch pore detector and the mugwort colpi detector) were powerful ones. The advantage of this type of feature is that they are exclusively designed for certain types of pollen. Since different pollen may show similar pore appearances, i.e., the circular pore as seen in some birch and grass pollen (Fig. 1a, d), pore detection is not always discriminative. Therefore, it is relevant to determine the number of pores/colpi. However, in our case, the total number of pores/colpi is not considered to be discriminative, because birch and mugwort both have three pores/colpi. This result suggests the use of the number of pore/colpus of a specific type, rather than the total number of pores/colpi.

Shape features, especially the series of features with respect to the distance between the center and the pollen boundary, provide a strong discriminative power. This is because grass pollen grains are usually larger than those of birch and mugwort. Moreover, the special shape of mugwort pollen produces much fluctuation in these distances; thus, this special shape also improved the discriminative power of circularity and of the number of pixels after skeletonization.

Statistical gray-level features showed limited discriminative power in our study. Only the feature “mean of gradient intensity” is listed among the top 12 features. The reason may lie in inefficient normalization of gray-level images or the limited number of pollen types. The introduction of a better normalization method or more features (e.g., Zhang & Wang, 2004) may lead to better performance of statistical gray-level features.

Six mugwort pollen grains were misclassified as birch pollen grains, five of which showed a colpus in equatorial view (Fig. 6a–e). This colpus appearance was not detected in our study. Two possible methods may help to improve the classification performance. The first method is to detect this colpus or to find more related features with respect to this pollen appearance; the second method is to introduce rejection in the classification and classify these pollen with extra help from the biologist. The reason why the mugwort pollen grain in polar view (Fig. 6f) was misclassified may be the relatively low light intensity in two of the colpi. Therefore, the mugwort detector requires better gray-level normalization.

The inclusion of pore features in the classification method reduced the error rate by almost a factor of 2. However, since the success rate was already very high (97.2%), this does not provide conclusive evidence that pore features are of vital importance for automatic pollen recognition. The reason for this lies in the fact that only a limited number of types of pollen were included in this pilot study. Therefore, we expect that pore features will become more important for automatic recognition when more types of pollen are included in a study [for instance, plantain (Plantago) pollen, which has a shape and texture similar to grass pollen but differs in the number of pores].

The high success rate of our recognition method provides a platform for further research. Priority will be given to (i) the segmentation of pollen images from slides obtained from daily practice using a pollen sampler and (ii) the expansion in the number of pollen types among which the allergenic pollen types can be discriminated. Finally, an automatic image acquisition system is a key element in building such a system in practice. These efforts may finally lead to a less labor-intensive and prompt recording system for allergenic pollen grains in the air.