Keywords

1 Introduction

Extraction of information from an image is useful to acquire its internal properties. The analysis of extracted information helps to initiate the modifications in image attributes as per the requirement of specific application. The preliminary stage of information extraction is to observe the illumination directed from surface. The illumination parameters of an image can be categorized as reflectance, shadow effect, movement of object, camera miss focuses etc. The reflection is a major disturbance affecting the quality of information retrieval. Any reflection induced in an image can be considered of two types: 1. Specular reflection and 2. Diffuse reflection. When a ray of light strikes the surface of an illumination source, part of ray gets immediately reflected from interface between surface and air owing to their different refractive indices. If this reflection is from single direction. This phenomenon is called as specular reflections (SR) and if it is reflected from multiple directions, it is called as diffuse reflections [1]. Most of the images get affected by specular reflections (SR).

Medical Image analysis is one of the growing fields of research. The most prominent factor affecting the accuracy of this analysis is the quality of an input image. The SR produced during capturing of medical images is major obstacle in this process. Thus, replacement of these SR pixels keeping the underlying image characteristics intact, is a must before any further medical image analysis. Removal of these reflections from the captured images is a prevalent challenge for the medical image processing. SR is a bright spot on an image which contains maximum part as a white intensity. It is prominent in medical images like cervix image, endoscopic image, cardiographic image, angiogram etc. Figure 1 shows the example of cervix image affected by SRs. This SR region can be easily seen with open eyes (shown as black box). The cropped SR region is also shown in the adjacent image.

Fig. 1
figure 1

Example of SR affected cervix image with cropped SR region

Full size image

Handful contributions are made in the literature to address the issue of SR removal. These contributions majorly focus on accurate detection of SR pixels and its proper replacement in original image.

In 2011, A. Artusi et al. presented a detailed review on specularity removal technologies suggested by various principles. Authors compared various methods based on dichromatic reflection model (DRM). Theoretical description was given for each category of DRM. They analyzed the significance and demerits of each model to brief the respective application domains. They also compared few state-of-art systems related to SR detection. Brief discussion about single and multiple image analysis is also included in the same review. Authors focused on hardware aspect of SR detection and removal methodologies. The paper also added visual comparison of state-of-art methods [2].

In 2017, H. Khan et al. reported a survey on highlight detection in color and spectral images. Authors covered the methods related to single and multiple images and analyzed various state-of-art methods with respect to strengths and assumptions. This survey provides a wide look on highlight detection and removal issue in color and spectral images [3].

The paper reports a review on the working principle of SR detection methods. The SR detection techniques can be broadly classified as use of dichromatic reflection model (DRM) principle, use of Kernel as a filter, using thresholding technique and considering SR as a classification or segmentation problem. The paper is arranged in same fashion. Section 2 to 5 concludes with table of comparison between various contributions of respective category and enlists objective (OQA) and subjective i.e. visual quality assessment (SQA) measures used by researchers. All these major categories are reviewed in further sections separately and common analysis is done in Sect. 6. The possible solution to remove the research gaps is explained in Sect. 7 based on detail review. The paper is concluded in Sect. 8.

2 SR Detection Using Principle of Dichromatic Reflection Model (DRM)

Any radiation is composed of reflections from interface and surface body. The DRM works on the principle of same characteristic of radiation or reflection [2]. The major objective is to separate this combination i.e. to get diffuse and specular reflections separated. Many researchers focused on same principle and suggested various methods of SR detection.

K. Yoon et al. proposed speedy detection of SR pixels in 2006. They calculated value of specularity invariant pixels and their ratio to separate diffuse components. However, this method is applicable in texture images and has limited accuracy due to approximation in normalization process [4].

In 2011, O. Meslouhi et al. suggested another method based on DRM. The method works on chromatic characteristics of highlighted areas. The experimentation is carried out on uniform and textured colposcopic images. The method is independent of any arbitrary threshold value [5].

The technique proposed by Ali et al. separates specular component from diffused component using factorization of non-negative matrix. The reflection function of skin is scaled with respect to changes in density and distribution functions of skin. This assumption made in scaling gave promising results in SR detection or separation [6].

J. Suo et al. applied DRM concept as signal separation problem for SR detection and removal [7]. However, the method suggested by them is incapable of differentiating the pixels with same hue and different saturation. It is also unable to incorporate smooth change in color.

Tao et al. [8] proposed a new measure called line consistency for depth estimation of SR region. Colors from multiple light sources are easily estimated by this method. However, it fails to separate saturated specularity.

In 2017, Asiedu et al. [9] designed a new model to generate specular free image. The authors worked on colposcopy images and used the principle of finite element analysis for accurate SR detection. The model is tested on hardware platform to check the compatibility and feasibility on real time processing.

Wang et al. [10] presented the method using minimization of global energy in polarization imaging. Color constraints are consider through polarization information and color distortion is reduced at maximum scale to achieve significant accuracy. The method is robust with respect to consistency of accuracy in detection of camera noise.

Lamprinou et al. [11] suggested a method recently in 2018 to remove glare present in laproscopic images. Authors promoted the use of forward and backward derivatives of non-stationary signals to act as a filter. The backward derivative is proposed to remove abrupt changes in images. Different cost functions are designed to optimization and system stability. The major cause of concern is database dependency of the method and computational complexity.

Table 1 summarizes state-of-art methods of SR detection based on DRM principle.

Table 1 State of the art SR detection techniques based on principle of DRM
Full size table

3 SR Detection Using Kernel as a Filter

Another approach suggested by few researchers is to use a kernel or mask as a filter to detect SR pixels present in image. This kernel is referred as structuring element (SE).

Lange [12] extracted green component of an image to get high glare to background ratio and treated it as a feature image. Morphological top hat filters are used to detect SR pixels from extracted feature image. Watershed segmentation is used to find the complete boundary of glare region. The method suffers from drawback of accurate selection of structuring element in morphological processing.

Sun and Sang [13] also suggested using kernel as a filter on image. Xue et al. [14] followed similar approach to use structuring element in top hat transform on intensity image (I) of cervigrams.

Yao et al. [15] worked on gastroscopic images to automatically extract SR pixels. Authors promoted the use of IS histogram (Intensity-Saturation histograms) from HSI color plane. Templates of various shapes like triangular, rectangular and anti-sector are applied on input images to perform template matching and detect appropriate SR region. The work achieved 71.9% true positive rate with area under curve (AUC) 0.869. Ulcer can also get detected accurately without any confusion with intensity values. Though authors proposed triangular template as a best for SR detection, the shape of the template may get change for each image modality.

Tchoulack et al. [16] proposed a FPGA based architecture on SR removal from endoscopic images. Histogram decomposition is done on each channel of RGB to compare the intensity levels. Specular mask is designed based on histogram intensity observations and smoothing filter is applied to detect accurate SR pixels. The performance of hardware is tested and reported as faster system. However, size of specular mask is a vital factor in maintaining the accuracy of SR detection.

Das et al. [17] suggested another way of using kernel for SR detection. Author worked on colposcopy images collected from hospitals. Simplest procedure of logical ANDing between white pixels of RGB color planes is proposed as SR pixels. Dilation followed by region filling is applied to get final replacement of SR pixels. K-means clustering is used to get accurate region of interest (ROI) from colposcopic images. Though the method is simple and easy to implement, its generalized behavior on various database is still a problem due to dependency on size and shape of the kernel.

In 2018, Kudwa et al. [18] proposed a unique way of focusing on SR pixels. Authors combined a feature image from saturation component of HSI plane, green component of RGB plane and lightness channel of CIE plane. The standard deviation filter is applied on feature image and dilation is performed to detect SR pixels from it.

Table 2 gives brief in sequence of above explained methods related to use of kernel for SR detection. This category suffers from the drawback of dependency of size and shape of kernel with respect to input database.

Table 2 SR detection state-of-art methods based on use of kernel as a filter
Full size table

4 SR Detection Using Image Thresholding

SR pixels always have bright intensities. Thus, some researchers proposed the use of various constants as a threshold on input image to get SR pixels. Most of the thresholds are applied on Intensity (I) and Saturation (S) images of HSI plane. Table 3 summarizes the SR detection techniques based on thresholding.

Table 3 Thresholding based SR detection state-of-art methods
Full size table

Zimmerman et al. [19] first introduced the selection of constant as a threshold on image. Authors suggested working in HSI color plane and proposed a multiplying factor of 0.4 and 0.6 to be multiplied with maximum intensity of intensity (I) and saturation (S) images respectively. Pixels less than threshold in S image and greater than threshold in I image are extracted. The magnitude of gradient image is considered as SR region. The constants suggested by Zimmerman et al. do not suitable for any database.

Shen et al. [20] promoted the threshold as specular degree of an image. Using this threshold, authors separated SR and diffuse pixels without any segmentation algorithm.

Alsaleh et al. [21] proposed three combinations of multiplication factors i.e. 0.5 & 0.17, 0.7 & 0.07 and 0.8 & 0.19 for I and S image respectively. The experimentation was carried out on various types of database. However, these constants need to be selected differently for each image modality.

Guo et al. [22] suggested performing binarization of intensity image using arbitrary selection of threshold. The accuracy of the said method is limited due to non-consideration of saturation parameter of an image. Authors worked on endoscopic images to get SR suppression using contouring.

A work on images having characters is carried out by Jiao et al. [23]. They promoted the use of two images captured from different angle to extract and merge specular free regions of these images. An algorithm based on image stitching is proposed in the work to avoid the need of special hardware for generating SR free image.

Digiovanni et al. [24] followed similar approach to apply thresholding on intensity image and experimented on colposcopy images. The technique proposed by them suffers from the trade-off of ease of implementation and less accuracy of SR detection.

The arbitrary selection of constants in the state-of-art methods of SR detection affects the accuracy.

5 SR Detection Using Image Segmentation or Classification of SR Pixels

Another approach suggested by few researchers is to consider SR detection as a classification problem and identify image pixels as SR and non-SR pixels. The use of segmentation on same line of approach is also promoted in the literature. Classification and segmentation based techniques work on input image pixel labeling and to separate the pixel based on its features.

In 1996, Hokland et al. [25] proposed the use of markov random field (MRF) to estimate the spread function and noise variance from input simulated images. The SR pixels get detected based on these models and they are restored separately using stochastic relaxation. Iterative de-convolution algorithm is used to remove the noisy components from image. The technique based on MRF proved to be better than wiener filter and it protects the occurrence of ringing effects.

Stoyanov and Yang [26] suggested temporal registration of non-rigid surface motion to recover chromatic information of SR saturated region. The registration is used to separate the specular and diffuse reflections. Authors applied this technique in robot assisted surgery. However, they have not considered the reflectance from noise and complex components.

The SR detection in facial images is promoted by Levine and Bhattacharyya [27]. Region based segmentation is used to separate specular and shadow regions with initial consideration of seed pixel. Support vector machine (SVM) classifier is used based on features of these expected separations. The SR detected pixels are replaced by mean of non-SR neighboring pixels and inpainting method called as ‘illumination compensation’. Authors justified the significance of the method for accurate detection of facial reflections.

Lee et al. reported a similar approach to treat the SR detection problem as a two label classification [28]. Authors suggested using perceptron neural network classifier to identify any pixel from an image as SR or non-SR pixel. Removal of these detected SR pixels is made by spatial smoothing filter. In advantage of working on single frame, this method is inefficient to detect SR pixels in gum areas of tooth. Thus, overall accuracy of the technique limits the usage.

Toshiaki Tsuji proposed assessment of variance in luminance component of an image to detect the SR region [29]. This method is effective to detect SR generated from various illumination sources. It removes the SR in original image but produces new SR with less intensity somewhere in other region. Author nullified this drawback of his own method by introducing the use of combination of flickering information of two images [30]. This method is composed of principle of reflection model, use of threshold for luminance of variance and segmentation of SR pixels using flickering information.

Doerschner et al. worked on different aspects for SR detection [31]. Authors worked on 36 videos or movies and using the concept of image velocity using spatio-temporal filters. These filters are applied on sequence of frames. Histogram of each frame is computed to classify the specular or diffuse region. This method is also suitable for 3D images and also predicts subjective perception.

The use of chaotic clonal selection algorithm is proposed by Akbar et al. [32]. The algorithm is an iterative calculation of threshold based on segmentation. An approach based on Walsh-Hadamard kernel is applied on the segmented image to get SR removal. The patch based coherency sensitive hashing algorithm is suggested for this SR in-painting using kernel. The complete SR removal process is compared with state-of-art methods and proved the significance of the same. The segmentation algorithm is also found effective than PSO through experimentation undertook by the authors.

Shah et al. worked on specularity removal of images of roads [33]. Authors considered the limitations of state-of-art methods applied on illuminant invariant images. The effect of wrong assumptions which rarely observed in road images is explained by the authors. They proposed a segmentation method using graph-cut algorithm to detect the SR patches from any road image. In-painting is done by morphological image processing. The quantitative step up of the SR removal method is explained with different performance measures and running time.

The state-of-art methods explained in this section are summarized in Table 4. The necessity of the every time training is a major demerit of using this classification principle.

Table 4 State-of-art methods based on SR detection as a classifier or segmentation problem
Full size table

6 Performance Analysis of SR Detection Techniques and Research Gaps

Quantitative analysis of any image processing algorithm can be carried out by comparison with original images. These original images need to be captured with proper illumination. Most of the reported literature for automatic SR detection (reviewed in Sects. 2–5) considered medical images under experimentation. However, it is not practically possible to get ground truth images in medical imaging. Thus, researchers preferred to have subjective quality assessment of their respective proposed SR detection and removal techniques. Table 1, 2, 3 and 4 enlists various objective assessment measures promoted by researchers. Most of the reported work is validated by using expert grading and mathematical quality attributes.

The state-of-art methodologies for SR detection and removal are overviewed in Sects. 2–5. All methods are categorized in four groups based on working principles. Summary of this review is given in Table 5.

Table 5 Summary of SR detection state-of-art methods
Full size table

All the categories explained in above sections provides explanation for the state-of-art methods of SR detection. Each category has some pros and cons. However, complete removal of specular reflections is must before any further image analysis. Thus, from the detail study of these categories proposed by various researchers, we suggest a possible way out to design a complete automatic system independent of database and image modality for detection and replacement of SR pixels occurring in an image.

7 Use of Automatic Threshold Selection Technique Before State-of-Art SR Detection

As discussed in above sections, most of the researchers proposed the use of arbitrary constants as a threshold and various image processing algorithms. However, these suggested threshold values are outcome of experimentation undertook for specific image modality. To fill the research gap and design a complete automatic SR detection system, we suggest to apply an automatic threshold selection algorithm on input image before applying state-of-art SR detection techniques. Thus, threshold will vary automatically as per the input image modality and other characteristics and can be passed forward to SR detection algorithm.

7.1 Survey of State-of-Art Automatic Threshold Selection Techniques

The automatic thresholding techniques can be broadly classified in five groups according to the information content, viz. histogram based, clustering based, entropy based, object attribute based and statistical relation based. In histogram shape-based methods, the major intensity, decimation in intensity range and non-linear nature of the smoothed histogram get analyzed. The method works on input image histogram to calculate the optimum threshold. In clustering-based methods, input image grey level pixels are divided in two parts as background and foreground pixels. Entropy-based methods use regional entropy, cross entropy of the foreground and background regions, original and binary image, etc. Object attribute-based methods focus on similarity between the gray-level and its black and white version. The statistic relation based methods use higher-order moments and/or correlation between pixels for threshold selection [34].

The reported techniques of automatic thresholding, principally depends on the calculation of either bi-modal distribution or multi-modal thresholding. To get SR pixels, which are always bright, bi-modal distribution is adequate and histogram based approach may be preferred for the same. A review on automatic threshold selection techniques based on histograms is given in Table 6.

Table 6 Survey of automatic threshold selection techniques
Full size table

Donald Bailey compared adaptive thresholding techniques given in Table 6 for performance analysis. As shown in Fig. 2, author concluded that Kittler’s minimum error method is best for automatic selection of threshold [35].

Fig. 2
figure 2

Comparison of automatic thresholding techniques [35]

Full size image

Sezgin and Sankur [34] compared the performance of thresholding techniques using five quality measures viz. misclassification error (ME), edge mismatch (EMM), relative foreground area error (RAE), modified Hausdorff distance (MHD) and region non-uniformity (NU). They calculated average score of each technique and rank of individual quality measure and concluded with Kittler and Kapura as best adaptive thresholding techniques.

Above discussion of literature related to automatic threshold selection promotes the significance of Kittler method due to its minimum error consideration. Thus, it may be applied on previously discussed state-of-art SR detection techniques to increase in the accuracy of algorithms.

7.2 Qualitative Analysis of Applying Kittler Method on State-of-Art SR Detection Techniques

We have experimented on 10 non-medical images collected from internet. State-of-art SR detection techniques viz. Zimmerman et al. [19] and Alsaleh et al. [21] are chosen based on the popularity. Kittler method of automatic threshold selection [39] is applied on these two techniques to replace arbitrary constants suggested in [19] and [21]. Figure 3 shows the comparative analysis of SR detection using original method and combination with Kittler method.

Fig. 3
figure 3

Comparison of use of Kittler automatic thresholding technique on state-of-art SR detection methods

Full size image

The SR pixels can be seen visually in original image. It can be easily observed from Fig. 3 that there is reduction in detection of non-SR pixels. The appropriate method of SR detection will produce sizable increase in the accuracy.

7.3 Quantitative Analysis of Applying Kittler Method on State-of-Art SR Detection Techniques

Kudwa et al. [18] proposed using Jaccard Index (JC) to measure the quality of SR detection method. Accurate detection of SR pixels and less detection of non-SR pixels increases the value of JC. Table 7 compares the JC of SR detection using [19] and [21] with the addition of Kittler method in [39]. This JC is calculated for manually marked SR pixels and actual detected SR pixels.

Table 7 Quantitative analysis of Kittler automatic threshold selection techniques on state-of-art SR detection methods
Full size table

Use of Kittler method shows increase in the value of JC. It justifies that more SR and less non-SR pixels are detected with the use of automatic thresholding.

8 Conclusion

The state-of-art methods of SR detection in the input image are reported in this paper. The methods are categorized with respect to working principle and compared through various aspects of performance analysis. The research gaps from study of these categories are also reported. To avail completely error free image for appropriate analysis, a fully automatic and database independent system of SR removal is the need of the hour. This can be achieved with the help of automatic threshold calculation. A detail survey of such automatic calculation of threshold is elaborated in Sect. 7. From the overall review and the experimental analysis, it can be inffred that Kittler minimum error threshold selection method canbe applied on input image before use of state-of-art SR detection principles. The proposed solution increases the Jaccard Index (JC) which demonstrates accurate detection of SR pixels and less detection of non-SR pixels. Perfect SR free image can be used for automatic diagnosis through computer aided system stages like feature extraction, classification etc.