Introduction

Breast cancer is the deadliest breast pathology occurring in women worldwide. Statistics [1] say that 1 out of 8 Indian women are afflicted by it during their life and it is predicted that 76,000 Indian women may die of the disease every year by 2020. The most disheartening aspect of this disease is the failure of early screening. It is a societal challenge and delivering an affordable solution to its prevention will benefit the society directly. Early and accurate diagnosis is critical to reduce disease burden, decrease morbidity and mortality rates and give early treatment, thus avoiding disfiguring surgery, and on the whole, improving the survival index up to 95% [2]. The chances of complete cure for breast cancer is 85%, if the tumor size detected is less than 10 mm [3]. Studies [4, 5] have shown that different radiologists interpret mammograms differently, at different times of inspection. The error in interpretation does not support the use of mammography as the sole screening tool. Hence, researchers propose to add thermal imaging as a modality along with breast examination and mammography to screen breasts; as mammography gives false negatives in early stages.

Human beings can maintain a constant body temperature, therefore, changes of more than a few degrees, is a clear mark of an abnormality [6]. The Stefan–Boltzmann Law says that objects with a temperature above absolute zero emit radiation, which is proportional to the fourth power of its absolute temperature. Thus, the infrared radiation emitted by the body can be converted into temperature values and mapped into an image [7]. This military technique was first applied to mammary glands clinically in 1957 to pick changes in blood perfusion [8]. In 1982, Food & Drugs Administration (FDA) approved thermal imaging to be used in conjunction with mammography for breast cancer detection [9]. Bryan F. Jones [10] in 1998, presented a study reexamining the use of infrared thermography as a signal of physiological dysfunctions. Therefore, it is vital to address new research on early breast cancer detection to screen the population routinely and should be cost effective with advantages over the gold standard of mammography.

This literature review aims to investigate the frontiers of the current research in the field of breast cancer screening using thermal imaging. This paper presents an extensive review of the literature related to breast screening using thermography and attempts to highlight the reliability of this technique. The paper is organized as follows: Section Methods for diagnosis of breast cancer and their limitations is about the several methods conventionally used for the diagnosis of breast cancer. Followed by Section Breast thermography : Biological rationale for thermal changes which explains the biological rationale behind thermography and its usage in decision making for breast cancer. Section Related works describes the methodologies used in past works for detecting breast cancer using thermal imaging. Section Discussion discusses the critical interpretations drawn from the past work. The article ends with the most pressing directions for future research and summarizes the conclusions of the survey in Section Conclusion.

Methods for diagnosis of breast cancer and their limitations

To improve the survival chances of patients and accurate diagnosis of breast cancer, many technologies have evolved. Following are the methods used to diagnose abnormalities of the breast:

Mammography

Mammography is an imaging modality that captures craniocaudal view (CC) and mediolateral oblique view (MLO) for each breast. It can detect cancer development at the 12th month when the tumor is beyond 1 cm diameter and X-rays can pass through it, which in many cases have already metastasized. Mammograms can be uncomfortable and traumatic because they often involve high compression of the breast tissue between two plates to get better contrast between non-cancerous and cancerous cells [3]. It also shows calcification. Mammography has suboptimal sensitivity and specificity values in women with dense breasts and in women with fibrocystic breasts. The mammogram sensitivity is 85% for women above 60 years, and is 64% for women under 50 years [4]. It also ionizes radiation in the patient and requires high quality, expensive equipment. Pregnant women are not recommended to undergo mammography. False positives are detected in 70% of the cases which triggers emotional stress in patients and results in painful biopsies. 10-30% of lesions are missed during mammography due to contrast variation, noisy images, tissue background that has the same characteristics as that of supporting breast tissues and edges. In addition, 42 pound pressure during the process may rupture the encapsulation around a tumor and release malignant cells into the bloodstream [5].

Ultrasound

Ultrasound or sonography uses sound waves to detect tumors. Since no radiation is involved, it is the best suggested method for screening pregnant women and younger women with dense breasts. It can distinguish between cysts and solid masses. However, it can neither detect tumors at deeper locations nor identify microcalcifications. The efficiency of ultrasound depends on the expertise of the physician interpreting the image [3]. It is relatively inexpensive and convenient to the patient. It is mostly used in adjunct with mammography to locate the exact area of suspicion [2].

Breast MRI

Breast MRI is a non invasive imaging technique that uses powerful magnetic field of strength of 1.5T and provides the highest quality breast images [6]. It can show the smallest of lesions that are not visible in the earlier two methods as seen in Fig. 1. But, it is a costly exam. It frequently reports false-positive diagnoses thus limiting its positive predictive value (PPV). It is not capable of detecting microcalcifications. This test is not recommended for pregnant women since a powerful magnet and a contrast agent is used, which produces allergic side effects [3].

Fig. 1
figure 1

Various methods for diagnosis of breast cancer

Full size image

Thermography

Unlike mammography, breast thermography is completely non-invasive, passive, private, contactless and there is no radiation hazard for patients. It is painless being simply a contactless image of the patient’s breasts. Thermograms are clinically interpreted based on color. Low heat levels are indicated in blue (healthy), whereas, spots in red, orange, or yellow indicate abnormality. Thermography is a functional test and a preventive process, which can be used by women from the age of 23, unlike mammography which is recommended for 40+ women only. An abnormal thermogram can be a significant biological risk marker in younger women under 40 for existence or continuous development of breast tumor. Figure 2 shows sample breast thermogram of a patient captured in various positions.

Fig. 2
figure 2

Sample thermograms : Positions (a) Front, (b) Right Lateral 45∘, (c) Right Lateral 90∘, (d) Left Lateral 45∘, (e) and Left Lateral 90∘ [7]

Full size image

In a study conducted on breast cancer patients using mammography and thermography, Gamagami [11] reported that 15% of cancers went unnoticed by mammography, but were caught by thermography. When thermal imaging is combined with mammography, the reported 85% sensitivity rate of mammography increases to 95%. The average size of tumors that went unnoticed by mammography and thermal imaging was 1.66 cm and 1.28 cm, respectively [12]. Thus, it is clear that infrared imaging can detect small tumors leading to early diagnosis [13]. Authors [24] studied the performance of thermography in combination with mammography and found the sensitivity to be 89% in women less than 50 years, which suggests that the dual imaging process can be one way to exploit the potential of thermography.

Breast thermography : Biological rationale for thermal changes

Human body emits part of its own thermal energy in the form of infrared radiation. Skin temperature pattern demonstrates consistent bilateral symmetry; any deviation from the normal is a good evidence of clinical abnormality, indicating metabolic and circulation changes. This is the underlying philosophy of using thermal imaging as a screening tool for breast cancer. Cancer tissues metabolize faster than other tissues, the heat produced in this process is conveyed to the skin surface, which hints at a possible malignancy or thermally active, fast growing tumor [15]. As well, due to excessive regional vasodilation caused by nitric oxide (NO) originating from cancerous lesion, there is an increased supply of nutrients and oxygen to the tumour [16]. The tumour location on the breast will have a higher temperature compared to the surrounding normal tissues. It is evident from literature that this temperature difference helps to identify an underlying tumour. The thermal camera captures the temperature variation over the surface of the skin in the form of an image. Temperature variations that characterize tissue metabolism are circadian [17] (“about 24 hours”) in periodicity. Rhythms associated with cancer cells are non-circadian and are indicative of malignancy. Women with asymmetric thermograms have 10 times higher risk of developing breast cancer than those with symmetric thermograms [18]. Thermography is capable of detecting cancer even before the patient is symptomatic. It is a fast, economic and risk-free process and provides dynamic information of tumors, if screened at regular intervals.

Rotational thermography has also been explored to screen tumors in lower regions of the breast that go undetected due to breast sag. A novel patented set up known as the Mammary Rotational Infrared Thermogram (MAMRIT) unit [14] is used for acquiring rotational breast thermograms. This setup comprises an IR camera mounted on a rotational arm enclosed in a chamber as shown by a schematic in Fig. 3. Each breast is suspended in the aperture of setup and is imaged every 30∘ under pre-cool and post-cool temperature settings. However, to carry on further research, there is no available database of images captured by a rotational thermography machine. Important procedural steps and care required to acquire thermal images are mentioned section-wise below.

Fig. 3
figure 3

Schematic representation of a patient table with aperture to suspend breast

Full size image

Pre-Thermographic Care & Patient Acclimation

As per the standard protocol presented in 1996 at the IEEE EMBS Conference [18], before the examination, the patient is cooled so that the hotspots arising from abnormalities are highlighted. The breasts must not come in contact with any surface that can alter its temperature. Patients are advised to avoid application of any ointment or perfumes, avoid above average intake of tea or coffee, avoid exposure or treatment of breasts. Patients are refrained from wearing tight clothes and avoid any physical exertion to ensure valid results. On arrival, the patient’s information is collected such as age, weight, family history, symptoms seen or previous breast treatments done. Strict protocol needs to be followed during thermography to get consistent results.

Procedure & Patient Safety

The examination cubicle is temperature and humidity controlled, with black homogeneous background. Incandescent lighting should not be used as it produces radiation, during examination. The disrobed patient has to sit for 10-20 minutes at rest, so that thermal equilibrium of the parts to be examined is achieved [2]. After achieving the thermal steady state, the patient is seated before the infrared camera with hands raised above the head and affected breast is to be examined from three views, namely, medial, frontal and lateral. Thermography is a real time system that captures a sophisticated heat-map of the breast and stores it for further analysis of hotspots. The first thermogram captures the baseline reading, and shall be repeated every 3 months to screen any abnormal development.

Thermograms and their Capturing Views

In breast thermography, the frontal view gives the best information of abnormality or asymmetry between the left and right breast. Thermograms of the underside of the breast (supine view) [19] should also be viewed, so that any tumor in the lower portion should not miss. Each pixel in the thermogram is a representation of a specific temperature of the parts. As the temperature of a body increases, it gives off more intense infrared radiation. Redness indicates increased circulation. Normal thermogram shows good symmetry, patterns represent baseline that don’t alter over time. Asymmetry in temperature suggests a suspicious region.

Database

Most of the existing literature uses the thermal images from the online, free, publicly available Database of Mastology Research (DMR) [20, 21] that contains breast images of 287 patients (240 - healthy, 47 - sick) of different sizes and asymmetric breasts. This dataset has images of resolution 640x480 acquired using FLIR SC-620 IR camera, with the patient’s age ranging from 29 to 85 years.

Thermographic Equipment Development

Thermal imaging system consists of an infrared camera and a display unit to exhibit the thermal image. Resolution and thermal sensitivity are the crucial parameters of an infrared camera. Older generation cameras were bulky and needed to be cooled to liquid nitrogen temperature to reduce artifacts in thermal images. Uncooled infrared technology coupled with the recent development of micro-bolometer sensors has revolutionized thermal imaging. There is a drastic improvement in sensitivity from 0.3K to 0.02K, over five decades, which has now facilitated the capture of detailed, reliable, high contrast thermograms that can detect small, localized hot spots. Modern trending IR cameras have high resolution and are compact, portable, easy to implement, almost maintenance free, can be used in any orientation [10]. Major companies that manufacture infrared cameras for use in medicine are Meditherm, FLIR, InfraTec., Medicore Co Pvt Ltd., etc.

Commercially Available Systems

An Indian startup “Non-Invasive Risk Assessment with Machine Intelligence” (NIRAMAI) [22] co-founded by Nidhi Mathur and Geeta Manjunath uses artificial intelligence to make detection of breast cancer possible at a low cost and is offering an alternative to mammography. Also, the NoTouch BreastScan (NTBS) machine [23] developed in the USA by UE LifeSciences uses two infrared cameras, each pointing at one breast. It uses Artificial Neural Networks (ANN) to identify the features and recognise the patterns for tumors. However, in a study conducted with 180 women who had biopsy proven breast cancer, NTBS gave an accuracy of 0.5 and thus failed in critical decision making. Thermal camera employed in Sentinel BreastScan [118] has thermal sensitivity of 80mK. A software analyses the breast scans of the patient and provides a report of the breast cancer risk. Reported sensitivity and specificity are 98% and 50% respectively. iTBra [119] created by Cyrcadia Health consists of wearable breast patches that measure breast temperature at 8 locations for each breast. This system relies on the biological circadian cycles [17]. Data is collected through thermodynamic sensors and sent to the lab for analysis where artificial intelligence algorithms estimate the breast cancer risk. A flattened circadian profile is indicative of the presence of cancer. Overall accuracy reported is 87% and it is claimed to be 30% more accurate than mammography for women with dense breasts. However, we did not find enough testing evidences of this commercial product.

Related works

For clarity, the research done in this domain is divided into 4 major areas in the paper. The survey focuses on reviewing the methods, techniques applied and results obtained for detecting breast cancer using thermography images. The literature on thermal image preprocessing and segmentation is discussed in Section Processing and Segmentation of Suspicious Region, feature extraction in Section Boundary Detection, Asymmetry Analysis, Feature Extraction, classification using artificial neural networks (ANN) in Section Using Artificial Neural Networks and Fuzzy Logic and computer modelling of breast in Section Numerical Simulations and Models of Breast. Section Patents discusses the patents filed related to thermal breast imaging.

Processing and Segmentation of Suspicious Region

Denoising & Preprocessing:

Noise and artifacts are introduced in the image during camera handling, image compression, image acquisition, storage, which degrades the image quality [25]. Thermal images lack texture, have low contrast [26] and clear edges are absent in them, which make the abnormality detection using asymmetry bit difficult. Qualities affected by image acquisition are considered in [25]. However, qualities affected by camera handling, lighting conditions are not accommodated in this work. Hence, further extension is expected by considering these additional features. In case of dynamic thermography, motion induced artifacts have to be removed to compare the different images properly. Kafieh and Rabbani [27] modelled the noise variance as a function of the image intensity and used wavelets for denoising breast infrared images. DCT [28], Lacunarity and Hurst coefficients were used in [29] to identify breast pathology from the images. Block matching and 3D filtering techniques (BM3D) [30] were adopted for removing noise from breast thermograms and features extracted from denoised images were used to identify the abnormality. There was a distinct difference in the feature values of denoised and raw images. The edge profile was preserved and the signal to noise ratio (SNR) was enhanced by 32% in denoised image.

Segmentation:

Using hotspot feature extraction gives high accuracy in classifying malignant and benign tumors as compared to features from the whole breast. Thus, segmentation of tumor i.e. region of interest (ROI) in a thermal image was considered essential for accurate classification of malignancy [31]. Segmentation depends on the distance from which the breast thermogram was captured, the image height, the breast size, image background, presence of noise etc. To separate the right and left breast, different types of segmentation such as region-based, threshold-based and edge-based have been explored in the literature. Canny edge detector [32, 33], Sobel edge detector, Hough transform [32, 34], C fuzzy techniques, projection profile approach [35] have been used to extract the body boundary and the lower parabolic boundaries of breasts from a thermogram.

Experiments conducted by researchers [31, 36] to detect breast pathologies in thermograms based on segmentation of ROI were limited to only classifying the cancer images, highlighting the suspected malignant tumor locations but not predicting tumor location and its nature. Dynamic frontal breast thermograms were preprocessed [37] and level set segmentation was performed to delineate ROI. Qualitative results were presented. Researchers claimed accuracy (86%), sensitivity (92%) and specificity (73%). The main disadvantage of the work was the suspicious region in the abnormal breast was not highlighted. The approach in [38, 39] used k-means clustering for segmenting the hotspots, however, this method did not yield the equivalent results with different values of k. Golestani et al. [40] compared segmentation methods, namely, k-means, fuzzy c-means [33] and level set for fibrocystic and inflammatory cancer cases to extract the hottest regions. It was established that the level set method outperformed the others by extracting almost the exact shape of the tumor.

de Oliveira et al. [41] used automatic segmentation on 180 thermal images; but their work was limited as they used one side lateral breast images to conduct their work. Sedong et al. [42] used 250 thermal images and the experimental results were analyzed and compared using Shannon entropy and logistic regression. Pramanik et al. [43] used wavelet based thermogram analysis on 306 images (123 unhealthy and 183 healthy). Ali et al. [44] used segmentation method on 63 thermal images, but method reliability check was not performed. To separate the breast area for feature extraction, authors [45, 46] used edge and contour filters. Eddie et al. [47] extracted the useful regions of thermal images by segmentation and classified them into normal or abnormal ones. They also assessed the menstrual cyclic variation of temperature with time to help detect breast cancer.

Boundary Detection, Asymmetry Analysis, Feature Extraction

Interpreting thermograms requires meticulous training and their misinterpretation can lead to high false positive rates. In order to get rid of subjectivity in the manual interpretation, computers can be used with machine learning algorithms to help in accurate classification of breast pathologies from thermal images. Dayakshini Sathish et al. in [3] reviewed various medical image models and included various steps to develop CAD tools for several imaging techniques. In accordance with their work, thermography CAD depends on asymmetry analysis and gives better accuracy.

Josephine J. et. al [48] considered 25 typical and 25 abnormal breast thermograms for an automated classification. Feature values were extracted and fed to backpropagation neural network and error correction learning rules were applied. However, the dataset used was very limited and images were not acquired in real time. The proposed methodology in [49] used thermal matrices of 454 images from DMR of women from different age groups and in multiple stages of breast cancer.

In 1997, Lipari and Head [18] observed asymmetry in frontal breast thermograms by computing the temperature differences between the contralateral breasts quadrant wise (shown in Fig. 4). They concluded that manual interpretation of thermograms compromises the accuracy of diagnosis. Therefore, in a future publication (1998), Head et al. [50] compared the temperature profile of contralateral breasts using automated approach which ultimately led to improved accuracy. Later in 2000, Head and Lipari et al. [51] isolated the breasts manually and computed infrared index for each abnormality. An analysis of family history, previous hormone therapy and previous biopsy of breast were assessed and results indicated that there was no correlation of these factors with infrared results. Their results pointed out that 35% of patients had abnormal IR image more than a year before they got diagnosed in mammography, which indicated that thermography is a risk marker.

Fig. 4
figure 4

Quadrant wise segmentation of breasts: Upper-Outer (UOQ), Upper-Inner (UIQ), Lower-Outer (LOQ), Lower-Inner (LIQ)

Full size image

A comparative study of the thermal signatures of healthy breasts with malignant tumors was performed in [52, 53]. Qi, Hairong et al. [54] used Canny edge detector and Hough Transform, to derive the edges and recognize the four feature curves, respectively, on images provided by Elliott Mastology Centre. Bezier splines were used to view brightness distribution (256 brightness levels). The results obtained from 3D histogram of thermal distribution clearly showed the difference between a cancerous image and a non-cancerous image. To attain more accurate classification, supervised pattern classification on a larger database was suggested. In [55, 121], asymmetry was identified using k-means clustering and k-nearest neighborhood [56] based on feature extraction. Statistical features [55, 57, 58, 121] like mean, variance, skewness, kurtosis, correlation, entropy, and joint entropy were used to quantify the distribution of different intensities in each breast. The feature values of both breasts were quite consistent for normal breast thermograms, whereas, there is a remarkable difference in the feature values of both breasts for abnormal breast thermograms. Results obtained in [39, 55, 57, 58, 121] points out that the high order statistics like variance, skewness, and kurtosis signify asymmetry. However, they used a small dataset of 24 thermograms to evaluate the performance of their method, hence their results cannot be generalised.

The study in [59] measured temperature gradients (Δ T) from thermograms of 1008 female patients and from the asymmetry analysis, the subjects were classified into three groups: normal (Δ T ≤ 2.5), abnormal (Δ T > 2.5, < 3) and potentially having breast cancer (Δ T ≥ 3). X. Tang et al. [60] proposed the LTI (Localized Temperature Increase) for breast cancer detection. LTI was used in breast thermograms and its amplitude was measured. High sensitivity of 93.6% and high Negative Predictive Value (NPV) of 91.2% was achieved. Patients with higher LTI amplitudes were considered as having higher possibilities. Optimal LTI amplitude threshold calculated was 1∘C for breast cancer detection. The limitation was poor specificity due to FPR (False Positive Rate) of 55.7%.

Hossein Ghayoumi Zadeh et al. [61] presented a fully automated approach based on fuzzy active contour [62] to detect edges and contour area in the thermograms. It segmented the cancerous areas from its borders. The manual and automated methods were evaluated using Hausdorff and mean distance. The obtained sensitivity and accuracy was 85% and 91.98% respectively, but only for limited dataset. M. Etehad Tavakol et al. [63] compared sixty contralateral breast thermal images for asymmetric thermal distribution. For a pair of images, they used mutual information and nonparametric windows to plot the joint histogram. The mutual information value was close to one for similar thermal images of left and right breasts.

Authors in [64, 65] used an approach based on Fractal Dimension (FD) for classification of thermograms. Features of the image were extracted using box counting algorithm [64]. From the past experiments it was evident that boundaries of benign tumors are well defined and malignant tumor boundaries are irregular. By exploiting this fact, the fractal dimensions of features extracted from breast thermography images had different values for normal breast and inflammatory breast tissues. However, they tested the approach on a very small dataset of 6 thermograms, and these results may not hold true for larger datasets. They also determined the stage of breast cancer based on the T component of the TNM (Tumor-Node-Metastasis) system as suggested by Surgical Clinics Journal [66]. The classification of tumor size using Fuzzy C-Means in 3 and 4 clusters with the use of 64x64 pixel box size in the box counting process was more consistently similar than with the use of 32x32 pixel box size [64].

Scales et al. [67] acquired twenty-one 128x128 8-bit grayscale thermal images and preprocessed them by applying the Canny edge operator. To get the breast shape as an ellipse, the morphological operators were applied to the edge detected image. Edge detection gave inaccurate results for the flat lower part of the breast. It was concluded that more robust and intelligent edge detection methods need to be devised to identify regions of interest. In [68], Kapoor et al. created an operating system independent GUI in MATLAB for automatic segmentation of hotspot and asymmetric analysis of breast thermograms in real time to detect abnormality.

A prospective, double blinded pilot study was conducted in [69] using FLIR ThermaCam E45 for 54 patients aged between 18-70 years. Minimum, average and maximum temperature values were measured for each tumor location and its normal contralateral side. To measure asymmetry, 0.5∘C was selected as a cut-off temperature difference. Fibroadenomas differed from malignancies significantly in terms of mean temperature, whereas there was no significant statistical difference between granulomatous mastitis and invasive ductal carcinoma. Cysts differed from malignant lesions in terms of maximum temperature. The sensitivity and specificity of thermal imaging was found to be 95.24% and 72.73% respectively. Using thermography, it was possible to differentiate fibroadenoma and cysts from invasive ductal carcinoma. Fibroadenomas are mainly seen in younger women, where other screening techniques fail to give good results. Thus, thermal imaging seems a promising tool to detect it at the earliest.

A retrospective analysis [70] of the clinical records of 100 normal patients, 100 living cancer patients, and 126 deceased cancer patients discovered that only 28% of the noncancer patients had an abnormal thermogram, compared to 65% of living cancer patients and 88% of deceased cancer patients. According to the p-test, it was seen that the clinical size of tumor as per TNM classification system was significantly larger (p = 0.006) in patients with abnormal thermograms. In a case study [39] conducted in 2018, one normal case of 50 years old who had undergone screening mammography and other case of 70 years old with abnormal cancer mass proven were included. Segmented tumor region from mammogram and asymmetrical skin temperature profile from thermogram were compared for the diagnosis of breast cancer. The maximum temperature difference between both the breasts of a cancer patient’s thermogram was found to be 1.1∘C, whereas it was less than 0.2∘C in normal cases. Since it was a case study, the results cannot be generalised unless randomly tested on a large database.

Using Artificial Neural Networks and Fuzzy Logic

The extracted features from the segmented breasts are used as inputs to classification algorithms to classify the breast thermograms. Since human eyes cannot perceive the colors in thermograms perfectly, hence to classify suspicious regions in thermograms, into benign and malignant and maximise accuracy, artificial intelligence [71] has been the latest area of interest among many researchers. Determining a suitable combination of features to compose a feature vector is essential for obtaining high precision as too many features pose the problem of overfitting the model. Using only a few features reduces the complexity of models, requires less time and is easier to understand.

Qi and Head [72] proposed automatic segmentation unlike the studies carried out in [18, 50, 51]. Abnormalities in left and right breast regions were obtained from the thermal histograms. Unsupervised learning was used to cluster abnormal pixels together. For segmentation and classification of breast thermal regions, K-Means, Fuzzy C-Means (FCM) [73], Gaussian Mixture Model — Expectations Maximization (GMM-EM) [74] and Bayesian networks [75] were employed and their results were compared. The results indicated that FCM segmentation gave the best accuracy at indicating the disease. Mahmoudzadeh et al. [76] suggested Extended Hidden Markov model (EHMM) to randomly sample breast thermal images and re-estimate EHMM parameters to optimize segmentation for easy interpretation of thermal patterns by clinicians. EHMM segmentation results were compared to K-means, Fuzzy C-means, Lloyd–Max, self organizing map (SOM) and standard HMM algorithms. Results indicated that EHMM is able to extract hotspots in the least time. However, they did not analyse the extracted abnormal regions further.

Nader [77] developed an automatic breast cancer detection software using MATLAB to analyze thermal breast images of 206 patients. 12 statistical features and 20 texture features were extracted and fed to a neural network classifier to differentiate normal and suspected cancer breasts. Success rate of 96.12% was noted using this software. In [78] the authors used a deep neural network for tumor segmentation and binary classification of breast cancer. Extreme learning machines [79] were employed to classify 1052 thermographic images as cyst, malignant and benign from the features extracted from the geometry and texture of images. Training accuracy obtained was 73.38% with a sensitivity of 78%, a specificity of 88% and a Kappa index of 0.6007. In a similar study [80], feature matrix was fed to the neural network to classify thermal images into TH1-TH5 category (Marseille system) based on vascular patterns. Gerald Schaefer et al. [81] extracted statistical features like moments, cross co-occurrence matrix, mutual information and Fourier analysis from breast thermograms and fed them into a fuzzy rule-based classifier for analysis. The classification accuracy achieved was about 80%, which is limited.

Sheeja V. Francis et al. [82] extracted statistical and texture features from breast thermograms using a curvelet transform based method. These features were fed into SVM for automatic classification and the accuracy obtained was 90.91%. Francis et al. [19] extracted first and second order statistics and texture features. Eventually, they reduced the number of features from 17 to 4 using Principal Component Analysis (PCA). An SVM was used to classify the breasts as normal and cancerous. The classifier achieved a sensitivity of 83.3%. In [83], a combination of ANN and genetic algorithm was employed to gather the features from 200 thermal images of patients aged between 18-35. The results revealed that thermal pattern and kurtosis were the most useful features for breast cancer classification. The proposed model had 50% sensitivity, 75% specificity and 70% accuracy. In their subsequent study [84], they used a fuzzy model for increasing the accuracy.

Several special types of ANNs like Complementary Learning Fuzzy Neural Network (CLFNN) [85] were used in [86] to classify IR breast images. To increase reliability, CLFNN takes statistical features such as family history and temperature difference between contralateral breasts into account. Tan et al. [87] used five different classifiers, namely, Feed Forward (FFNN), Probabilistic (PNN) and Fuzzy Neural Networks (FNN), Gaussian Mixture Model (GMM) and Support Vector Machines (SVM) for 90 breast thermal images. Out of all, FFNN, GMM and SVM classifiers had better performance as indicated in Table 1. Lagrange Constraint Neural Network (LCNN) [88] used multispectral thermal images to provide a better diagnosis. Wavelet transformation [89] with ANN was used for multidimensional features of the IR image. Dimensionality reduction was performed to downsize the number of features computed and the resulting images were classified as healthy or cancerous using a multilayer perceptron (MLP) neural network. Borchartt et al. [90] used free LibSVM classifier to classify breast pathology from the extracted features. In [91], a back propagation ANN was used to predict clinical outcomes. The ANN output was matched with the actual clinical diagnosis. The ANN predicted the outcome of 18 out of 19 images correctly, however the dataset used was too small and the statistical parameters for analysis were few. Both benign and malignant tumors increase vascularity of the breast area and cause temperature changes, thus limiting the specificity of thermography.

Table 1 Performance of Various Classifiers Used For Breast Cancer Detection
Full size table

U. Rajendran Acharya [92] extracted texture features from 50 breast thermal images using co-occurrence and run length matrices and fed them to an SVM classifier for classification of breasts into normal and malignant. The classifier gave an accuracy of 88.10%. On similar lines, authors in [93] fused multiple adaptive thresholding techniques to identify the hotspots in breast thermal images. They extracted textural features using algorithm from [92] and chose the thresholds of temperature such that it maximizes the linear combination of sensitivity and specificity. The obtained sensitivity and specificity using the approach in [93] is 90% and 94.3% as compared to 85.71% and 90.48% from [92]. A study in [94] correlated the locations of hotspots in thermograms with the tumor locations marked in mammograms and histology images of 60 patients. Thirteen sets of features were evaluated by using SVM with radial basis function kernel [94] to classify breast thermograms as healthy, benign and malignant. To avoid an erroneous conclusion, the suspicious areas were categorized as in the upper half or lower half of any breast. The limitation of this study was the small experimental dataset. Table 1. reports the number of thermograms used, the sensitivity and specificity of the different classifiers mentioned in the paper.

A Convolutional Neural Networks (CNNs) based methodology, shown in Fig. 5, was harnessed in [95, 96] for faster and more reliable diagnosis of breast cancer using thermal images. CNNs require a large amount of data for training. Data augmentation [96] was done to tackle expensive and imbalanced medical data set problems. The performance and reliability of CNN was enhanced by implementing data augmentation and a hyper parameters optimisation algorithm based on tree parzen estimator, thus minimizing bias and overfitting that might occur during the training process. Their work concluded that smaller and simpler CNN’s performed better than state-of-the-art ones like ResNet, SeResNet, VGG16, InceptionResNetV2 and Xception. They are more viable than past systems based on statistical and texture features [55,56,57,58, 69, 77, 81, 82] for diagnosis. The highlight of the work was the two possible DMR database split approaches used in the training framework and their comparative performance, which hasn’t been discussed in any previous study. Metrics like accuracy (92%), precision (94%), sensitivity (91%) and F1-score (92%) with ROC-AUC summarised the CAD’s performance. Thus, the use of data augmentation ensured the need of minimum number of patients to train the system. In a very recent work (2020), a software was developed for detecting early breast cancer automatically by analyzing 140 thermal breast images. Breast characteristic features based on bio-data, image analysis, and image statistics were extracted and fed to a CNN optimized by Bayes algorithm [120] to classify the breast images as normal or suspected. An accuracy of 98.95% was obtained.

Fig. 5
figure 5

Database split methodology using CNN in [95]

Full size image

Adaboost classifier [33] was used to select and integrate the best features that are invariant to scaling and translation from breast thermal images to classify them into malignant, benign and normal classes. They apportioned the data into training and test sets, with an 80-20 split. Using a 10-fold cross validation, the accuracy computed was 95%. To solve the problem of imbalanced class distribution i.e. less number of malignant cases than healthy, authors in [97] suggested the use of a combination of different classifiers instead of one, so that the common intersection space where all classifiers make incorrect decisions will be small and the strengths of individual classifiers can be used to full potential for achieving high accuracy. This collective decision making approach of a combination of classifiers helped achieve high sensitivity without sacrificing specificity and outperformed other classification algorithms.

Numerical Simulations and Models of Breast

Numerical simulations have facilitated researchers to understand the thermal interactions occurring within the female breast and study the effect of factors such as metabolic activity, tumor position and depth of tumors. Mostly simulations of the female breast in literature have been modelled using the Pennes equation [98] as it provides accurate temperature predictions. In Eq. 1, q, c and k are the density, specific heat and thermal conductivity, respectively. The subscripts t, b and a refer to tissue, blood and arteries, respectively, x is the blood perfusion rate per unit tissue volume and qm is the metabolic activity within the tissue.

$$ \rho_{t}c_{t}\left( \frac{\partial T_{t}}{\partial t}\right) = \nabla.(k_{t}\nabla T_{t}) + \omega_{b}c_{b}(T_{a} - T_{t}) + q_{m} $$
(1)

Authors in [99] used Lyapunov exponent modeling to detect abnormal lesions from breast thermal images and differentiated the patterns as malignant and benign. A 2D model [100] of the female breast with and without tumor was numerically solved using finite element analysis. The breast model had varying layer thickness to imitate the actual shape of the female breast. By changing the tumor location, its size and blood flow rate, temperature profiles were plotted for a normal and malignant breast. Breast surface was scanned using numerical simulation by identifying 7 parameters and analysis of variance (ANOVA) was performed using a 2n design (n = 7) in [101]. For optimizing the parameters, Taguchi method was used. It ensured that the signal from the tumour is maximized, and noise from other sources is minimized. In a further study conducted in 2001, the surface temperature and tissue temperature profiles of a normal breast and malignant breast were analysed using a 3D model [102]. Tumours of different sizes were placed at various locations to study the effect of depth of tumor on temperature distribution of the breast. Any shift in tumor position was also recorded in results. It was a common observation in [100,101,102] that shallow tumors generated more heat as compared to deep seated tumors. Breast with malignant tumor generated higher surface temperature than a normal one. At the tumor location, the tissue temperature profile appears distorted.

A book chapter by Amri et al. [103] focused on using the Cartesian breast model. They implemented one such model in [104] (shown in Fig. 6a) to study the surface temperature patterns of the female breast. It consisted of fat and gland layers with a tumor embedded. They varied the tumor depth and tumor diameters and observed that the maximum temperature rise was 1.2∘C for tumors 5 mm deep. Their results lacked relevance as the domain of their model was a rectangular prism, which does not correctly represent the morphology of the female breast. Chanmugam et al. [105] presented a 2D breast model in hemispherical domain with six non-concentric layers using COMSOL FEM software, as shown in Fig. 6(b). The effect of parameters discussed in [100] was investigated and vital features were picked out to estimate the tumor size and location. Hatwar and Herman [106] developed the model presented in [105] and performed transient simulations to estimate size, location and blood perfusion of the embedded tumor. Starting from 0, they measured the temperature at 12 different positions in increments of 2 in the axisymmetric model. The accuracy for tumors of depth 2 cm was within 1%, but the accuracy of estimation declined considerably for deep seated tumors.

Fig. 6
figure 6

Simplified models (a) Cartesian model [104] and (b) Hemispherical breast model with seven tissue layers [105]

Full size image

Patents

For a long time, researchers are working on the problem of early detection of breast cancer - it being the deadliest amongst all diseases in Asia. Some patents filed by a few organizations and researchers are briefed below.

  • Method of measurement of the skin temperature overlying tissues of breast in contrast with other tissues was employed in [107] to detect abnormal patches painlessly. Theoretical limitation to this invention is the tumour must be large and shallow for the temperature of the skin to rise. However, deep-seated cancers that have grown serious over time do register in the image. This invention can be counted as a valuable new diagnostic tool for the physician to use in conjunction with other tools for cancer diagnosis and diagnosis of other vascular diseases.

  • A patent of a system [108] consisting of thermographic imaging device configured to acquire thermographic images of a female torso to detect the risk of breast cancer was filed by Niramai Health Analytix Pvt. Ltd. in 2016. A database was configured to store the thermographic images with a qualitative analysis module configured to automatically perform a qualitative analysis of the thermographic images and generate a qualitative score based on the qualitative analysis. A quantitative analysis module was also configured to automatically perform a quantitative analysis of the thermographic images and generate a quantitative score based on the quantitative analysis. A scoring module was configured to correlate the qualitative and quantitative scores with a score indicating the risk of breast cancer. A decision fusion rule was also utilized to determine whether tissue within a specific region is cancerous, non-cancerous, or is suspicious of being cancerous.

  • Determination of malignant tissue based on its contour from breast thermal image of a patient undergoing cancer screening was discussed in a patent filed by Niramai Health Analytix Pvt. Ltd. in 2017 [109]. Pixels with a higher and lower temperature value were displayed in different colors. A patch of pixels with raised temperature as compared to the temperature of surrounding tissue pixels was identified and classified as malignant or non-malignant based on the boundary contour irregularity calculated for that patch of pixels. The measure of irregularity was calculated by, i) selecting a plurality of points along a best-fit ellipse around the pixel patch, ii) calculating a distance between the points along the boundary contour and the points along a best-fit ellipse. A shape of the boundary contour of the pixel patch was determined to be irregular or regular depending on the distance threshold, meaning that the tissue associated with the pixel patch is malignant or non-malignant. However, no work was done to estimate the size of the tumor.

  • A system for thermal breast cancer detection by capturing multiple thermal images in various angles was invented in [110]. A segment-by-segment analysis of the thermal images employing inverse heat transfer analysis was performed to calculate the probability of positive breast cancer identification using a threshold. The threshold for a patient was varied depending on at least one variable from: age, weight, history of alcohol consumption, history of tobacco use, the existence of a mutation in BRCA 1 or BRCA2 gene. Applying inverse thermal conduction algorithms to the thermal images made it easier to detect deeply buried tumors. Multiple thermal infrared images taken over time increased the probability of correct early detection of smaller and deeper breast tumors.

  • A system and method for classifying the hormone receptor status of malignant tumorous tissue identified in a thermal breast image was disclosed in [111]. The malignant thermal image was analyzed to define a boundary contour of the breast. Then, the breast regions were segmented into regions of elevated temperature. Probability mass function was determined based on pixel temperatures within the first segmented region. A second probability mass function was determined based on pixel temperatures within a second region. A distance measure (any of: the Jensen Shannon distance, the Kullback Liebler distance, mutual information, or any standard defined distance function) between the two functions was calculated and provided to a classifier (any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naive Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof) trained to classify the malignant tissue as hormone receptor positive and negative otherwise, based on the distance measure.

  • Kakileti & Siva Teja [112] from Niramai Health Analytix Pvt. Ltd. invented a system and method for isolating blood vessels in a breast thermal image. Candidate vessel pixels which satisfy one or more of intensity-based or temperature-based or shaped-based criterion were identified and a constraint of local maximallity was imposed on them to eliminate spurious non-vessel pixels. Those candidate pixels were then marked with a different color so that they could be visually differentiated. The vessel structures were provided to a classifier system (any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naive Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof) which classified the tissue in the thermal image as malignant or non-malignant based on tortuosity of the vessel structures.

  • Niramai Health Analytix Pvt. Ltd. [113] filed another patent in 2018 disclosing a method for breast cancer screening which classified hot spots seen in a thermal image of both breasts as possibly malignant based on a measure of symmetry. A hot spot comprising a patch of pixels with an elevated temperature with respect to surrounding tissue was identified using either, a mean temperature of pixels in the patch or a median temperature of pixels in the patch or a highest temperature of pixels in the patch. The hot-spots in each breast were binarized by setting a value of 1 for pixels in the hot spots and a value of 0 for other pixels. A measure of symmetry was calculated comprising a ratio of an area of a smaller hot spot to an area of a larger hot spot from the thermogram. The measure of symmetry was provided to a classifier system (any of: Support Vector Machine, a neural network, a Bayesian network, a Logistic regression, Naive Bayes, Randomized Forests, Decision Trees and Boosted Decision Trees, K-nearest neighbor, a Restricted Boltzmann Machine, and a hybrid system comprising any combination hereof) trained to classify an unclassified hot spot as malignant or non-malignant.

Discussion

Main findings

Degradation in a thermal image is caused by noise, blurring, fading and artifacts. Reported reviews in Section Processing and Segmentation of Suspicious Region revealed that there is not much significant work done on the quality improvement of thermal medical images. In the studied literature, wavelet-based denoising [27, 28], block matching and 3D filtering technique (BM3D) [30] were the methods adopted for noise removal. For ROI extraction, Canny edge detector, Sobel edge detector, Hough transform, C fuzzy techniques and clustering methods are used. Few researchers like [41] used automatic segmentation but used only one side lateral breast images. The work was studied on a limited basis in clinical trials which pointed out the crucial need for development of a large scale breast thermogram database.

Localisation of the tumor from thermal image is critical for diagnosis of breast cancer. However, in most of the prior work, textural and statistical features extracted from the entire thermal images were used to obtain results. There are limited techniques that segment the tumor region and use its features for classification. Existing segmentation algorithms mentioned in Section Boundary Detection, Asymmetry Analysis, Feature Extraction used clustering [33, 38, 39, 55], thresholding [93], edge detection, Hough transform [32] and active contour techniques [61]. Thresholding techniques [93] are strongly governed by thresholds for segmentation. Rather, using fixed thresholds for segmentation of thermal images leads to reduced performance due to varying range of body temperatures. Head et al. [18, 50] automated the process of asymmetry analysis to compare contralateral breasts in terms of their temperature profiles. In [39, 55, 58] the use of high order statistics over low-order statistics was suggested to detect an asymmetry.

Several ANN based classifiers are discussed in Section Using Artificial Neural Networks and Fuzzy Logic and their performance parameters like accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) are tabulated. The training set used in literature is comparatively smaller, hence, results cannot be validated and generalized. To improve the utility of thermography in clinical practises, it is important to maximise the performance of classifiers for easy interpretation and diagnosis.

Pennes bioheat equation, discussed in Section Numerical Simulations and Models of Breast, is used to study the thermal interactions within the breast. Although there is limited literature on modeling of breasts, numerical modelling in conjunction with high-resolution thermograms is proved to be a good diagnostic tool. 2D [100, 104,105,106] and 3D [102] models of female breast with and without carcinoma have been implemented and the effect of size and location of the tumour on surface temperature distribution was studied.

Despite the success of mammography, due to its flaws, there is a need for promoting additional research in thermography to refine it. Researchers in [114,115,116] proposed to add thermal imaging as a modality along with breast examination and mammography to screen breasts; as mammography gives false negatives in early stages. A study [115] conducted with 132 patients pointed out that the accuracy of mammography (76.9%) was more than thermography (69.7%). Yao et al. [116] found that mammography has a better accuracy in detecting tumors larger than 2 cm in diameter, whereas thermography is a cut above ultrasound and mammography in case of tumors less than 2 cm in diameter. Wishart et al. [24] found thermography to be effective in women under 50 years, whereas in women over 70 years reduced vascularity in breasts causes thermography to fail. Arora et al. [117] reported that thermography had up to 97% sensitivity and 11% specificity for breast cancer detection. However, the study was not blinded as patients with suspicious lesions detected by mammography and recommended for biopsy were selected. Thus, the results of effectiveness of thermography are controversial and we found a lack of consensus on the outcome measure of this tool with the clinicians. It is important to conduct a blind, randomised trial with adequate cases of every stage of cancer, obese women, large breast sizes, etc. to check the feasibility of thermal imaging as a screening test. The findings confirm that this technique is valuable in the early prognosis and can add curative potential to breast cancer in the next 5 years. It can be an initial screening test in poor countries or rural areas having no access to the costly mammography machine, provided some limitations are overcome.

Limitations

Preliminary survey shows that the analysis of thermal images in the studies conducted has been subjective which ultimately led to inconsistency in the diagnosis of breast cancer. Since, the results obtained were not well controlled or blinded, researchers, clinicians lost interest in the technology and are sceptical about the usage. The biggest challenge facing researchers in this field today is that the literature lacks a prospective, double blinded study, where experts reading the thermograms are blinded to biopsy site. Challenge faced and not addressed conclusively yet, is, if thermography can replace mammography successfully. It is a well identified, yet, an under appreciated societal challenge. Earlier primitive thermal technology was discredited due to high false positive and high false negative rates, limitations on equipment, image resolution, low accuracy for deep tumors, its subjective nature and inability to localize a lesion. On the contrary, with the incorporation of the latest technology, cheaper and fast thermal cameras are available now. To get consistent results, more investigation is needed on interpreting thermal images accomplishing standard protocols. The role of thermography can truly be evaluated only through clinical practice and trial in mass breast cancer screening. Being a preventive test, to improve healthcare, asymptomatic patients should be screened using thermal imaging for breast abnormalities. It is difficult to distinguish between breast cancer and inflammation zones through thermography. Cold tumors are difficult to be caught on a thermogram. A high rate of false positives has been identified as one of the flaws of the diagnostic tool. There is no literature that guides the clinician to an approach for making a decision for further treatment after an abnormal thermogram is detected. There also prevails a lack of work in distinguishing different stages of breast cancer starting from early stage using thermography. Thermography is still not endorsed for detecting clinically occult breast cancer due to the above highlighted technical challenges and limitations.

Conclusion

Breast cancer is a critical global health problem. Since it is not preventable, to curb the mortality rate of this malady, aggressive work needs to be done on its early detection. Our review depicts the current state of research and the progress that has been made in context of the techniques used for classifying the breast as healthy or malignant using thermography images along with its diagnostic virtues and limitations over the past two decades. A significant finding of the literature work conducted is that there is no large scale database of thermographic images. In this paper, the challenges and future potential research opportunities are highlighted. From the investigation of highly diversified studies, it is evident that the sensitivity and specificity of mammography is less than optimal for patients with dense breasts. There is no one tool that can truly predict the presence of cancer, except biopsy. Thermography used in combination with mammography had sensitivity of 95%. Thus, combining different screening tools can help clinicians achieve the best diagnosis.

Thermography is a patient friendly tool that stands out in diagnosing early signs of breast cancer thus facilitating earliest treatment. Human eye cannot easily differentiate between the thermal patterns, hence, there is a dire need of a smart, automated system that can accurately understand the patterns, predict malignancy without erroneous results and produce acceptably high true positive rate. However, a high false positive rate has been identified as one of the flaws of thermography. Research is needed to make the thermogram interpretation objective and develop an accurate model for classification of breast patterns. The possibilities with such systems are very broad and open new opportunities for research. Thermography can become the appropriate first-line service for any woman who simply wants to do a safe, precautionary screening. At present, it can be concluded that it cannot substitute mammography in clinical practice due to insufficient evidence. If thermography is coupled with an agent administered to the patient, it can help to detect tumor accurately. From the study of published results using infrared imaging, it was seen that the sensitivity was between 0.78 and 0.94. The specificity was between 0.73 and 0.92. The reported results are heterogeneous due to different devices and algorithms used in different works to classify thermal images.

The limitations of thermography like false-positive and false-negative need to be reduced further by using appropriate combination of the feature extraction techniques, types of the segmentation and classification algorithms on a comprehensive database. Computer simulations can facilitate automated interpretation of thermograms to help clinicians perform breast cancer routine check-up using this technology as a screening tool. Deep learning based method has proved to be a promising tool to detect breast cancer with upto 98.95% accuracy rate. The evidence and gaps deriving from the past experiments suggest that high quality scientific research in this field is the need of the hour to revolutionize the quality of healthcare and link multidisciplinary advancements. Considering the small cost and non-invasive nature of this technology, pursuing further studies would be worthwhile to chase this endeavour.