Keywords

1 Introduction

Thermoregulation in humans is affected by their metabolic activity and sweating. Related to the temperature distribution, the human body can be considered as a symmetrical system around the sagittal plane. This is the idea behind the use of temperature for diagnosis or triage of diseases [7]. As Hippocrates said, “if the temperature of an area of the body diverges of its symmetrical, then diseases must be chased in these” [1]. It is especially related to the tumor growth when angiogenesis plays a fundamental part of in the processes of proliferation, migration and cellular differentiation, and when neovascularization increases the temperature in the region near to a cancer.

Thermography, like ultrasound (US), is a functional examination. Both present no risks to the patient and do not use ionizing radiation, and have safety and cost benefits. The use of US imaging or sonography in medicine requires very good operator training: such a person, in most countries, must be the same doctor that elaborates and signs the examination report. On the other hand, to aid the diagnosis of breast cancer, the most widely used imaging examination is mammography. This is, in part, due to the fact that mammography (like thermography) does not require a medical doctor for acquiring the patient images: only a well-trained nurse or assistant who follows an established and protocol of training is necessary; then, diagnosis can be done later by the specialists. This possibility allows performing examinations in a more efficient way in large hospitals, big clinics and in a very populated and poor country for triage proposes. Additionally, combining the two main positive aspects of these most used and traditional examinations in medicine, some authors have pointed out that thermography can detect cancer earlier than US imaging and mammography, due to their relation with fluid perfusion, neo vases formation and in the beginning of a disease [12, 15].

Historically, the computer was used for the first time to help with medical diagnosis around the year 1960. At that time, the key idea was to achieve a diagnosis elaborated by the computer. Due to several factors such as low processing power of computers and the lack of powerful algorithms , for decision making, the idea did not progress. By the 1980s, the idea of using the computer for the diagnosis reemerged, supported by the increase in the processing power of computers, advances in image processing and improvements in artificial intelligence algorithms. This second wave was supported on the premise that the computer system gives a second opinion or a complementary opinion for diagnostics , but never pretending to replace the human specialist. The first thermal sensors became available for diagnosis in the 60s. The 1990s promoted advances in infrared (IR) sensors (IR digital cameras allow the implementation of more elaborated software , accurate enough to be used for medical diagnostics) . During the current millennium, a new generation of sensors has arrived, being more rapid, sensitive and allowing images with more resolutions.

Although, the computational power of computers has increased continuously since then, allowing use of more sophisticated machine learning algorithms , such as support vector machines (SVM) [33], fuzzy neural networks, and deep learning. And, even though, big data treatment s have grown in parallel, producing more techniques for storage, maintenance and retrieval, and allowing for use of not only an image per patient but a complete history of the patient’s life with many sorts of examinations patients can have, included in their files along with many other significant data sets related to heath. The idea nowadays is to use the computer as a tool to help the physician save time, and to promote more efficient and correct diagnosis without taking from the physicians and patients the role of protagonists (i.e., continuing in the trend of the end of last century). With this in mind, this chapter suggests image processing techniques to help in the medical diagnosis of breast diseases by using thermal imaging . In the literature an inconsistent point among the articles using infrared breast imaging is the existence of multiple protocols for data acquisition , that is. These aspects are discussed in the next section. After acquisition, the images and data are stored and submitted to a number of techniques which generate the result of the examination, which is the report that is to be sent to the patients and their physician. These techniques are explained in the following sections, which consider storage and retrieval, preprocessing, 3D reconstruction , extraction of features, classification and performance evaluation .

2 An Overview on the Image Acquisition Protocols

Image acquisition protocols present at least three aspects: the preparation of the patient; the room environmental conditions and the procedures performed on the patient during the examination.

The preparation of the patient is related to recommendations to be followed before the exam (first aspect). Although, almost each hospital or health center uses its own protocol, there are common patient recommendations. For instance, these are for patients do not do: smoke; drink caffeine or alcohol; practice physical exercises; and apply any cream or oil type in the region of the breast and armpit. A second aspect of recommendations is related to the environmental conditions of the examination room: Temperature, relative humidity and air circulation should be controlled during the examination and the same of all capture. This second aspect is, in some way, almost the same among the groups (excluding the room temperature ) using thermal examinations over the world, as well.

However, considering the procedures performed on the patient (third aspect) there are much more variations. This third aspect ranges from the induction of changes in the body temperature of the patient (Passive Static Thermography), or no temperature alteration (Active Dynamic Thermography) ; duration of cooling or heating of the breasts; patient’s position (lying, sitting or standing) during the examination; patient’s position relative to the camera (angles and distance); position of the arms (on the head or supported at the waist, the named akimbo position), and number of examinations used on the diagnostic decision.

Related to the induction of changes in the breast temperature, the acquisition protocol may be classified as static or dynamic (passive or active). This nomenclature is considered in relation to the presence of the forced heat transfer in the patient body and consequent increase in the transient terms of the associated equations that describes the phenomena of thermal distribution on healthy and unhealthy tissues of the breast. In static protocols the patient is on stable temperature state with the environment of the examination room over the duration of the exam. This type of acquisition is suitable to identify hot and cold areas and to measure the symmetry in the distribution of skin temperature: it is the most used type. On the other side, an acquisition protocol is named dynamic when the camera is used for monitoring the recovery of skin surface temperature after a thermal stress (for example, heating or cooling) or chemical stress (e.g., on the use of some drug to promote vasodilatation or vasoconstriction in the vascular system). In other words: the dynamic case study the temperature of the skin surface in transient part of the processes and, allows to analyze changes over time, this turns important the acquisition of more than one frame at same position in the examination in order to enable the use of the transient set of data. That is, to promote the use of the information related to the changes of the skin surface temperature plus the interior temperature radiation and convection improves the identification of the age of the blood vases of the breast. This is important because normal veins (the one with almost the same age then the patient) present vasoconstriction (i.e., have constriction behavior) on the cold. However, this behavior is not visible for the bad formed and incomplete vein produced by the fast malignant proliferation of cells induced by the liquids perfusion near a cancerous region.

A fourth aspect related to the acquisition protocols is the dates or number of examinations used to promote the current diagnosis of a patient. In this aspect it can be classified as simple, sequential, or monitoring. In a simple acquisition, a unique examination (capture in a specific date or a single image) of the patient is used, that is, only a moment of time of her life is considered. In sequential mode, a series of images are acquired sequentially at a time interval as in dynamic protocols for monitoring skin temperature variation do not in the same day, but after some interval of days, weeks, or months. That is, as the dynamic protocol more images are used, but in monitoring mode, images are captured over longer interval on the same type of examination, for instance every 6 or 12 months, in order to monitor suspicious region, for early detection of a disease , to verify the progress of a disease and treatment after an therapy [27, 29]. Of course, the fourth aspect can be considered as a repetition of the third. For this, the protocols which deserve more analysis and experimentations are the static and the dynamic (Passive or Active) acquisition protocol grouped in the third aspect commented before. In order to see what can be considered the best one for the interested community on promote the union of effort for better use of IR images on diagnosis, data from these two types of acquisitions, obtained by our group, are available in a database of breast thermal images, named Database for Mastology Research with Infrared Image—DMR-IR [27], and can be used by researchers on breast infrared images . This, as far as we known, have never be done before for breast diagnosis (for locating perforating vessels in breast reconstruction surgery the use of cold and hot stimulation was used and compared with static thermography by Kołacz et al. [16]). In both, the patient attempts the common preparation and we used to ask them for, before capturing the images, pass for acclimatization in a room with controlled temperature (between 20 and 22 °C for 10 min) with arms raised over the head and with the breast naked. The 2 acquisition forms used on experimentations are: (1) A static protocol, where five captures are done: 1 frontal, 2 laterals in an angle of 45° with the camera and 2 laterals in direction of the right and left sides of the body (Fig. 1). (2) A dynamic protocol , where an electric fan is turned on, in the breasts for 2 min and twenty frontal images are acquired, in fifteen seconds interval between them [28]. The resulting infrared images present 640 × 480 pixels and were captured using a FLIR thermal camera , model SC620, which sensitivity of 0.04 °C range from −40 to 500 °C.

Fig. 1
figure 1

User interface of DMR-IR database (http://visual.ic.uff.br/dmi)

Full size image

3 Storage and Retrieval

The Database for Mastology Research with Infrared Image—DMR-IR contains infrared images and temperature matrixes, digitalized mammograms and clinical data acquired in the university hospital (Hospital Universitário Antônio Pedro—HUAP) of Federal Fluminense University, Rio de Janeiro, Brazil. The data are from patients of the gynecologic department of HUAP. There are data from healthy patients and from patients with a number of breast diseases , including cancer. This research is approved by the Ethical Committee of the HUAP and registered at the Brazilian Ministry of Health under number CAAE: 01042812.0.0000.5243. The DMR-IR is accessible through a user-friendly interface (http://visual.ic.uff.br/dmi) for managing and retrieving information. All data are from those patients that agree on signing a term of understanding and knowledge about the research details and consent on the use of their data for the research. Figure 1 illustrates one of the navigations pages of the static protocol with some patient information for filtering the wanted cases and images. Images can be downloaded in JPG or BMP format for mammograms and thermograms (infrared images). The thermograms can also be downloading as a 2D array of temperature . Reports of the exam can be downloaded in txt or xml format as well.

A relational model was employed to construct the database and a client–server application is used for data management. A retrieval tool was implemented using a client side application that provides a search dialog to compose the query. A server side application executes it using the tool and returns the results to the client [13, 31]. As in some cases, textual-based information is not enough for retrieval, especially when user’s search requirements are about image visual features , a CBIR (Content-Based Image Retrieval) tool was implemented. The image retrieval is based on extraction of interest points. An interest point of an object represents a specific area on the object around which the local image structure is rich in terms of local information about the image content.

Figure 2 shows the CBIR model for image retrieval given an image as sample and performing ASIFT [20]. In this, we use the frontal position of the thermograms . For image representation, the local visual feature considered is the Bag of Words (BoW) [32]. The basic idea of BoW is that a set of local image patches is sampled using some detected features on key points. They are composed by a vector of visual descriptors for each patch independently. Then the patches created are included into a set of visual words which constitute a codebook. After that, an image is described as a histogram-based representation of the codebook, and a classifier (the Support Vector Machine —SVM in the implemented tool) is used to find the image more similar to one associated to that the user have presented as sample for retrieval [25].

Fig. 2
figure 2

Content-based image retrieval (CBIR) with bag of words (BoW) model [25]

Full size image

4 Preprocessing

Mainly, the preprocessing step performs the segmentation of the data that is the identification of the Region of Interest (ROI) of the images, and the separation of this from the rest of the frame (that will be considered no relevant or the background). For diagnosis purpose, the thermal matrix and image are exported by the camera. Both data can be used in an analysis. Images can use for segmenting visually the ROIs, and the thermal matrixes (i.e., the file with thermal values, one per point in the scene observed) are used for feature extraction to help in diagnosis.

The segmentation process (i.e., the ROI), in the breast application , frequently generates two areas or masks, one for the right breast and the other for the left breast. According to the physicians of our groups there are two possible ways to consider the breast analysis relating to ROI segmentation [4, 8]. Both exclude head and neck and begin in the inframammary fold. Figure 3 shows these two types of ROIs: including (Fig. 3a, b) in the ROI the area of possible lymph nodes related to breast and armpits or excluding this area (Fig. 3c, d) [23].

Fig. 3
figure 3

Two possible breast regions of interests (ROI). a Right breast and b left breast of segmentation without armpits. c Right breast and d left breast of segmentation with armpits [23]

Full size image

The purpose of inclusion of the armpits area is to investigate the ability to detect altered lymph nodes, like it is done in sentinel lymph node exams (that verifies if a cancer is in metastases) [18]. Ground truth (i.e., segmentation is done manually by specialist) for this type of segmentations and the results achieved by two automatic approaches can be seen in http://visual.ic.uff.br/en/proeng/marques/ or downloaded in http://visual.ic.uff.br/en/proeng/marques/gt.php. Details about how they are done can be found in Marques et al. [19]. The same process of separation of relevant data can be applied over the array of temperature, i.e., the temperature to be analyzed can be cropped and the elements outside the region of interest (ROI) can not be used (Fig. 4).

Fig. 4
figure 4

Series do dynamic acquisition and the region of interest (ROI) on the images

Full size image

For comparison of the influence of the use of this additional area on diagnosis, both ways of segmentation are performed in the same group of patients and the same methodologies for features extraction and diagnostic suggestion are conduced to a complete scheme by Resmini [23]. Segmentations used in this comparison are available in http://visual.ic.uff.br/en/proeng/rresmini_thesis.

5 Three-Dimensional Reconstruction

The geometric modeling of the patient breast is useful for many application such as: (1) to merge temperature information with others [5] specially with 3D exams which represent the same patient like magnetic resonance, (2) to represent a model of the breast with a 3D mesh for surgical reconstruction after mastectomy modeling, (3) to identify points of correlation between IR exams and models of the patient body for finite elements analysis or other forms of treatment of the inverse problem (i.e., what is inside of the body that can result in a thermal distribution of skin ’ surface) [3, 11, 24] and (4) for some numerical analysis and other examinations like Electrical Impedance Tomography (EIT) [14] and 3D Ultrasound [35]. Moreover, three dimensional meshes of the real model can be used in surgeries simulations, for the personal project of prosthesis, for breast reduction, and other procedures.

We proposed two approaches for reconstruction. The first one uses a frontal image and two lateral (orthogonal to this) images of the breast for shape reconstruction. Figure 5 shows some steps of this approach [34]. The second extends the first by proposing an apparatus to be used during the capture with thermal camera and two Kinects (MicroSoft Xbox 360), improving the quality of the 3D mesh generated [2]. The steps of the second one can be seen in Fig. 6.

Fig. 5
figure 5

Breast contour detection using a frontal and lateral view for each breast. Surface modeling from such curves are achieved after positioning the lateral curves on a plane parallel to the coronal over the nipple and using the bottom curve to define the size and shape of the breast [34]

Full size image
Fig. 6
figure 6

Schemes for data acquisition the infrared image and at same time the range cameras (kinects) for breast geometric reconstruction, the range image achieved and the obtained result [2]

Full size image

From the two-dimensional thermal images (Fig. 5) it is possible to reconstruct the patient breast geometry. The 3D geometric model proposed is based on fitting two curves (for each breast a frontal and lateral view is used). The first step is the identification of the points of the inframammary fold of the patient’s IR image). These points are computed from IR image adjusted by using the least-square method (LSM) [34]. These define the bottom curve used to define the size and shape of the breast. The spatial shape is defined by positioning the border of the lateral views orthogonally to this over the nipple. The curves are joined as indicated in Fig. 5. A computer graphics border representation can be used to generate the 3D mesh. Then calculated curves and surfaces are compared with those obtained from the process of asking the patient to be inside a laser scanner device in order to capture the shape and measure how well the calculated curves and surface fits the patient breast shape. The steps for these are:

  1. 1.

    Identification of the 2D coordinates of the breast border from the IR image red line and blue line in Fig. 5;

  2. 2.

    Definition of the middle point (or nipple position) of the inframammary fold image (red line) that will joint this curve with the vertical points (blue line);

  3. 3.

    Translate coordinates of all the points to the same axis and origin, and calculation of the 3D representation of the coordinates of the points, i.e., the (x, y, z) representation for the frontal breast lower curve as can be seen in order to have an orthogonal spatial limits of the breast;

  4. 4.

    Creation of an NURB surfaces using this limits, and the a 3D model of the breast [34].

The validation of generated model was realized using 3 volunteers that have been submitted to 3D laser digitalization of their bodies. Then the model of the real bodies obtained with the scanner and the proposed “steps” re compared: i.e., the laser scanned points with the proposed model of reconstruction from thermal images. It was found that, on average, the differences between the 3D shape from this method and the acquired model by laser scanner differ between 4.42 and 6.03%. The maximum value of differences is between 4.77 and 6.08%. In addition, the general analysis about average of maximum value, and the minimum value found on the validation is considered small and very much acceptable, turning the 3D model created by this methodology very close to real patient breast [34].

Range images are the name of the outcomes of a collection of imaging devices available to produce a 2D image considering the distance of the points in a scene to a specific point. This is normally associated with some type of sensor of the device. These images have as pixel values a measure of the corresponding distance (e.g., brighter values mean shorter distance). If the sensor used to produce the range image is properly calibrated, the pixel values can be used directly to compute distance in the desired length unit, such as centimeters or inch.

In the developed of a second technique for breast shape modeling we used two Kinects, and it consists of three stages: the first is calibration of the Kinects in an apparatus (Fig. 6 left); the second step carries out the registration of the clouds of points acquired by the sensors (Fig. 6 center); then, the reconstruction of the surface of the virtual object (Fig. 6 right). Comparison of this technique and the real breasts of five (5) volunteers and two (2) phantoms are done [2]. For the volunteers the breasts are captured by a laser scanner (as previous commented) and for the phantoms they are measured directly by a mechanical devise. The mean differences among the surfaces areas are 3.55%, or 0.93 considering the Dice similarity coefficient [9], in average, and we achieve mean differences among the distances of the real nipples and the reconstructed models of 3.51% in this technique.

6 Feature Extractions, Classification and Evaluation of Performance for Diagnostic Tools

The developed tools for diagnostic aid [10] can be divided as a function of time in static or dynamic. They originate from the use of data archived by the Passive Static Thermography or by the Active Dynamic Thermography , respectively.

Using the static protocol, 3 works to perform diagnosis have been developed by our group. In a first one [26], Lacunarity and the Hurst coefficient are calculated from each breast ROI. Both these features use the concepts of the fractal geometry and were computed in three approaches : using each ROI independently, combining both patient ROIs by subtracting one from the other or by feature subtractions. A total of 133 features were extracted in Serrano et al. [26] work from a sample of 28 volunteers. Classification algorithms of WEKA (http://www.cs.waikato.ac.nz/ml/weka/) were used. To identify the best result achieved, the performance of a cancer x normal classification were considered by plotting the true positive rates (sensitivity) against the false positive rates (specificity) of each used classifiers, i.e., the receiver operating characteristic (ROC) curve were considered. The Naïve Bayes technique achieved the best score for correctly identifying cancer using these features: it presented 0.958 of area under the ROC Curve (AUC).

In a second work [22], a total of 320 features were computed: most of them are from the geo statistic (Ripley’s K function, Moran index, and Geary coefficient), but also features from fractal geometry (using Higuchi fractal dimension approach ) and statistic (average and standard deviation and second order moments). They are computed from each breast separately without combination or subtraction of images or features. The same 28 patients are considered. These characteristics are grouped in an array of data processed by the WEKA (http://www.cs.waikato.ac.nz/ml/weka/) classifier software , as the previous one. The principal component analysis (PCA) is performed in two ways: with the base divided into four groups of features and with the entire amount of data. The Support Vector Machine (SVM) was used for patient classification by the PCA best group. The best accuracy is 82.14 and 91.70% is the best sensitivity obtained [21].

In a third work [4], the used features were from fractal geometry (Higuchi fractal dimension and Succolarity), basic statistics (average, standard deviation, and thermal amplitude), histograms from the thermal matrix, geostatistics (Moran index and Geary coefficient), and Ripley Diversity Index, using a total of 40 features computed from 51 patient’s ROIs. Some tests were performed to find the most expressive set of features by Principal Components Analysis (PCA) and genetic algorithm feature selection. The work used SVM for classification and the AUC to perform comparison of outcomes (resulting in 85.20%).

Using data from the dynamic protocol , Silva et al. [28] proposed a methodology for analyzing temperature variations in order to detect a number of breast abnormalities (including cancer), using unsupervised and supervised machine learning techniques, which characterizes the methodology as hybrid. The sequence of thermograms from each patient is firstly segmented and the ROIs are registered using the initial frame as reference. Then, for each point, functions representing the change of temperature during the examination are built and the k-means algorithm is applied on these functions using various values of k. Indices of clustering validation are applied to evaluate the possible groups for each value of k, generating values to be used in the classification model. Data mining tools and hyper parameter optimization (CASH) were used to combine groups in order to classify the patients’ breast. Further, classifiers based on Bayesian networks, neural networks, SVM and decision tree were used. Among 39 tested classification algorithms, K-Star and Bayes Net obtained classification accuracy of 100%. Furthermore, among the Bayes Net, Multi-Layer Perceptron, LibSVM [6] and J48 classification algorithms, an average accuracy of 95.71% was obtained.

Dynamic protocol data were also used by Silva [30] to compute ROI features (first and second order statistical, clustering, histogram, fractal geometry, and diversity indexes) and then to organize temporal series. These were after reorganized in subseries with different cardinalities. The top subseries of features were selected and used in SVM to classify the breast. The leave-one-out technique is employed for validation using 64 patients (32 healthy and 32 with some disease ). Two features were extracted from each series and submitted to the classifier, they are: the amplitude of the series and the square root of the sum to the square of the series. The values were normalized between 0 and 1. The SVM classifier in the WEKA tool was used with all their default parameters. The SVC (Support Vector Classification ) learning was used (in both C-SVC and Nu-SVC). Detailed results of each group of feature (i.e., its sensitivity, specificity, accuracy, Youden index and the AUC, for each analysis, are presented in Appendix B of Silva [30]. The best results of the analysis are shown in Fig. 7.

Fig. 7
figure 7

Accuracy by group of features using active dynamic thermography [30]

Full size image

The horizontal axis of Fig. 7 shows accuracy in percentage, it is possible to notice that the greatest accuracies were obtained by the features from simple statistical group and by the union of all the features, achieving 97%. On the other hand, groups formed by the diversity index (horizontal and vertical) and lacunarity do not obtain good results.

7 Conclusions

In this chapter, we present several approaches that have been implemented related to the use of infrared images to breast modeling and diseases diagnosis; they consider all aspects of the process of diagnostic tool implementation: capture thermal matrix of patient body, storage and retrieval of images from a database, segmentation, 3D reconstruction , feature extraction and classification. The experiments and implementations were done by the Visual Lab group of the Fluminense Federal University in Niteroi, Rio de Janeiro, Brazil. The results are promising and it promotes diseases classification with very high accuracy . Each aspect considered in this chapter is an important application of infrared images for breast modeling and diagnosis. Test showed that the proposed methodology is able to detect breast anomalies, thus contributing to adopted thermography for breast cancer screening programs. For diagnostic recommendations the achieved results support the statement that IR analysis is able to detect breast anomalies and to insert the thermography in clinical routines for breast diseases screening . When considering the dynamic protocol versus the static one, tools for diagnosis implemented using the variation of the patient temperature (the dynamic protocol data) always achieve better results even when very simple approaches are used like only the temporal series of data [28, 29].