1 Medical Imaging

1.1 Introduction

Medical imaging is like a portal to view internal parts of bodies which are used for assisting medical diagnosis and analysis. This creates a visual representation for medical professionals to get an understanding of the current functioning or previous ailment of organs, tissues, or any interior section of the body. Biomedical images provide a wide range of patterns and designs like bones, muscles, etc. to diagnose diseases.

1.2 History

Prevalence of medical imaging started after the discovery of X-ray by Wilhelm Conrad Roentgen, a professor in Wuerzburg University in Germany, in 1895. While working in the lab, he discovered that only few rays showed internal parts of human body where the rays could penetrate for other its was opaque. He received a Nobel Prize in 1901 (Bradley 2008). The discovery of X-ray opened up new avenues in research for understanding human body structure. Interest was in the area of surgery and medical diagnosis. Few months after the discovery, experts in Europe and the United States started to use radiographs as a guide for medical professionals specially used in wars to locate bullets in wounded soldiers (Fig. 20.1).

Fig. 20.1
figure 1figure 1

Wilhelm Roentgen’s first medical X-ray of his wife’s hand (Kevles 1996; Sample 2007)

Full size image

X-ray tomography started becoming popular in 1900. Slices of anatomy information also called as “tomograms” were taken into consideration to view the internal body pattern. Other topographies that are frequently used are computerized axial tomography (CAT) scanning or computed tomography (CT) scanning that came around the 1970s and magnetic resonance imaging (MRI) that does not X-ray (Bradley 2008; Toennies 2012) (Fig. 20.2).

Fig. 20.2
figure 2figure 2

X-ray machine showing crack in the bone structure (http://www.imaginis.com/faq/history-of-medical-diagnosis-and-diagnostic-imaging)

Full size image

In the 1950s, nuclear medicine became famous. Today positron emission tomography or “PET” scanning is a well-known nuclear medicine. Instead of emitting gamma rays, it emits positrons (Bradley 2008). Depending on the focus of scanning, contrasting material (agent) injections are given to patients to enhance the visibility of certain tissues or blood vessels.

The benefits of these techniques are (Toennies 2012; http://www.imaginis.com/faq/history-of-medical-diagnosis-and-diagnostic-imaging):

  • Small quantity of X-ray is required.

  • Improved quality of images.

  • Easy to store the medical data as images for analysis.

  • Improve diagnosis by proper analyzing techniques.

1.3 Types of Medical Imaging Modalities

Technology has made it easy to obtain medical images without having invasively extracting information. There is a huge range of image modalities that could be considered to classify the types of medical images or techniques that are considered in biomedical image analysis.

  1. (a)

    Radiography

    This imaging modality uses various rays like X-rays, gamma rays, and other radiations to view internal structures (Carroll 2014). X-ray beams are projected on the object, and the object would absorb the radiation to display structural design, composition, and density.

    The study of anatomy using radiographic information is known commonly as radiographic anatomy (James and Dasarathy 2014).

    Sub-classification of radiography:

    • Projectional radiographs (commonly known as X-ray) produce 2D images. They are normally used for detecting diseases in the lungs, stomach, intestines, etc. (Radiology – acute indications 2017; Radiographic Standard Operating Protocols (PDF) 2015) (Fig. 20.3).

    • Fluoroscopy – X-ray used at low dosage and used for image-guide surgeries for getting visual display of internal working of organs (Wang and Blackburn 2000; Last Image Hold Feature 2010) (Fig. 20.4).

  2. (b)

    Computed Tomography (CT or CAT Scan)

    Computed tomography (CT) scan or computerized axial tomography (CAT) scan makes use of combinations of many X-ray beams which are computer processed. Different angles or sections are taken into consideration to get the final scanned images which are handled by the computer. These images are obtained without having to cut open the patient’s body (CT Scan 2018; Shrimpton et al. 2011) (Fig. 20.5).

  3. (c)

    Magnetic Resonance Imaging

    MRI scan is a technique applied by radiologists that uses magnetism of the huge magnetic coil drum, and algorithms are used to reconstruct image of body structures (Bradley 2008). The MRI scanner is a circular drum that contains a magnet and a sliding table as shown in Fig. 20.6. The patient is placed on a moveable bed that is inserted into the magnet. The magnet creates strong magnetic field/signals which are later used by Fourier transformation using compressed sensing concepts to produce the final image, what we all know as MRI (Zhu 2003).

  4. (d)

    Nuclear Molecular Imaging

    Nuclear medicine when used in diagnostic imaging is commonly referred to as molecular imaging which uses properties of particles that are emitted from radioactive material to diagnose or treat various diseases. Figure 20.7 shows the nuclear medicine image of the whole body which is used to diagnose bone-related diseases such as fracture, infections, abnormalities, etc. (https://en.wikipedia.org/wiki/Nuclear_medicine).

    They are sub-classified as:

    • SPECT – 3D images taken from gamma-based cameras from different projected angles are combined to reconstruct this image (https://en.wikipedia.org/wiki/Nuclear_medicine).

    • PET – positron emitting radionuclide (a form of gamma rays) is captured after it reflects from the human tissue molecules to form PET image (Bailey et al. 2005). It is commonly used in neurology, cardiology, muscular-skeletal imaging, etc. (Carlson 2012) (Fig. 20.8).

  5. (e)

    Ultrasound

    Ultrasound is a sound wave-based application which is considered to be a diagnostic imaging technique. These sound waves are known to have very high frequencies which are way higher than human hearing ability (Novelline 1997). Famous application is in obstetrics scanning which means ultrasound scanning of pregnant women to view the development of the fetus. It is also used to view internal body structures such as organs, bones, muscles, etc. (DistanceDoc and MedRecorder 2011; Ultrasound Imaging of the Pelvis 2008) (Fig. 20.9).

  6. (f)

    Functional Near-Infrared Spectroscopy

    Functional near-infrared spectroscopy is an optical imaging technique that is non-invasive. This technique used low-level light to see the internal working of the brain and its activity through the movement of blood flow in them (http://researchimaging.pitt.edu/content/near-infrared-spectroscopy-nirs-brain-imaging-laboratory; Coyle et al. 2007) (Fig. 20.10).

  7. (g)

    Magnetic Particle Imaging

    Magnetic particle imaging is an imaging technique used for tracking superparamagnetic iron oxide nanoparticles. It is highly sensitive, and depth of the structure can be analyzed (https://en.wikipedia.org/wiki/Magnetic_particle_imaging). This technique has been used for researches in areas such as cardiovascular performance, neuroperfusion, and cell movement tracking (Weizenecker et al. 2009; Yu et al. 2017) (Fig. 20.11).

Fig. 20.3
figure 3figure 3

Abdominal radiographs (https://en.wikipedia.org/wiki/Projectional_radiography)

Full size image
Fig. 20.4
figure 4figure 4

A fluoroscopy X-ray image shown during implant surgery (https://en.wikipedia.org/wiki/Fluoroscopy)

Full size image
Fig. 20.5
figure 5figure 5

CT image of the skull showing different cross-sections (http://pleasantonimaging.com/services/computed-tomography-ct/)

Full size image
Fig. 20.6
figure 6figure 6

Brain MR image reconstruction process (Zhu 2003)

Full size image
Fig. 20.7
figure 7figure 7

Nuclear molecular image of a whole body (https://en.wikipedia.org/wiki/Nuclear_medicine)

Full size image
Fig. 20.8
figure 8figure 8

PET scan image of the breast (left) and cancer cell detection shown using a tracer (right) (Images courtesy of Dr. Kathy Schilling, Medical Director of Lynn Women’s Health and Wellness Institute)

Full size image
Fig. 20.9
figure 9figure 9

Ultrasound image of a fetus in the womb (https://en.wikipedia.org/wiki/Ultrasound)

Full size image
Fig. 20.10
figure 10figure 10

Brain monitoring with near-infrared spectroscopy (http://www.aorticdissection.com/DISEASES%20OF%20AORTA.htm)

Full size image
Fig. 20.11
figure 11figure 11

Magnetic particle imaging of lungs tracking drug deposition (Dolovich and Labiris 2004)

Full size image

1.4 Comparison of Medical Imaging

Few imaging techniques are being used for a long time, and the benefit of them still exists which make them the most famous and well known across the medical field. Table 20.1 shows some of these techniques and their comparison as regards the common properties that make these images feasible.

Table 20.1 Comparing medical images (Goel et al. 2016)
Full size table

2 Medical Image Analysis

2.1 Introduction

Analysis of medical images has been the integral part of any diagnoses, treatments, procedures, etc. These analyses could be carried out by medical professionals to help predict or take action with regard to the patients’ health. Since these images are obtained non-invasively and can be stored, they serve to act one important aspect in electronic health record (EHR) for future references (Toennies 2012).

Reasons for carrying out medical image analysis are as follows (https://www.doc.ic.ac.uk/~jce317/history-medical-imaging.html):

  • Clinical study – to detect patterns or structure in images that could describe or prove hypotheses of the study. These are used for scientific analyses and future case study analyses for educational institutions for training budding medical professionals.

  • Diagnosis – to diagnose chronicle illness or diseases by detecting tumor or other patterns. Doctors or experts in their field identify the medical conditions of patients.

  • Treatment planning – after diagnosing comes the treatment to illness. Analyses need to be done about the course of action to be taken for diseases which could be drugs or medical procedures. Planning for the treatment needs serious research regarding previous health conditions or allergies. This can be obtained from medical image history of patients.

  • Computer-aided surgeries – advancement in technology has given an automated assistance to doctors in various areas of health care from diagnoses, treatment, surgeries, post-surgery care, etc. They are used as guided tools for surgeries. Doctors have even started performing remote operations which could save millions of lives (Toennies 2012; http://www.imaginis.com/faq/history-of-medical-diagnosis-and-diagnostic-imaging).

Modern radiologists have various tasks to be performed during the diagnosis process. Medical image data is not only about reading the image, but other aspects contribute toward the analysis which are as shown in Fig. 20.12.

Fig. 20.12
figure 12figure 12

Flow chart of image analysis by a radiologist. (Image from J. Galeotti, class material from “Methods in Medical Image Analysis”, Carnegie Mellon University 2018)

Full size image

Image processing has a wide range of applications especially in medical area. Visual images have been contributing to various medical analyses (Goel et al. 2016; Rao and Rao n.d.). Few of the applications are mentioned below:

  • Tumor detection

  • Fracture detection

  • Structural disabilities

  • Cancer detection

  • Heart defects and diseases

  • Tuberculosis

  • Birth defects

  • Neurological functioning

2.2 Image Pre-processing

The medical field majorly deals with data problems like understanding, acquiring, accessing, denoising, cleaning, and analysis of data as shown in Fig. 20.13 (Image from J. Galeotti, class material from “Methods in Medical Image Analysis”, Carnegie Mellon University 2018).

Fig. 20.13
figure 13figure 13

Data-related problems (Maier-Hein et al. n.d.)

Full size image

Image data experts have been trying to extract information based on content and textual description, but image feature extraction has been the key point (Scholl et al. 2011; Deserno et al. 2009). Feature analyses range from the entire image to specific localized section to some structural-based approaches (Scholl et al. 2011; Long et al. 2009; Tagare et al. 1997).

Medical image pre-processing has a series of steps that need to be taken into consideration as shown in Fig. 20.14 (Goel et al. 2016).

Fig. 20.14
figure 14figure 14

Steps in image pre-processing (Goel et al. 2016)

Full size image

2.3 Challenges in Medical Image Analysis

There are a number of specific challenges in medical image processing (Thirumaran and Shylaja 2014) (Fig. 20.15). They are:

  • Pre-processing of image using image enhancement and restoration for best quality of image data

  • Automated and accurate image segmentation of features of interest (region of interest)

  • Automated and accurate image registration and fusion of multiple images

  • Classification of image features or properties

  • Simulation software that can be used to rehearse and plan procedures, evaluate access strategies, and plan treatments.

  • Latest being is visualization of the environment in which image-guided procedures or reconstruction of working of human body in 3D.

Fig. 20.15
figure 15figure 15

Flow of medical image processing (Image from C. Parmar, D.B. Joseph, A. Hosny, J. Quackenbush, H.J.W.L. Aerts, “Data Analysis Strategies in Medical Imaging”, DOI: 10.1158/1078-0432.CCR-18-0385 Published August 2018)

Full size image

Medical image analysis has key tasks, which will be explained in detail in subsequent sections of this document such as:

  • Classification

  • Segmentation

  • Registration

  • Deep learning (DL)-based analysis

2.4 Conclusion

Images play an important role in health care. Technology advancement in medical image has helped doctors to get an insight into the human body without having to cut open the body (Goel et al. 2016; Binh 2010) and to achieve the best possible diagnosis, treatment, and other surgical procedures via image analysis obtained after noise removal and high-quality resolution (Tsui et al. 2012).

3 Medical Image Classification

3.1 Introduction

Recently, rapid development in the combination of machine learning (ML) and medical field has become a popular and active topic in research area. Thus, medical image classification plays a significant role in computer-aided diagnosis (Lai and Deng 2018). The main concern for researchers in this area is how to extract features from medical image and classify them into the same model to achieve an accurate result for identifying the parts of a patient’s body which are affected by the specific disease (Aberle et al. 2010).

The main purpose of image classification in the medical field is specifying the affected parts of the human body by disease, instead of gaining the high accuracy result; therefore, in this chapter we discuss the various medical image classification techniques in detail (Miranda et al. 2016).

3.2 Overview of Image Classification Techniques

Image classification process is divided into three stages, namely, pre-processing, feature extraction and feature selection, and classification. After the pre-processing level, by using feature extraction methods, analyze the images to extract the most appropriate features from input data for classification process, and then using feature selection methods, select the most correlated features to reduce the dimension of data which can be effective in improving performance of classification methods (Miranda et al. 2016; Lashari and Ibrahim 2013) (Fig. 20.16).

Fig. 20.16
figure 16figure 16

A classification process (Miranda et al. 2016)

Full size image

Some of the main feature selection techniques are:

  • Genetic algorithms-based optimization

    The generic algorithms are one of the powerful methods which are based on natural selection (Kaushik et al. 2013). This technique has some disadvantages, which cause deflection in medical image segmentation (Cao et al. 2017).

  • Linear discriminant analysis

    It is one of the dimensionality reduction techniques which goal is to preserve most of the features without eliminating any data in order to separate different classes as much as possible (Dhawan 2008; Sharma 2015).

  • Principal component analysis (PCA)

    Another method of dimensionality reduction is PCA which is used for transformation methods to reduce a number of correlated variables to a smaller number of variables in a new subspace (Ashour and Salem 2015).

    Therefore, PCA is an applicable technique in medical image processing that can be used in feature extraction, feature selection, image segmentation, and image registration (Ashour and Salem 2015). PCA cannot be efficient in selecting features, if input images include noises (Dhawan 2008) (Fig. 20.17).

Fig. 20.17
figure 17figure 17

Performance of PCA on ultrasound images (Cao et al. 2017)

Full size image

After completing the feature extraction and feature selection steps, start classifying images.

This part provides a brief explanation of some of the classification techniques which are more applied to classify and detect abnormalities in medical images (Lashari and Ibrahim 2013).

  1. (a)

    Neural Network Classification

    Neural Network (NN) is a computational model that has an important impact on classification by using supervised and unsupervised learning techniques. Neural Network models have some advantages; the first advantage is they are non-linear models; therefore, they are flexible in performing any complicated real-world application models. The second advantage of Neural Network is universal functional approximations which enable to approximate any function with ideal accuracy results, and the third advantage is Neural Network can regulate itself to the input data without any characteristics operational. In other words, it is a self-adaptive model (Lashari and Ibrahim 2013; https://pdfs.semanticscholar.org/1ba9/d67c80b6a762c11b9d519367e9e13a9c5c4f.pdf).

  2. (b)

    Support Vector Machine (SVM)

    SVM is the machine learning model that uses different algorithms to analyze data for the purpose of reaching the efficient classification outcome (Sharma 2015). Furthermore, this model is a binary classifier which provides the maximum separation line between two classes. However, SVM has disadvantages as it needs longer time for training data and does not manage discrete features (Lashari and Ibrahim 2013; Ehteshami 2017).

  3. (c)

    Statistical Classification Methods

    These models are based on supervised and unsupervised approach. Supervised learning method can be accomplished by using Bayesian decision theory because it is based on statistical classification and probabilistic methods. For instance, nearest neighbor and Bayesian model are the most practical classifiers. In addition, for performing supervised methods, besides training data and test data, it requires label data as well (Table 20.2).

Table 20.2 List of medical image classification methods (Lashari and Ibrahim 2013)
Full size table

The unsupervised learning technique classifies the data by separating the feature space, like K-means (Miranda et al. 2016; https://pdfs.semanticscholar.org/1ba9/d67c80b6a762c11b9d519367e9e13a9c5c4f.pdf; Dhawan and Dai 2008).

3.3 Medical Image Classification Challenges

  • The first challenge is the variety of features in medical images which make challenges for training dataset; as a result, the classification outcome will be decreased.

  • The second challenge is about the size of the medical images. Since, the medical images are very small, extracting and selecting enough valid information from dataset is not easy (Lai and Deng 2018).

3.4 Conclusion

Study in medical image classification can be beneficial for both computer-aided diagnosis and teaching purposes in medical fields. Recent research in this area could be helpful for analyzing and diagnosing diseases rapidly. This chapter has provided the overview of image classification methods and algorithms which are using medical images to identify the human body in order to distinguish images showing diseases from ones which do not (Miranda et al. 2016).

4 Medical Image Segmentation

4.1 Introduction

Today, with growing usage of computed topography (CT) and magnetic resonance (MR), X-ray image, digital mammography, and other imaging modalities, analyses of these images manually are not possible; therefore, digital image processing and computer algorithms, such as image segmentation methods, play an important role in diagnosing diseases and progressing biomedical research areas especially in medical imaging applications (Petitjean and Dacher 2011; Pham et al. 2000). Image segmentation is a process that divides an image to many homogeneous sub-regions which have the same characteristics as color, depth, and intensity (Withey and Koles 2007).

For instance, MR images which provide high resolution of three dimensional (3D) are the most common applications that use image segmentation techniques. Image segmentation can analyze both 2D and 3D images, and the main difference between them is processing the pixels in 2D and voxels in 3D (Despotovi 2015) (Fig. 20.18).

Fig. 20.18
figure 18figure 18

Visualization of brain MR image. The square in the 2D MRI slice represents an image pixel (i, j), and the cube in 3D image illustrates an image voxel (i, j, k) (Despotovi 2015)

Full size image

In the following section, some important methods of medical image segmentation are reviewed in order to introduce the segmentation process and its importance in analyzing medical images accurately for diagnosing disease.

4.2 Review of Medical Image Segmentation Techniques

As it is mentioned before, segmentation is a technique that provides wide diagnostic insights in the medical field. Using this technique in medical images can be improved, detecting of image’s boundaries, cell counting, scaling organs of human body and many other applications automatically (https://www5.cs.fau.de/research/groups/medical-image-segmentation/).

In this part, some of the medical image segmentation methods are provided:

  1. (a)

    Intensity-based segmentation method

    In this method, pixels in 2D images and voxels in 3D images are classified based on their intensity, for instance, brain MR images contain three tissue types, namely, cerebrospinal fluid (CSF), white matter (WM), and gray matter (GM), which can be identified based on intensity after using intensity-based segmentation method (Despotovi 2015) (Fig. 20.19).

  2. (b)

    Thresholding segmentation method

    This technique thresholding histogram gray scale images which it is one of the traditional and simplest method in medical image segmentation that it can be categorized in intensity-based methods. In other words, while thresholding segmentation is applied on medical images, intensity histogram is used in order to distinguish intensity value and separate different classes. The result of applying thresholding method on an abdomen CT image is illustrated in Fig. 20.20 (Despotovi 2015; Aggarwal 2010).

    Thresholding method includes multiple groups, such as (Despotovi 2015):

    • Local threshold which is dependent on the position in the image

    • Adaptive thresholding

    • Global or single thresholding

    • Multi-thresholding

    Thresholding is an efficient and fast technique; however, it has some limitations; first, choosing an appropriate value for threshold for different medical images causes many difficulties, and second, in low-contrast images, it processes the distributed class of pixels rather than connected areas, so it is required to use connectivity algorithm before thresholding process (Despotovi 2015; Sahoo et al. 1988).

  3. (c)

    Region growing segmentation method

    The major purpose of this method is to form a region for segmentation based on more homogeneity between pixels or voxels (Withey and Koles 2007). In addition, this method can be categorized in intensity-based segmentation as well. Initially, for processing the region growing techniques, require selecting a seed point by an operator manually, then after testing similarity of neighborhood pixels or voxels, region continue growing until to get to the heterogeneity pixels (Withey and Koles 2007; Despotovi 2015).

    Region growing obtains impressive result to segment medical images. For example, in brain MR images, it could segment brain tumor and brain vessels successfully (Despotovi 2015; Passat et al. 2005; Haralick and Shapiro 1985). However, this technique has some disadvantages: first, it is sensitive to noise so that it can affect the segmented region and disconnect it from the related region, and second, initializing the seed point by human is difficult and time-consuming because the operator requires to select a seed point for each different region (Kaur and Singh 2011) (Fig. 20.21).

  4. (d)

    Edge detection segmentation method

    Edge detection algorithm is one of the common methods that are used in medical image segmentation. The basic idea of this technique is to detect the boundary or any interruption in the image (Upadhyay and Kashyap 2016). This method contains different algorithms (Aggarwal 2010):

    • Hough transform based

    • Edge relaxation

    • Border detection method

    Some limitations of this method are that it is sensitive to noise, segmentation can be completed by combination of edge detection and region growing techniques, and some lines appear that are not edge in outcome which effect on the final result (Aggarwal 2010) (Fig. 20.22).

  5. (e)

    Classification segmentation method

    Classifier methods are known as statistical pattern recognition which is a type of segmentation techniques that can divide images to feature space by labeling them.

    This method is divided into supervised and unsupervised learning. Supervised learning classification is a time-consuming method because the image has to be segmented manually rather than using automatic segmentation for other images; therefore, supervised needs training and test dataset and also label dataset. Moreover, another disadvantage of supervised classification is that using the same training dataset for various images reduces the quality of result (Pham et al. 2000; Withey and Koles 2007; Anbeek et al. 2005). Some examples of supervised classification method are K-nearest neighbor and Bayesian classifiers (Withey and Koles 2007).

    Unsupervised classification is a type of statistical clustering which uses expectation-maximum (Pham et al. 2000).

Fig. 20.19
figure 19figure 19

Illustration of an example of intensity-based segmentation method. (a) Original brain MR image and (b) related segmentation image. The segmentation image indicates three main tissue types (Lashari and Ibrahim 2013)

Full size image
Fig. 20.20
figure 20figure 20

An example of thresholding segmentation. (a) Original abdomen CT image and (b) segmented abdomen CT image by thresholding technique (Aggarwal 2010)

Full size image
Fig. 20.21
figure 21figure 21

Illustration of the result of applying region growing method on a brain MR image. (a) Original brain MR image with selected seed point manually and (b) segmented brain MR image by region growing method which indicates the final segmented region (Despotovi 2015)

Full size image
Fig. 20.22
figure 22figure 22

An example of edge detection technique. (a) Original abdomen CT image and (b) result of applying edge detection technique on abdomen CT image (Aggarwal 2010)

Full size image

4.3 Conclusion

In this chapter, we discussed some segmentation techniques briefly and also limitations and disadvantages in medical images. In conclusion, image segmentation is one of the most challenging areas in image processing study which can be most efficient in computer-aided diagnosis and identify diseases in medical images especial 3D images. It is expected that these algorithms will become more practical in biomedical field to achieve faster and higher-quality diagnosis of diseases from 3D medical images reconstructing and visualizing the anatomical structures (Despotovi 2015).

5 Medical Image Registration

5.1 Introduction

In this section, we discuss the domain of image registration. We provide the elements of this field, current developments in this direction, the applications, and future scope of this topic. Medical image registration is one of the key elements in analysis of medical data. It is essential for mapping medical images and data to the correct physio-structural components of the body, for relating changes in captured data across spatio-temporal plane, and for generating an atlas of human structural biology. Registration consists of matching points of interest between source and destination images, transformation of source data to the target data, and optimization of the result to best suit the application. We next detail each of the abovementioned three aspects of the registration process and review the related literature.

5.2 Transformations (Deformations)

As previously mentioned, a typical registration process consists of three key steps: correspondence, transformation, and optimization. Although correspondence can be considered a first step of the process, it often is dictated by the transformation process. In fact, the registration process can be summed up in the following formula for the optimal transformation:

$$ \Psi \left(\left(\mathrm{D},\mathrm{S}\upphi \mathrm{W}\right),\mathrm{R}\left(\mathrm{W}\right)\right) $$
(20.1)

Here, Ψ denotes the quality of the transformation process. This is often considered as a measure of the transformation in literature but can also be considered as qualitative measure when manual validation of the transformation process is performed. This is often the case where medical professionals exert their choice of the registration result to enhance automated transformation. Automated transformations mostly reply on quantitative transformation measure. D and S denote the destination and the source image, respectively, with the aim being to transform the source image to destination. The transformation is aided by ϕ, which denotes the transformation process used on the domain of pixels, W, of the source image. A reward function, also known as a regularization term, is used to balance the transformation process. We will focus on the transformation process next. The transformation process is often referred as the deformation process. Transformation processes have unique properties. Transformation properties are mostly ill posed according to Hadamard definition (Hadamard 1923). In perspective of real-time processing transformation parameters are as less as possible. Likewise, the constraint should ensure that as little as displacement is incurred in the transformation process.

We now discuss the variety of transformations available and presented in recent literature.

Transformation process can be categorized into physical model-based approaches, geometric-based methods, and knowledge-based paradigms. It is noted that this approach of categorization is not independent but has significant intersection. The aim of this categorization is to enforce logical categorization to the transformation field.

Physical-based models use plant models like elastic model by Navier-Cauchy is planar or hierarchical approach (Bajcsy and Kovačič 1989; Gee and Bajcsy 1999). The physical models can be linear (Leow et al. 2005; He and Christensen 2003) or non-linear (Rabbitt et al. 1995; Pennec et al. 2005) in nature. These include elastic as well as diffusion model.

The next approach is geometric-based transformations. They include radial basis, elastic body, and piecewise affine models. The radial basis functions use kernels to find the transformations of source to target image. The radial basis function falls under kernelized approach wherein the distance better source and target is optimally mapped via kernel thereby generating the transformation parameters. Typical kernelized formulation is expressed as:

$$ \Gamma (x)=\sum \limits_{i=0}^{N-1}{\vartheta}_iF\left(x,{p}_i\right) $$
(20.2)

where Γ () denotes the transformation operation, ϑi is the weight of the ith parameter of the kernel transformation, and pi is the ith parameter of the kernel. F () is the kernel used. Kernels like radial basis and thin plate spline have been frequently used in literature (Zagorchev and Goshtasby 2006; Yang et al. 2011; Bookstein 1989; Bookstein 1991). Recent advancements in this field include Tensor-based deformation, wherein each dimension is deformed using Tensor transformations (Declerck et al. 1997; Rueckert et al. 1999). Another variation in this direction often perused in literature is the use of piecewise affine model (Hellier et al. 2001).

The final approach that is presented in literature is the knowledge-based approach. In many situations, further information is available on the registration process. This can be in terms of the statistical variability of the transformations (Cootes et al. 1995), availability of biophysical models (like tumor growth models) (Clatz et al. 2005; Hogea et al. 2007), and biomechanical models of human organs (Bharatha et al. 2001; Hensel et al. 2007).

5.3 Matching (Correspondence)

Matching process consists of two aspects, namely, location matching and interpolation. Interpolation is typically used within each iteration of the registration process wherein we find the most plausible value of the transformation based on the neighborhood kernel. The key aspect of matching can, however, be considered as location matching. Location matching between source and destination (or target) images is what we primarily consider as matching. If we register a functional image to a structural image, for example, matching a PET image or fMRI data (functional MR image) to a CT Image, we need to map functional image to structural image. In such case, the functional image may be first mapped to a template image, which provides structural bearings. It is then related to the structural image.

Registration of two functional images in similar lines entails two such intermediate mapping, one for each functional image. Thus, for matching either intrinsic or extrinsic markers are used. Extrinsic marker methods rely on human intervention to relate the markers, while the intrinsic ones use various algorithmic approaches including feature and intensity measures. We now discuss some of the key works related to image location matching.

The most commonly used matching method is geometric in nature. Geometric methods relate two images by minimizing geometric criterion at landmark locations in the image. Geometric matching includes finding points of interests, creating correspondence between the suitable points, creating transformations with the suitable points instead of using correspondence, and finally joint use of correspondence and transformation. Key point detection procedures include the famous Harris point detectors (Triggs 2004; Mikolajczyk and Schmid 2004), invariant feature detectors (Mikolajczyk et al. 2005), multiscale approach (Kadir and Brady 2001), and histogram-oriented approach like SURF and SIFT (Mikolajczyk and Schmid 2005; Morel and Yu 2009). Using the feature points detected correspondence-based matching can be used. They include use of descriptor distance as well as incorporating geometric constraints (Cheung and Hamarneh 2009; Ni et al. 2008; Torresani et al. 2008). Use of Gaussian mixture model (Jian and Vemuri 2011) provides an approach for transformation matching. The above methods are cleverly weaved together by the famous iterative closest point algorithm (Besl and McKay 1992).

The geometric-based techniques are extended to spatio-temporal to incorporate matching within modal, multimodal, and temporal data. In this situation, care is taken to constraint the prior mentioned matching processes to suit the variability of the data. The key criterion in such situations is the cross-correlation coefficients using either intensity or attribute features (Kim and Fessler 2004). Information theoretic approach is also used in literature with mutual information being the primary measure (Viola and Wells III 1997).

5.4 Optimization

Optimization framework is the core of the registration process. Using the optimization framework, we can choose the best transformation parameters which generate the best measure of transformation for a minimum cost function. The cost function is typically considered as a difference between source and target images. An implicit factor which plays a key role in the optimization process is the computational efficiency as well as the rate of convergence of the optimization process. As more and more registration algorithms are required to be near real time, this implicit constraint becomes highly significant. Based on the nature of variables, the optimization process can be broken into continuous parameter optimization, discrete parameter optimization, or hybrid optimizations. We will discuss these algorithms next.

As mentioned previously, optimization process may be separated into continuous, discrete, and hybrid forms. Continuous approach assumes the optimization parameters to be real. This allows the objective function for the continuous optimizations to be differentiable. Such case therefore utilizes iterative update to real optimal value using delta increment. The typical update rule is provided in the following equation:

$$ {\theta}_{t+1}={\theta}_t+{\alpha}_tf\left({\alpha}_t\right) $$
(20.3)

The above equation shows the update of the continuous parameter using step size alpha and the update function f, which is often the derivative function. The above framework has mutated into multiple forms. They include gradient descent approach (Klein et al. 2007; Moré and Thuente 1994) which is the closest to the original formulation. Conjugate gradient methods on the other hand have better converge rates that gradient descent. They try to exploit the knowledge from previous gradient direction to generate a new direction of descent based jointly on the previous gradient direction and the derivative (Fletcher and Reeves 1964; Polyak 1969; Hestenes and Stiefel 1952; Hager and Zhang 2006). Other similar approaches include Powell Conjugate, (Maes et al. 1997), Gauss-Newton (Ashburner and Friston 2011; Haber and Modersitzki 2007; Modersitzki 2008), Levenberg-Marquardt (L-M) (Kybic and Unser 2003; Wu et al. 2000; Gefen et al. 2003), and stochastic gradient descent.

Gauss-Newton (G-N) works by optimizing the sum of squared error term differential. This is referred to in the literature as the Jacobian, and the search direction is denoted by the following equation:

$$ g={\left({J}^T\left(\theta \right)J\left(\theta \right)\right)}^{-1}\nabla \left(\theta \right) $$
(20.4)

where J(θ) denotes the Jacobian operation, ∇(θ) denotes the delta derivative, and T is the transpose operator. The L-M technique modifies the G-N method by adding a weighting term to the Jacobian above. The modified formulation is shown by the following equation:

$$ g={\left({J}^T\left(\theta \right)J\left(\theta \right)+\zeta I\right)}^{-1}\nabla \left(\theta \right) $$
(20.5)

All the previously described approaches relay on being able to compute the gradient, which can be very demanding due to the vastness of the data source. In such case, the gradient is approximated by a stochastic version.

The second set of methods assumes the optimal parameters belong to a set of discrete values. One such approach is to use Markov random fields, which are probabilistic graphical models. Graphical models consist of graphs consisting of vertices and edges (G = {v, e}). Max-flow min-cut principle is the key formulation (Ford and Fulkerson 1956) and is the fundamental approach for graph segmentation. Alpha-expansion approach was used by Bokov (Greig et al. 1989; Boykov et al. 2001) by using extensive label check for registration. Belief propagation (Frey and MacKay 1997; Murphy et al. 1999) is another technique wherein local message exchange happens between nodes and backtracked to recover the best solution. Linear programming (Komodakis and Tziritas 2007; Komodakis et al. 2008) has also been used in literature to solve discrete optimization problems.

The last set of approaches uses hybrid or miscellaneous approaches like Greedy learning (Liu et al. 2004; Xue et al. 2004) and neural algorithms and evolutionary methods (Hansen and Ostermeier 2001). We will discuss DL methods in a separate section. These hybrid methods are heuristic or meta-heuristic in nature. Greedy methods relay on choosing the best conditional solution in each iteration without enforcing combinatorial check on the optimality of the solution. Evolutionary techniques on the other hand use genetic-based techniques to create mutation of parameters and choose the mutation that has the best survivability.

5.5 Brain Registration

Brain registration occupies a special place in the domain of registration techniques. There are multiple reasons behind it. We know well that the brain is by far the most complicated organ with millions of neural connections and pathways. It also has volumetric multidimensional network connections and functional linkages. Due to large variation between populations further complicated by deformations due to disease like Alzheimer’s disease. Several attempts have been made to generate brain template or atlas. They include Talairach atlas (Talairach and Tournoux 1988) developed from a physical brain. Digital brain atlas has been developed from physical models (Kruggel and Yves von Cramon 1999; Nowinski and Thirunavuukarasuu 2001; Roland and Zilles 1994; Roland et al. 1997) at Harvard and Montreal Neurology. To encompass diversity of intermodal and intra-modal brain variability, probabilistic atlases were developed by considering distributions of the brain landmarks and their intensity and other features. International Consortium of Brain mapping has been led in this direction (Mazziotta 2002; Mazziotta et al. 1995). Deformable brain atlas has been another direction of research wherein use of non-rigid registration has allowed the pliability of brain map between subjects. This approach is also suitable for longitudinal brain study (Thompson et al. 2000; Ganser et al. 2004; Woods 2003). The utility of this approach is however extremely dependent on the registration technique used.

5.6 Conclusion

In this review we visited the need of image registration, particularly in perspective of medical image analysis. We reviewed the different components of the registration process, namely, the correspondence problem, the transformation between source and target domains, and finally the validation of the transformation process via optimization algorithms. We reviewed the cutting-edge techniques of each of the three components mentioned above and provided their pros and cons. Figure 20.23 (Gholipur 2007) shows the flowchart of the registration process detailing the source and target images, transformation, correspondence (and interpolation) and optimization processes, and the iterative loop.

Fig. 20.23
figure 23figure 23

Flowchart detail of the image registration process (Gholipur 2007)

Full size image

We further elaborated the image registration process of the brain. Due to the uniqueness of the brain, registration process can happen between structural (anatomical) data and functional (image/data). Brain atlas has been a unique approach related to brain registration. We provided review of the cutting-edge works in this domain. Figure 20.24 (Gholipur 2007) provides comparative approaches available in this domain.

Fig. 20.24
figure 24figure 24

Varieties of brain image registration, wherein the input data can be anatomical or functional (Gholipur 2007)

Full size image

As detailed through the paper, most of the techniques in image registration require significant medical domain knowledge as well as signal processing expertise, thereby making the domain significantly challenging. A recent advancement in this domain is the use of deep learning and artificial deep networks to auto-train the registration process parameters. This is nascent and has a significant potential in future. Another aspect related to the registration process is the validation of its results. Currently, manual validation by domain experts is the norm. Such techniques require domain expertise, are laborious, and in many cases have limited availability. In the future, DL is anticipated to provide support in this direction as well.

6 Deep Learning

6.1 Introduction

Deep learning is based on artificial neural network, which is built over basic biological system of working of brain network. Deep learning is a part of machine learning in artificial intelligence (AI) that learns patterns from unsupervised/supervised data to improve the future prediction/recognition of new patterns using complex algorithms over different layers to achieve it.

Application of deep learning has become a trend in the current state of art in each and every field, for example, pattern recognition in images, speech, image and art restoration, language processing, news feed generating, and classification.

6.2 Concepts of Machine Learning, Deep Learning, and Artificial Intelligence

There is always a confusion about basic concepts between artificial intelligence (AI), machine learning, and deep learning because they are all inter-related and inter-connected. According to experts like Mr. Calum McClelland, Director of Big Data at Leverege:

  • AI – “AI involves machines that can perform tasks that are characteristic of human intelligence.”

  • ML – “Machine learning is simply a way of achieving AI.”

  • DL – “Deep learning is one of many approaches to machine learning.” (Fig. 20.25)

Fig. 20.25
figure 25figure 25

Difference between AI, ML, and DL (Image by Curt Hopkins, Managing Editor, Hewlett Packard Labs)

Full size image

Machine learning algorithm can be classified as supervised, semi-supervised, or unsupervised learning:

6.3 Deep Learning Architectures

Neural Network was inspired by the biological working of brains. Multiple machine learning algorithms are combined together at different layers of the processes for specific purpose to solve on give complex data to find useful information/prediction is the overall concepts of neural network.

A deep neural network hierarchically combines multiple layers of neurons which contain important features. This network does memorize information from the training data and makes prediction on the test data or unseen data. Hence deep learning is very popular in computer vision and medical imaging area (LeCun 2013; Razzak et al. n.d.) (Fig. 20.26).

Fig. 20.26
figure 26figure 26

Example of deep neural network (Image by M. Nielsen, http://neuralnetworksanddeeplearning.com/chap5.html)

Full size image

Some of the popular deep learning algorithms are compared in Table 20.3 such as Convolutional Neural Networks (LeCun 2013), Deep Neural Network, Deep Belief Network, Deep Autoencoder, Deep Boltzmann Machine, Recurrent Neural Network (Roell 2017), and Generative Adversarial Network.

Table 20.3 Comparison between few of the deep learning models (Lashari and Ibrahim 2013)
Full size table

6.4 Research Trends

Deep learning is trending with lot of research going on the health-care area. In terms of medical images, all the sections mentioned in the paper like classification, segmentation, registration, etc., can be done on a larger scale by using deep learning techniques. Some applications of deep learning in medical image are as mentioned below:

  1. (a)

    Disorder classification

Classification of disease is a basic requirement and needs accuracy. Table 20.4 shows few researches carried out for various parts of the body (Razzak et al. n.d.; Ker et al. 2017).

  1. (b)

    Tumor detection and segmentation

Table 20.4 List of few trending research papers in classification
Full size table

Lesion/tumor detection and segmentation have been a very important research. Deep learning algorithms are now capable of handling diseases that could be missed by doctors. This provides a double assurance for their diagnosis. A few researches have been conducted in recent times (Razzak et al. n.d.; Ker et al. 2017) (Table 20.5).

  1. (c)

    Robotics surgery (autonomous)

Table 20.5 List of few research papers in detection and segmentation
Full size table

The Da Vinci robot revolutionized surgery avenues. This device acts as robotic limbs for surgeons. Accuracy is very important in these situations for fine detail and limited spaces are prone to human error which can be reduced by machine (Faggella 2018; https://www.youtube.com/watch?v=0XdC1HUp-rU) (Fig. 20.27).

Fig. 20.27
figure 27figure 27

da Vinci surgical system (https://www.materprivate.ie/dublin/centre-services/all-services/robotic-surgery/)

Full size image

Medical instruments for tracking, detecting, and performing surgery are common across the health-care arena. Hence a medical image obtained provides opportunity for using deep learning (Table 20.6).

  1. (d)

    Virtual reality for visualization

Table 20.6 List of few research papers in robotics surgery
Full size table

Major research companies have started exploring 3D technologies such as augmented reality and virtual reality (VR) visualization of human body for best equipping the doctors, medical professionals, and medical students to prepare them to provide the most personalized services. This provides a new avenue to get detailed understanding, practice to train and education them to deal with difficult situations (Fig. 20.28).

Fig. 20.28
figure 28figure 28

3D visualization of human body (https://www.mechdyne.com/healthcare-amp-medical.aspx)

Full size image

Recent trends have been in the research to provide innovative tools and technology to cater to the VR needs using deep learning techniques. Here are few research papers and articles related to the field (Table 20.7).

Table 20.7 List of few research papers in VR for visualization
Full size table

6.5 Challenges of Deep Learning

Apart from data issue for medical images, there lie challenges of using deep learning:

  • Black-box problem – even though neural network is pretty clear, it is a huge collection or combination of machine learning algorithms for getting data, pattern recognition, building predictive models, and understanding the results. Selection of these algorithms for each dataset and problem statement varies.

  • Overfitting – there is a different training model created from the training data and testing on unseen test data. Performance of model varies. There will be lots of error that could create a bad prediction.

  • Optimization of hyperparameters – choosing the right combination of hyperparameter to configuring is a critical point to get optimal solution model. These hyperparameters vary from model to model and dataset to dataset.

  • High-performance hardware – deep learning surely requires a high-performance hardware to support the huge dataset and various algorithms are internally used. Sometimes even with high-performance system, it will take days.

6.6 Conclusion

Deep learning paves way for new avenues for doctors/medical professional to provide accurate, faster, and cheaper diagnosis, treatment, etc. Deep learning has more benefits even with tradeoff or challenges mentioned above. Future of medical images is tending toward deep learning.