Keywords

1 Introduction

Physiological signals deliver relevant information on the status of the human being, which helps the doctor to give a diagnosis for specifics pathologies, and therefore provide appropriate treatment. However, in many cases, these tasks become more complicated since patients can present several pathologies that must be managed simultaneously. Additionally, physiological parameters change frequently, requiring a rapid analysis, and high-risk decisions [1] that result from the interpretation of the human expert that analyses the available clinical evidence.

Recently, studies the analysis of multimodal signals, for diagnostic support using multimodal has increased [2, 3] in data fusion. This last covers the analysis of different sources and types of data. Its aims is to provide information with less uncertainty [4] and potentially allows ubiquitous and continuous monitoring of physiological parameters [5] and reduce adverse effects of the signals due to sensor movements, irregular sampling, bad connections and signal noise [6,7,8,9,10]. Data fusion can include different processes such as association, correlation, combine data, and information achieved from one or multiple sources to identify objects, situations, and threats [11].

This paper presents a literature review of the data fusion oriented to clinical diagnosis discussing and identifying their most common techniques, properties, and highlighting advantages, disadvantages, challenges, lacks, and gaps. This review was carried out from Scopus and Web of Sciences database, based on these search criteria: (i) (physiological signals) and (diagnosis decision support); and (ii) ((“data fusion”) or (“information fusion”) or (“multimodal”) and (diagnosis or diagnostic)) and (“physiological signals”). The selected papers were reported between years 2013 and 2018 in journals of quartile 1 and quartile 2 principally. Also, a data fusion framework oriented to clinical diagnostic was proposed for physiological signals processing based on the Joint Directors of Laboratories (JDL) model. The rest of the document is organized as follows: in section two, a description of the physiological signals is presented. In section three, we describe the most common multi-modal fusion models, spotlighting data processing, and fusion techniques; Section four contains the proposed architecture; and finally, the conclusions and future work are presented.

2 Physiological Signals Description

The physiological signals provide information that can be analyzed by specialists to determine with more accurate the diagnosis and treatments, besides, may be used for retrospective studies by research organizations [12]. Physiological signals are obtained through a large number of biomedical measuring devices, such as multi-parameter vital signs monitors, electroencephalograms, electrocardiograms, electromyograms, thermometers, motion sensors, oxygen saturation, glucometers, among others. These signals give a lot of information of the organs, but they have multiple problems of noise derived from internal and external causes.

Each signal or group of signals have different application for monitoring of vital signs or diagnostic such as cardiovascular diseases [13], apneic events [14], assesses the activity of back muscles in patients of (scoliosis, identify locomotion modes and measure tissue oxygenation) measure the level of anesthesia during surgery [15], eye tracking [16], non-invasive assessment of blood flow changes in muscle and bone using photoplethysmography (PPG) [17], pulmonary embolism, acute respiratory distress syndrome [18], heart valve disease [19], changes in the severity of aortic regurgitation [20], Arterial aging studies [21], Human motion disorders [22], Epilepsy [23] among others. Some signals are applied for brain–computer interfaces (BCI), which provide people suffering partial or complete motor impairments, through a non-muscular communication channel to transmission of commands to devices that allow managing an application, e.g., computerized spelling, robotic wheelchairs, robotic arms, teleoperated mobile robots, games or virtual environments [24, 25].

Different signals are analyzed for developing diagnostic support systems; an important group of them capture information synchronously or asynchronously from different human being organs. Figure 1 shows a classification of these signals as follow: (i) bioelectric signals: they are variations of biopotential versus time, e.g. Electrocardiogram (ECG), electrooculography (EOG), electromyography (EMG), electroencephalography (EEG), and electrocorticography (ECoG); (ii) Bioacoustic signals: These provide plot of recording of the sounds, e.g. phonocardiography (PCG); (iii) Biooptic signals: they correspond to measures based on detected light intensity from different tissues, flows of the body, among others, e.g. photoplethysmography (PPG); (iv) biomechanical signals: they are pressure measures mainly, e.g. blood pressure (BP), intracranial pressure (ICP), body move (BM), systolic volume (SV); (v) bioimpedance signals: correspond to electrodermal activity e.g. skin conductivity (SC) or galvanic skin response (GSR); (vi) biochemical signals: These are based on chemical components measures e.g. blood glucose (BG).

Fig. 1.
figure 1

Physiological signals classification

Full size image

ECG is widely used to understand and investigate cardiac health condition [2, 26, 27]. EOG is related to the eye movement which is derived from Cornea-Retinal Potential [28, 29]. EMG is acquired using electrodes through a muscle fiber skin to observe the muscle activity. It is also associated with the neural signals, sent from the spinal cord to muscles [30, 31]. EEG signals indicate any nervous excitement by detecting brain activities derived from neurons in the brain that communicate through electrical impulses [15, 32, 33]. ECoG records are an electrical activity of the brain by means of invasive electrodes [23, 34]. Obtaining information from bioelectric signals becomes extremely difficult due to limited data and presence of noise which significantly affects the ability to detect weak sources of interest [26, 35].

PCG acquisition is plain, non-invasive, low-cost and precise for assessing a wide range of heart disease (e.g. cardiac murmurs) [19, 36]. However, they are altered by external acoustic sources (such as speech, environmental noise, etc.) and physiological interference (such as lung sounds, cough, etc.) [37]. Respiratory rate (RR) [18], can be altered by noise and movement artifacts [38]. PPG signal consists of direct current (DC) and alternating current (AC) components. The AC component represents the changes in arterial blood volume between the systolic and diastolic phases of a cardiac cycle. The DC component corresponds to the detected light intensity from tissues, venous blood, and non-pulsatile components of arterial blood, an example of transmission type is a fingertip pulse oximeter (Spo2), which is clinically accepted and widely used. Clinical applications of PPG sensors are limited by their low signal to noise ratio (SNR), which is caused by the large volume of skin, muscle, and fat and relatively small pulsatile component of arterial blood [17, 39].

BP is defined by systolic and diastolic pressure, and it is measured in millimeters of mercury (mmHg), but main forms of noninvasive blood pressure measurement are divided into intermittent and continuous blood pressure measurements [40, 41], consecutively affecting the calculated measure of systolic volume (SV), ICP is the pressure within skull [42]; BM capture body movements [22, 43]; SC is the electrodermal activity, indicator of sympathetic activation and a useful tool for investigating psychological and physiological arousal [44, 45]; BG indicates the amount of energy in the body [43, 46]. Finally, the temperature measurement (Temp) is a measure of the ability of the body or skin to generate and release heat [3, 43]. These signals can be easily altered by movement and body mass, environmental noise, intermittent connections, etc. In Table 1 is shown a summarize of some applications of physiological signals for monomodal clinical support systems.

Table 1. Physiological signals applications
Full size table

3 Signal Fusion

Multiple information about the same phenomenon can be acquired from different types of detectors or sensors, under different conditions, in multiple experiments or subjects. Particularly multimodal fusion refers to the combination of various signals of multiple modalities to improve the performance of the systems decreasing the uncertain of their results. Each modality contributes a type of added value that cannot be deduced or obtained from only type of physiological signals [51, 52].

There are several techniques of multimodal fusion reported in the literature, like the sum and the product, which have been used for data fusion, and consecutively these operators have evolved into more advanced ones, particularly through the results of soft-computing and fuzzy operator research (Fig. 2) [53] which are widely discussed in [54] as follows: (i) Fusion of imperfect data are approaches capable of representing specific aspects of imperfect data (Probabilistic fusion, Evidential belief reasoning, fusion based on Random set theoretic fusion, Fusion and fuzzy reasoning, Possibilistic fusion, Rough set based fusion, Hybrid fusion approaches (the main idea behind development of hybrid fusion algorithms is that different fusion methods complement each other to give a more precise approach); (ii) Fusion of correlated data provide either independence or prior knowledge of the cross covariance of data to produce consistent results; (iii) Fusion of inconsistent data is the notion of data inconsistency (Spurious data, Out of sequence data, Conflicting data), and (iv) fusion of disparate data is the input data to a fusion system, which is generated by a wide variety of sensors, humans, or even stored sensory data [54]. However, categorizations most used are described in [11, 52, 55,56,57]; which consists of three types of fusion: (i) early: the characteristics obtained from different modalities are combined into a single representation before feeding the learning phase, it is known as feature fusion, and its major advantage is the detection of correlated features generated by different sensor signals so to identify a feature subset that improves recognition accuracy; In addition, the main drawback is to find the most significant feature subset, large training sets are typically required [11, 50, 58]; (ii) intermediate: it can cope with the imperfect data, along with the problems of reliability and asynchrony between different modalities, and (iii) late [59]: it is known as fusion level decision each modality is processed separately by a first recognizer, and another model is trained on the unimodal predictions to predict the actual single modal gold standard [33], main decision-level fusion advantages include communication bandwidth savings and improved decision accuracy. Another important aspect of decision fusion is the combination of the heterogeneous sensors whose measurement domains have been processed with different algorithms [11, 50, 58, 60].

Fig. 2.
figure 2

Evolution of data fusion operators [53]

Full size image

The simplest approach to multimodal analysis is to design a classifier per modality and joint the output of these classifiers combine the visual model and the text model under the assumption that they are independent, thus the probabilities are simply multiplied [61]. Nevertheless, accurate synchronization of multimodal data streams is critical to avoid parameter skews for analysis [62]. Table 2, shows a summarize advantages and disadvantages of this multimodal fusion.

Table 2. Advantages and disadvantages multimodal fusion
Full size table

In general, the main problem of multimodal data processing is that the data must be processed separately and must be combined only at the end, the dimensionality of joint feature space, different feature formats, and time-alignment. The information theory provides with a set of information measures that not only assess the amount of information that one single source of data contains, but also the amount of information that two sources of data have in common [52, 61].

In Table 3 is shown multiple studies of fusion of several physiological signals alongside the techniques applied for specific clinical diagnostic decision support with their respective accuracy (Acc). We highlighted the applications in emotion recognition, monitoring and reduce the false alarms hart diagnosis, and the applicability of ECG signals for fusing with other signals for several diagnostics.

Table 3. Multimodal fusion systems
Full size table

4 Proposed Model

Different architectures and methodologies of data fusion have been reported in [11, 60, 79, 80], based on the Joint Directors of Laboratories (JDL) model which focus on the abstraction level of the manipulated data by a fusion system. We proposed a general framework for processing and fusion of multimodal physiological signals oriented to diagnostic support systems. The architecture consists of four levels (Fig. 3), where the level 0 has for purpose make the acquisition of different physiological signals and realize the pre-processing, which consists of the stage of filtration, feature extraction, and normalization; Level 1, is composed by a spatial-temporal alignment and data correlation, the latter checks the proportionality of the information, i.e., if the information is not consistent will be feedback to the preprocessing stage, otherwise the process continues. Subsequently, the association of information executes a classification with multiple hypothesis tests, which tracks multiple targets in dense environments with the help of Bayesian networks or similar techniques, providing labels to each signal obtained from the sensors, but when the objective position is doubtful, data estimation is performed with the maximum posterior method that is based on Bayesian theory, and is used when the X parameter to be estimated is the output of a random variable with a known Pr P(X) function, consecutively the system performs an analysis verifying the status of the labels, if at any moment a different label to those assigned to the physiological parameters is identified as false alarm, it is eliminated by means of the algorithm; afterwards, sets of characteristics obtained are fused to form vectors of significant features. Consequently, level 2 has the function to determine the possible pathologies presented by the patient through learning machines; finally level 3 includes the decision level, which will determine the best hypothesis for the pathology, providing a clinical diagnosis and a possible treatment, besides this determines the assessment, risk, and impact of the process based on forecast system. All stages allow including hard and soft data, context information, together medical criteria and a mapping system based on performance quality metrics that allow optimizing the processing.

Fig. 3.
figure 3

Proposed data fusion oriented to diagnostic.

Full size image

The proposed model was developed to diminish the high rate of false alarms in services of constant monitoring, supply a timely diagnosis and a possible treatment to the pathology of the patient, providing support the specialist.

5 Conclusion

In this work were discussed multiple physiological signals alongside multimodal data fusion systems applied in clinical diagnosis support systems, highlighting advantages, disadvantages, shortcomings, and challenges. It has highlighted the capability of multimodal data fusion systems because of allowing obtaining more reliable and robust psychological or physiological information using multiple sources respect to unimodal systems, revealing an increase in the accuracy of diagnoses, and demonstrating complementarity of modalities. Additionally, multimodal data fusion yields important insights processes and structures, spatiotemporal resolution complementarity, including a comprehensive physiological view, structures, quantification, generalization and normalization [81]. Nevertheless, accurate synchronization of multimodal data streams is critical to avoid parameter skews for analysis.

For some diagnosis, the results can be considered low. Therefore, studies in this field must follow. We consider that other signals can be included in the data fusion systems and complement it with information quality evaluation systems as the proposed in [82]. In addition, we proposed a physiological signal fusion architecture, based on the JDL model; in order to provide a more reliable diagnosis and treatment based on evidence, all of the above to support the specialist in their decisions; The interface for the model will present continuous monitoring, without alterations with minimum response times, and easy to use.

Finally, to develop more effective clinical decision support mechanisms, an architecture was proposed, which covers all levels of development of diagnostic of the assistance systems in the field health taking into account the gaps found in the literature such as lack traceability of the systems from acquisition until results, visualizations, and treatments. Besides, other problems such as signals that cannot be directly merged and must be done separately, the low availability of data in the time, the high computational cost of complex models, and limitations about the assessment of situation and risk.