Keywords

1 Introduction

Heart rate (HR) and heart rate variability (HRV) are important factors for indicating overall health and fitness. In fact, it depends on several physiological and psychological conditions and varies with the variation of this condition. Moreover, they are one of the main parameters to be monitored in the case of COVID-19 [1]. These parameters are usually measured by medically trained staff, but in some specific situations such as for people suffering from tachycardia or bradycardia, continuous monitoring is important. In fact, by defining “tachycardia” as the consistently high resting HR over 100 bpm, this disease is correlated with increased risk for cardiovascular diseases, and it can even be a sign of an underlying heart condition as explained by Dr. Bindu Chebrolu, a cardiologist at Houston Methodist [2]. Increased HR at rest may result in increased work by the heart, as well as indicating an issue with other physiological pathways. If the HR is closer to 150 bpm or higher, this may be indicative of a condition such as Supraventricular Tachycardia (SVT) requiring medical attention.

A resting heart rate below 60 bpm is considered “bradycardia”, but maybe common, particularly in individuals with good cardiovascular fitness or individuals taking certain medications. But sometimes bradycardia can be caused by damage to heart tissues from heart disease or heart attack or inflammatory diseases, such as rheumatic fever or lupus, and can be dangerous [3].

Several portable commercial devices are available in the market for continuous monitoring of HR, such as an oximeter, smartwatch, and smart bracelet. These devices are nowadays smart objects often included on the Internet of Medical Things (IoMT)-based biomedical measurement systems for healthcare monitoring [4]. However, they are adjunctive hardware to manage and carry on. Instead, a device that anyone has always had on himself/herself is the smartphone, and in [5] the authors have proposed an overview of possible applications of smartphones to measure health parameters including blood oxygenation [6] or blood pressure [7].

With the aim to stimulate scientific research, in this paper, we present an overview of the most recent work presented in the literature proposing methods to use the smartphone for measuring HR. Indeed, 83.40% of the world’s population owns a smartphone [8].

The main HR measurement can be classified into two categories as shown in (Fig. 1), which are contact and contactless measurements.

Fig. 1.
figure 1

A classification of measurement methods for heart rate evaluation by smartphone.

Full size image

Contact measurements are mainly based on the acquisition of the photoplethysmography signal (PPG) or acceleration signal. Contactless measurements are based on the acquisition of a remote PPG signal. In particular, the PPG signal is analysed by time or frequency domain methods, and an automatic classifier. Each method offers advantages and disadvantages, especially concerning measurement accuracy. The critical overview of this method is presented by highlighting the main causes of accuracy degradation and the magnitude of factors influencing the HR measurement.

The paper is organized as follows. In Sect. 2 the measurement method based on contact measurements is presented. In Sect. 3 the measurement method based on contactless measurements is analysed. Finally, some conclusions.

2 Contact-Based Measurement Methods

In contact methods acceleration signals and PPG signals are used to evaluate the HR.

Concerning the first kind of signal, Khairuddin et al. propose a methodology to measure HR and HRV by smartphone depending on the acceleration sensor embedded in a smartphone [9]. The smartphone is placed on the user’s chest near the heart and acquires the z-axis signal of the accelerometer. The user must wear light clothing and must be in a supine body position as shown in (Fig. 2).

Fig. 2.
figure 2

Measurement condition for the accelerometric method proposed in [9]

Full size image

After the acquisition, the signal needs to be filtered to remove the noise and make easier the identification of the peak (Fig. 3).

Fig. 3.
figure 3

Acceleration signal with peak value (R-wave)

Full size image

Although we preferred to use the same terminology as that in the seminal work [9], it is worth noting that in other literature works such as in [10] thesis signals are named seismocardiograms and ballistocardiograms.

The Authors use the moving mean differences method to find the peak value in the signal [11, 12]. From the signal the HR is evaluated by using Eq. (1) and the HRV by monitoring the heartbeat interval RR evaluated by Eq. (2).

$$HR=\frac{number\, of\, heartbeat \times 60}{time\, taken [s]}$$
(1)
$${RR}_{n}={t}_{{HB}_{n}}-{t}_{{HB}_{n-1}}$$
(2)

where the \({t}_{{HB}_{n}}\) is the time of the nth times of the peak of the heartbeat (R-wave).

The HRV is calculated by computing the standard deviation of RR by using Eq. (3).

$$HRV=\sqrt{\frac{1}{N}\sum_{n=0}^{N-1}{{(RR}_{n}-\overline{{RR }_{n}})}^{2}}$$
(3)

The method is experimentally tested by the Authors in 3 experiments on 3 young healthy male students. The age of the students is in the range of 22–25 years old. They compared the results from their proposed method with the result taken by electrocardiogram (ECG).

The experimental results show a maximum difference between the smartphone results and the one achieved with the ECG device equal to 1 bpm. The maximum difference between the HRV evaluated with their proposal and the ECG device is 4 ms.

Dspite these results, the number of experiments is not sufficient to provide a valid statistic. Moreover, the young age of the participants does not allow to test the proposal in the case of heart diseases, tachycardia, bradycardia, or stiffness of the cardiovascular system (typical for elderly people).

As a concern, the measurement method itself, tight or thick clothes can attenuate the accelerometric signal decreasing the (Signal to Noise Ratio) SNR and the measurement accuracy. The method can be ameliorated by a pre-analysis of the acquired signal for evaluating the SNR and checking whether it is suitable for successive steps. Further causes that can misstate the measurement and increase the uncertainty are related to signal artifacts due to movements. Techniques based on automatic classification can be applied to recognize this case and give an alarm about the reliability of the measurements.

Sukaphat et al. [13] propose a methodology to measure HR from the PPG acquired by using the smartphone camera to catch the light emitted by the (Light-emitting diode) LED flash and reflected by the blood in the fingertip. The method was previously proposed by Lamonaca and all in [14].

The fingertip is preliminarily positioned on the smartphone camera to capture the frame’s image. Successively, the colour in the captured frame is converted from YUV to RGB format and the PPG value is achieved as the mean value of the red channel. It should be noted that the proposed method considered the correct placement of the finger by using the measurement method proposed by Lamonaca et al. [15]. The HR is evaluated by analysing the PPG signal in the frequency domain, i.e., by detecting the frequency with maximum amplitude. The block scheme of the algorithm proposed in [12] is shown in (Fig. 4).

Fig. 4.
figure 4

Flowchart of the algorithm proposed in [12].

Full size image

The proposed method was tested by 10 people both male and female aged between 20 to 22 years old and with weights between (42 to 88) kg. The results of the proposed method and the results taken from the digital blood pressure were different on average by 0.57%. Furthermore, the p-value for students t-test value is 0.237 that which is greater than the chosen alpha value equal to 0.05. So, the null hypothesis that the heart rate measured by the method proposed by the Authors has the same meaning as the ones achieved by using the gold standard with a trust of 95% can be accepted.

Despite this result, it must be highlighted that the number and the typology of people used in the experiments are not sufficient to evaluate a valid statistic. They did not include old people and people with heart problems or bradycardia and tachycardia, as clearly shown in the experimental results (always in the range (60; 100) bpm, except for one case with the subject with an HR equal to 56 bpm that can be normal for a sportive healthy young subject). Moreover, they should declare how they acquire the data since the cuff for the blood pressure monitor can alter the PPG signal [14]. Finally, the synchronization of the PPG acquisition with the blood pressure monitor is not considered.

Also, Devaki et al. [16] suggested a system to measure the heart rate by using PPG using the mobile phone as a data acquisition device by using the camera and LED flash.

The methodology preliminarily records the video for the fingertip by the camera for 30 s then sends the data to MATLAB to process it. The main steps of the recorded video are:

  1. (i)

    finding the mean values of the red channel of each frame,

  2. (ii)

    collect all the values achieving the PPG signal,

  3. (iii)

    filtering the obtained signal by using bandpass 8th order Butterworth filter in the frequencies (0.8, 7.0) Hz,

  4. (iv)

    evaluating the Power Spectral Density (PSD),

  5. (v)

    find the frequency with the maximum amplitude as HR.

The Authors compared their results with the results measured by using a pulse oximeter. The p-value for students t-test value is 0.6 which shows that the readings from the pulse oximeter and smartphone are similar to those values measured for normal and hypertensive patients. Despite this result, the Authors did not give enough information about their experiments, for example, age, sex, and a more comprehensive analysis of the error they achieve, when and why they achieve the maximum error should be interesting. For example, in the case of a normal subject, an error of 8 bpm occurs for an HR equal to 60, which should represent a good case. An error equal to 23 bpm is shown for HR equal to 105, in the case of the Hypertensive subject. Finally, also in this case 10 healthy patients and 10 hypertensive patients seem to be a small sample.

Sabatini et al. also propose a methodology to measure HR and HRV by smartphone depending on the PPG [17]. With respect to the previous method, they use a second-order Butterworth high-pass filter with a cut-off frequency of 0.5 Hz. A further difference is a method for detecting the peak value of the PPG in the time domain based on the Pan-Tompkins algorithm. The HR is the average value of the peak-to-peak distances recorded in 30 s divided by 60.

To evaluate this method the Authors compared it with the results achieved by the Apple watch and used r2 statistics to achieve information about the goodness of the linear interpolation of the points having as x value the apple watch measurement and y value their method measurement results. It is interesting to note that the Authors present a study showing the influence of the colour channel on the PPG signal, and also in this case red and green channel seems more suitable with respect to the blue one concerning the SNR. Moreover, the Authors highlighted the problem introduced by movement and bad positioning of the finger on the camera. However, in this case, only 8 subjects were used with no information about their health conditions.

Ayesha et al. suggested a system to measure the HR by using PPG acquired by a smartphone camera and elaborate on this signal by using a Convolutional Neural Network (CNN) [18]. In this case, after acquiring the video of the fingertip with the smartphone camera, the average value of each frame channel is detected, and three signals are achieved. Then, the signals are filtered to increase their SNR and a detrending filter is applied to remove the stationary components. Then the signals are normalized by dividing them by their maximum absolute values. The next step is the enhancement by applying a (3x3) kernel moving average filter. Principal Component Analysis was applied to map the source signal into eigenvectors. The first eigenvector is the component with the highest variance, and therefore it was chosen as the PPG signal and a bandpass filter was used to attenuate frequencies outside the range of (0.4, 4.0) Hz that corresponds to (20, 240) bpm. This latter signal is used as input to the CNN. The network consists of 4 convolution layers with ReLU as an activation function and batch normalization and a final fully connected regression layer. The output is the HR given in bpm as shown in (Fig. 5).

Fig. 5.
figure 5

Neural Network Architecture proposed in [18].

Full size image

The network was pre-trained using a publicly available PPG-BP dataset [19], containing 657 PPG signal samples collected from 219 subjects by using a finger PPG sensor, together with HR, systolic BP and diastolic BP recorded simultaneously. Using the signal skewness metric, the poor signal samples were discarded [20]. Each sample contains 2100 data points. The sample was clipped to 1800 datapoints and segmented into 3 subparts. Each subpart became an input to the network.

The network was trained for 15 epochs with an SGD optimizer and a learning rate of 0.00001.

They were fine-tuned on a dataset consisting of 51 samples where videos of fingertips were recorded with the Redmi Note 8 smartphone camera. Data were collected from 24 participants, both male and female in the age group of 5 to 77 years. The Authors did not declare, in this case, how they measure the reference values.

The evaluation of the model by using the test data (i.e., the dataset available in [19]) gives a Mean Absolute Error (MAE) of 7.01 bpm and an error percentage of 8.3%. This value is comparable with the error achieved by other methods that do not use artificial intelligence and so can be implemented in the smartphone probably more easily.

The dataset used in this paper is more significant with respect to the ones used in the previously discussed paper both as concerning cardinality and heterogeneity. Unfortunately, a deep discussion about the reason for some significant difference between the estimated HR and the true value (more than 10 bpm in 4 cases, and the maximum difference is 21 bpm) is not given. Further data that should be reported is the clinical history of the people used to build the dataset.

3 Contactless Measurement Methods

Qiao et al. suggested a system to measure the (HR) and (HRV) by using PPG signal extracted from the video of the person’s face. This technique is contactless, and it uses the smartphone camera [21, 22].

The main idea is that part of the light is absorbed by the skin, and the rest is reflected. The intensity of the reflected signal is proportional to the blood volume flowing through the tissues. The evaluation of the blood volume over in time is the PPG. The workflow for this method is shown in (Fig. 6) it consists of five stages. The first stage is to detect the Region of Interest (ROI). It is in the face near the nose because it is full of blood vessels, and it is not affected by nose shape and eye movement. The next stage is to acquire the PPG signal correlated with each color for each frame and calculated the average, from (Fig. 6) the green color has the highest value, so the proposed method deals just with the green color. The third stage is signal processing, to remove noise due to movement and handshaking artifacts. In the fourth stage, the peak detection algorithm is applied to calculate the HRV as the temporal difference between two peak occurrences.

In the last stage, the Power Spectral Density (PSD) is applied to the signal to estimate the HR.

Fig. 6.
figure 6

Overall workflow. (a) Detect face and extract RoI. (b) Calculate the mean of separate channels in every frame and extract the raw BVP signal (g(n)). (c) Process the raw BVP signal with multiple filters to get the final BVP. (d) Detect the peaks of the BVP signal for HRV calculation. (e) Calculate the PSD of BVP in frequency domain for HR calculation [21]. In the pictures, the amplitude values come from pixel intensity. The frequency is in Hz.

Full size image

To calculate the HR, a band pass filter with cutting frequency of (2.0, 5.0) Hz is applied to the PPG then the PSD is evaluated by using Welch’s method [23], and the HR is computed by using the Eq. (4). It is evaluated in pulse per minute (ppm)

$$HR=60\times {f}_{HR}$$
(4)

The HRV is calculated by finding the interval heartbeats \({IBI}_{n}={t}_{n}-{t}_{n-1}\) where tn is the time of the nth detected peak. So HRV is calculated by finding the Root Mean Square of Successive IBI Differences [24] as shown in Eq. (5).

$$HRV=\sqrt{\frac{1}{N-1}\sum_{i=1}^{N-1}{({IBI}_{i}-{IBI}_{i+1})}^{2}}$$
(5)

where N is the number of frames.

To evaluate the method, the Authors used a publicly available dataset named “TokyoTech Remote PPG Dataset” [25] consisting of 9 people (8 male and 1 female) for each one of them 3 videos were acquired for testing three situations: relax, exercise, and relax. The video length is 1 min. The proposed method is characterized by MAE equal to 1.49 ± 2.20 bpm for HR and 24.33 ± 28.66 ms for HRV.

Although the cardinality of the testbed seems not too big, and a lack of information about the health status of people is evident, the method shows potentiality for a reliable HR evaluation, and this is important especially in pandemic times. Indeed, the contactless method reduces the risk of contact between the operator and the subject and reduces the time and costs to sanitize the smartphone at each measurement. Experimental results suggest also that possible research should be devoted to ameliorating the HRV evaluation since the variability of the error is comparable with the error itself, which is more than 20% of the measurement.

4 Conclusions

Smartphone are a pervasive tool, i.e., typically anyone has at least one of them always with him/her, also during the night. Thanks to embedded sensors, to their computational capability, and connectivity, the smartphone can be considered a powerful device able to acquire biological signals, evaluate vital signs, and transmit them in the cloud for the online monitoring of patients by clinicians or for self-monitoring.

Heart Rate (HR) and Hearth Rate Variability (HRV) are important vital signs to be monitored in sudden heart crises, stress, and in the case of COVID-19.

Since, unlike other specific medical devices, the smartphone can always be present with a person, this paper the potentiality of the smartphone as a measurement instrument to evaluate HR and HRV without using other adjunctive hardware is investigated.

The overview has highlighted both contact and contactless method to evaluate the HR and HRV. The acquired signals are the PPG acquired by using the camera and the acceleration signal acquired using the accelerometer. The processing methods is based on elaboration in the time domain and frequency domain or by using a Neural Network.

Despite all these research efforts holding a high potential, from this investigation, it was highlighted that a lack of standardization in the experimental protocol, such as how to choose the best location of the smartphone on the body, how to filter and analyze the signal, and how to report the comparison of the results with an accepted gold standard technique, could limit clinical acceptability and prevent recommending this approach as a self-tracking tool, as patient-obtained data quality could be prejudiced. A further main weakness of the research is represented by the fact that reported results are often derived from the observations of few subjects with normal HR and HRV, thus limiting their validity. Also, a lack of knowledge of the technical characteristics of the embedded sensors or of the smartphone itself could impact the evaluation of the HR and HRV measurement uncertainty.

These limitations could be a starting point for future research lines focused to define common test methodologies, gold standards, nomenclature, in order to give a common metrological basis for the comparison of the results achieved with the many methods nowadays available and transforming the common smartphone into a device able to acquire in a simple and non-invasive way, without using any additional hardware, physiological signals and reliable and validated clinical parameters.