1 Introduction/background

Identifying patients at risk of deterioration in the inpatient setting as quickly as possible is important to minimize injury and promote faster recovery. It has been shown that earlier intervention results in better outcomes [1]. Equally, reduced monitoring has been demonstrated to result in in-hospital cardiac arrests, resulting in increased morbidity to patient [2].

Continuous monitoring of patients has been shown to rapidly identify deteriorating patients and improve outcomes through a more prompt response from a Medical Emergency (MET) or Rapid Response (RRT) teams, increased admission to the Intensive Care Unit, and reduced incidence of In-Hospital Cardiac Arrest [3,4,5]. Invasive ICU level monitoring, however, is significantly associated with alarms, alarm noise, and false alarms, all of which contribute to disturbed sleep and a poorer quality of recovery [6].

Improved patient monitoring on the wards or in non-traditional settings such as nursing home aims to replicate the augmented monitoring in high acuity inpatient areas such as the Intensive Care Unit, the Emergency Resuscitation Bay, and the Post-Anaesthetic Care Unit (PACU) but reduce the impact and disruption to patient care to promote quitter and more peaceful recovery. Any future continuous non-contact monitoring system must therefore have the ability to maintain privacy, be applicable across a number of different pathologies, and be readily deployable without causing a burden or additional workload in a busy inpatient unit.

Respiratory rate (RR) is a sensitive marker of patient deterioration and is generally either observed through manual counting by bedside nursing staff, or via mechanical and electronic methods, most routinely by thoracic bioimpedance via ECG leads, or via capnography. While RR has been reported as the strongest individual predictor of adverse events [7], it is often the most neglected sign and also the most inaccurately recorded [8, 9].

Infrared thermography (IRT) records heat radiation along the infrared spectrum, as opposed to recording visible light seen in traditional video. It has been postulated as a modality for contactless monitoring of patients as early as 1984 [10], with benefits ranging from the identification of phenomena not visible to the naked eye, including changes in temperature around orifices and on skin surfaces. It has the ability to maintain patient anonymity, allowing its use in healthcare settings. Its first clinical use was initially for fever screening at airports and hospital entrances during the SARS epidemic [11]. In addition to temperature monitoring, additional research into vital signs monitoring has focused on heart rate [12] and RR [13,14,15]. And while multiple studies have shown good correlation between gold standard RR and IRT derived RR [13, 14, 16,17,18,19,20], they were performed in ideal laboratory conditions with healthy volunteers. And while a single pilot study has been done on thermal video inpatient populations in the PACU showing the feasibility of monitoring patients in the hospital setting [21], these were undertaken in elective surgeries in an otherwise well population, where the range of observations did not extend into that where a patient might be classified as having respiratory compromise with no cases having a respiratory rate greater than 20, and only a single case with a respiratory rate less than 8 [9]. Secondly, this study used transthoracic impedance via ECG monitoring as a ground truth measurement of respiratory rate, does not accurately record RR in all circumstances [22, 23]. Lastly, this paper demonstrated significantly poorer correlation at rates > 15/min, limiting usefulness with potentially deteriorating patients.

The goal of IRT in patient monitoring is the deployment of a system that could potentially identify unwell and deteriorating patients, who often present with extremes of respiratory rate, and where ECG bioimpedance or manual counting remain the only modalities available for measurement. However, it is still unclear whether IRT would be able to capture a signal across a variety of pathologies in potentially deteriorating patients in a clinical setting to an equal or greater degree to the clinical standard.

This pilot study aimed to deploy existing algorithms on inpatients in a busy metropolitan ICU to:

  1. (1)

    Assess the effectiveness of IRT as a modality to assess respiratory rate in patients compared to the gold standard of manual counting with specific interest on the optimal positioning of the camera and the type of algorithm used.

  2. (2)

    Compare its accuracy to the routine clinical standard (Thoracic Bioimpedance via ECG).

  3. (3)

    Assess the degree of correlation of IRT across different respiratory frequencies including those that might be seen in a deteriorating patient.

2 Materials and methods

This was a prospective observational single centre study in the intensive care unit of a major metropolitan hospital (Box Hill Hospital, Intensive Care Unit) completed between February and June 2018. All subjects gave informed consent for before participating in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Eastern Health Research and Ethics Committee (LRR 033/2017). Patients were included if they were at least 18 years old, extubated, haemodynamically stable and able to consent or be consented for by family. Due to the fact that the nostrils would not have been visible, rending analysis impossible, patients with full face non-invasive ventilation masks were not included in this study. Any patient, however, receiving either high flow or regular nasal prong oxygen delivery, was included. Given the busy nature of the ICU and the desire to be as minimally intrusive to the usual daily workflow, patients were selected on the likelihood that measurements would not interrupt ward rounds, nursing management, or consults from other clinical teams. Additionally, patients with impending discharges were not selected. Both medical and surgical patients were included in this study, and there was no selection on the basis of the primary disease or pathology.

Prior to the study, it was decided that a clinically significant difference between observations would be 10% at a respiratory rate of 20 breaths/min. Power calculations were performed on this basis, with a standard deviation of 3, and a power of 80%. Based on these calculations, a minimum of 18 patients were needed.

Thermographic footage was captured with an Optris PI 450 infrared camera (Optris GmbH, Berlin) fitted with a standard O29 lens (29° × 22°, 18.7 mm focal length). This camera has an operating range in the long wave infrared spectrum (7.5–13 µm), a spatial resolution of 382 × 288 pixels, a temperature resolution of 0.04 °C, and a recording frame rate of 80 Hz. These settings were used for the recording of all videos. The camera was connected to a laptop running proprietary software Optris PI Connect (Release 2.18.2236.0) which was used to visualise and save the videos. Camera self-calibration was suppressed during recording.

The camera was mounted on a tripod atop a rolling trolley to be brought by the patient bedside. Recordings were taken during the patient’s stay in the ICU, typically in between ward rounds when the recording was not expected to interfere with normal ICU activity. Thermal videos of the face of the subject were taken at two different distances: (1) > 1 m from the face (far) from the end of the bed, and (2) 0.4–0.6 m from the face (near). Based on previous studies, [16] the minimum time needed to perform analysis was 30 s. To ensure adequate sampling, a minimum of 40 s was recorded, of which the first 30 s was used for analysis. The patients were free to move in their beds or chairs during recording, but were asked to refrain from talking. Patients remained clothed and covered throughout. During recording, the mode value of the respiratory rate using transthoracic impedance via the ECG electrodes was recorded from a Phillips Intellivue MX800 bedside monitor (Koninklijke Philips N.V.), with the ECG electrodes being placed in the standard 5 lead configuration. Although ECG monitoring has been previously used as the gold standard for respiratory rate monitoring, [21] this has found to be inaccurate and prone to bias [22]. Given ventilated patients were not included in this study, the gold standard was taken to be manual counting of respiratory rate separately counted by two impartial observers counting chest rise and fall movements, with the mean value rounded to the nearest whole number.

Frame-by-frame temperature data were extracted from the raw thermal video files using a modified version of a C++ module provided in the Optris PI Connect Software Development Kit. The program was compiled with Microsoft Visual Studio 2015 (Version 14.0.25431.01 Update 3). The data were imported into MATLAB (Mathworks, Natick, MA, USA Release 2017b Version 9.3.0.713579) for respiratory rate estimation using custom- designed scripts.

2.1 Description of image analysis

For each video, a rectangular region of interest (ROI) containing the nares was selected in the first analysis frame of the video (Fig. 1). For the far videos, the size of this rectangle was 21 × 11 pixels. For the near videos, it was 41 × 21 pixels.

Fig. 1
figure 1

Sample workflow of respiratory rate estimation using thermography

Full size image

Movement of this ROI over time was tracked using a Kanade-Lucas-Tomasi administered in MATLAB. The KLT feature tracker is a technique commonly used in computer vision to follow certain image features (edges, points, etc.) from one frame to the next. A time series was constructed using the mean temperature within this area. This raw signal was then detrended to remove creeping noise due to suppression of calibration, smoothed with a moving average. A window size of 11, where the size is defined as the number of raw observations used to calculate the moving average value, was used, then, filtered with a 4th order Butterworth bandpass filter (passband 0.1–0.85 Hz, equivalent to 6–51 breaths/min) (Fig. 1).

Respiratory rate was estimated from this processed signal by applying an estimator to a sliding window of 1200 frames with a step size of 80 frames (Fig. 1), resulting in a respiratory rate calculated every second based on the previous 15 s of breathing. Two rate estimators were used: (1) autocorrelation and (2) Fast Fourier Transform (FFT). For autocorrelation, respiratory rate was calculated using the lag to the first peak greater than zero. For FFT analysis, a Hanning window was applied and the resulting signal padded with zeroes (to allow spectral interpolation for a resolution of 0.25 breaths/minute) prior to applying the FFT. In both cases, the frequency band of interest was restricted to 0.1–0.85 Hz. The total signal length was limited to the first 2400 frames (30 s) for consistency, resulting in 16 estimates. The estimates were rounded to the nearest whole number, and the mode value was taken to be the thermographic respiratory rate estimate from the same videos. An explanation of the technical aspects of both techniques has been described in more detail by Barbosa et al. [16].

2.2 Statistical analysis

Agreement between two variables was estimated using both Pearson Correlation and the Bland–Altman method, in which bias was the mean difference between either: (1) thermography and chest movements, or (2) ECG and chest movements. The upper and lower limits of agreement were 1.96 standard deviations of the differences above and below the mean difference. To assess whether the bias on the Bland–Altman plot was related to the magnitude of measurements, linear regression was performed. Relationship between two variables was assessed using Pearson product-moment correlation. Statistical significance was defined as p < 0.05. Data were imported into Stata (StataCorp 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP) for statistical analysis.

3 Results

Patient characteristics are summarized in (Table 1). During the period of February to June 2018, a total of 543 patients were admitted to the Intensive Care Unit. Of these 348 met the basic inclusion criteria of being extubated and able to consent during the hours of 0900–1700 when the study took place, though not all might have been available to participate in the study at the time of data collection. This was due to routine nursing management in the ICU/ward rounds, the presence of a non-invasive ventilation mask that would have obscured the nostrils, the presence of families or other clinical teams seeing the patient, impending discharge from the unit, or having a only one IRT setup available for use at a time. In total, 27 patients admitted to ICU during the study period were enrolled. Three patients submitted a second set of observations on a subsequent day, and as a result a total of 30 datasets of thermal videos were collected. No patients dropped out following consenting and recording of images. The median age was 62.3 years (IQR 48.6–74.3). Eight patients received oxygen through standard nasal prongs. In a further 8 cases, the patients had high flow nasal cannulae. Nasogastric tubes were present in 2 cases. The remaining 12 patients did not have medical equipment in the nostrils at the time of recoding.

Table 1 Patient demographic data
Full size table

A summary of the correlation between both computer vision algorithms and thoracic impedance vs manual counting at both near and far distances can be seen in Figs. 2 and 3.

Fig. 2
figure 2

Bland–Altman plot comparing IRT derived respiratory rate and manually counted chest movements at the near distance. a Autocorrelation (AC) at 0.4–0.6 m. b Fast Fourier Transform (FFT) at 0.4–0.6 m

Full size image
Fig. 3
figure 3

Bland–Altman plot comparing IRT derived respiratory rate and manually counted chest movements at the far distance a AC at > 1.0 m. b FFT at > 1.0 m

Full size image

3.1 Comparison between ECG and chest movements

The Pearson correlation coefficient for ECG thoracic impedance and manually counted chest movements (n = 23) was R = 0.683 (p < 0.001). Using the Bland–Altman method, mean bias was − 1.348 breaths/min. Limits of agreement were − 11.96 and 9.267 breaths/min (Fig. 2b). Visual inspection of the Bland–Altman plot suggested that the bias was not constant. Using the difference in RR as a dependent outcome variable and the mean RR as an independent predictor, a regression line with a slope of − 0.59 (p < 0.001) was determined (Fig. 4).

Fig. 4
figure 4

Bland–Altman plot comparing ECG thoracic bioimpedance derived RR and manually counted chest movements

Full size image

3.2 Comparison between thermography and chest movements (near distance)

Using the autocorrelation estimator, the Pearson correlation coefficient for thermography and chest movements at the near distance was R = 0.960 (p < 0.001). Mean bias was − 0.667 breaths/min (p = 0.082). Limits of agreement were 4.631 to 3.298 breaths/min (Table 2). The Bland–Altman plot is shown in Fig. 2a. Using the FFT estimator, the correlation coefficient was R = 0.935 (p < 0.001). Mean bias was − 1.000 breaths/min. Limits of agreement were − 5.910 and 3.910 breaths/min (Table 2).

Table 2 Correlation and difference against mean, dose-dependent bias, and Pearson correlation for infrared thermography (IRT) and ECG derived RR (ECG) compared to chest movements (CM) at 0.4–0.6 m
Full size table

3.3 Comparison between thermography and chest movements (far distance)

Using the autocorrelation estimator, the Pearson correlation coefficient for thermography and chest movements at the far distance was R = 0.508 (p = 0.004). Mean bias was − 0.133 breaths/min. Limits of agreement were − 11.99 and 11.73 breaths/min. Using the FFT estimator, the correlation coefficient was R = 0.493 (p = 0.006). Mean bias was − 2.567 breaths/mi. Limits of agreement were − 15.77 and 10.63 breaths/min (Table 3).

Table 3 Correlation and difference against mean, dose-dependent bias, and Pearson correlation for infrared thermography (IRT) and ECG derived RR (ECG) compared to chest movements (CM) at > 1.0 m
Full size table

3.4 Correlation at different respiratory frequencies

Table 4 demonstrates the correlation between manually counted respiratory rate and ECG and IRT-derived rates at different respiratory frequencies typically associated with hypoventilation, normal respiration, and hyperventilation: < 12, 13–18, and > 19 breaths a minute. Over all frequencies, there was a significant relationship between the RR estimated using IRT, and via manual counting with Pearson Correlation coefficients between 0.796 and 0.943. There was no appreciable difference in the performance of either algorithm. The highest correlation occurred at rates > 19 bpm and at a distance between 0.4 and 0.6 m. (r = 0.938, p < 0.0001 using the FFT algorithm, r = 0.976, p < 0.00001 using autocorrelation). At a distance > 1.0 m, there was no significant relationship between either algorithm and the ground truth at any respiratory frequency. Correlation between counting and ECG-derived RR demonstrated significance only at > 19 bpm (r = 0.562, p = 0.029).

Table 4 IRT algorithm-derived RR and ECG bioimpedance derived RR performance during hypoventilation (< 12 bpm), normal ventilation (13–19 bpm), and hyperventilation (bpm)
Full size table

4 Discussion

This study employed a method of contactless measurement of respiratory rate that was previously only done in the experimental or well patient setting, and employed it on critically ill patients. We demonstrated that IRT-based measurements can identify respiratory rates below 12 and above 20, which are values that typically occur in deteriorating patients. Our findings suggest that IRT is not only feasible in a busy Intensive Care Unit, and provides greater accuracy and less bias than ECG based thoracic bioimpedance. We also demonstrate that these algorithms can be accurate despite the presence of devices in the nose such as nasogastric tubes or oxygen cannulae.

It has been suggested that the algorithms perform poorly at higher rates of respiration where the pattern and character differ from normal [16]. By analysing the data across frequencies and dividing these frequencies into cut-offs commonly applied to hypoventilation, normal ventilation, and hyperventilation, we have demonstrated that IRT performs equally well at all respiratory rates. ECG monitoring, on the other hand, tended to overestimate respiratory rate on the lower range and underestimate the rate on the higher range. Assessment on whether this bias is constant or changes with the magnitude of the measurements and is dose dependent was also performed via linear regression of the Bland–Altman plots in a method suggested by Ho [24], which also suggests the bias is not only greater in ECG monitoring compared with IRT, but also larger at extremes of respiratory rate in a dose-dependent fashion. Should this be the case and if ECG RR monitoring were to be used in isolation, both hypoventilating and hyperventilating patients would be wrongly classified as having a normal respiratory rate, potentially resulting in incorrectly identifying patients as clinically stable when they are in fact deteriorating.

While there was no substantive difference in the accuracy of RR measurements using two separate computer vision algorithms, there was a substantial difference in accuracy at larger distances. The camera only maintained accuracy at a distance of 40–60 cm, which would not be practical in day to day use. The region of interest in the close distance was 41 × 21 pixels, and approximately half this in the far distance. While Hochhausen et al [21] demonstrated accurate measurement at the end of the bed at a distance of greater than 60 cm, they employed a camera with a resolution of 1024 × 768 pixels. The overall pixel density in this circumstance, although not clearly listed, would likely approximate the 41 × 21 pixels used in our study. With the algorithms we employed, this pixel density likely represents the minimum resolution required for accurate RR monitoring.

While IRT in this study was performed in a busy ICU with patients with critical illness across several different pathologies, there were several features that could potentially limit its effectiveness in day to day use. Patients knew they were being recorded for the purposes of respiratory rate monitoring, and as a result may have unintentionally made their respirations more obvious than they might have if they were not being monitoring. Any future study would have to optimize the placement of the camera to allow continuous monitoring so that patients are not aware of their being monitored. It is, however, unclear whether placing a camera at a higher angle would result in a poorer view of temperature changes around the nose, or if the larger distance would preclude any meaningful measurement.

As with previous studies [21], the ROI was selected manually, with data interpretation and analysis being performed after the video had been recorded. To perform RR calculations automatically would likely involve automated body part tracking algorithms, common in RGB cameras [21] but poorly characterized with IR-based cameras. This feasibility study was also performed on only a small number of patients. A larger scale study would need to be performed in order to validate these results.

5 Conclusions/future directions

The results of this study show promise in IRT being able to accurate measure the respiratory rate of patients contactlessly and unobtrusively. Given the ECG method of measuring respiratory rate has been shown to be inaccurate in keeping with previous literature, and that routine bedside counting can also be inaccurate [22], IRT may provide a better way of monitoring respiratory rate in circumstances where capnography is not available. Future studies could incorporate greater extremes in body habitus and the presence of pathologies including sleep apnoea, and include more sensing modalities including capnography. Additionally, employing higher resolution cameras with facial and body part recognition algorithms that include chest movements and temperature oscillations around the mouth, could potentially study multiple patients with a single device. They might also establish other features of the respiratory rate not currently available with conventional measurement techniques, including the character of the respiratory rate or potentially even an approximation of tidal volume.