1 Introduction

Measurement of cardiac output (CO) is recommended in many clinical situations of potential haemodynamic vulnerability including major surgery [1, 2]. Trending of CO allows clinicians to track changes over time and these changes can represent patient-related developments or the effects of medical interventions [3]. Hence, repeated measurements of CO provide essential information in clinical decision-making.

The reference standard for measuring CO is the pulmonary artery thermodilution [4]. For practical purposes this method is most often used in the form of continuous CO (CCO). However, use of pulmonary artery thermodilution has been widely abandoned outside cardiac surgery due to its invasive nature and complication rate [5,6,7]. Likewise, CCO is not suited for detecting changes in CO within minutes [8]. These shortcomings may be overcome by echocardiography. Echocardiographic measurement of CO can be performed by either volumetric analysis throughout the cardiac cycle or by measuring the velocity time integral (VTI) of the Doppler curve in the left ventricular outflow tract (LVOT) in combination with LVOT diameter and heart rate [9]. These methods are, however, time consuming, demand considerable echocardiographic skills and are subject to inter-observer variation. Therefore, use of repeated echocardiographic CO assessment is currently limited to research.

Automated software has been introduced for transthoracic echocardiographic measurement of CO. The software tracks spectral curves in the LVOT and gives CO immediately from traced VTIs, automatically measured heart rate and a previous calipering of the LVOT diameter. These features address the main limitations of conventional echocardiographic CO measurements, including the capacity to minimise physician driven variation, and may constitute a measuring technique that bridges the clinical gaps between information potential, patient risk and practical utility.

However, implementation of the automated software method (automated CO) requires determination of its performance in a relevant patient population. The aim of this study was to assess the precision and accuracy of CO measurements and the trending ability in post-cardiac surgery patients. Automated CO was compared with the reference standard, CCO derived from the pulmonary artery catheter (PAC), and the manual echocardiographic method (manual CO).

2 Materials and methods

This prospective, observational, single-centre study was approved by the Central Denmark Region Committee of Health Research Ethics (1-10-72-200-17) and was conducted in accordance with the Helsinki II declaration. The study was performed from December 1st 2017 to March 1st 2018 in a university hospital with approximately 600 annual on-pump cardiac surgeries in which preoperative PAC insertion is part of standard care. Inclusion criteria were elective on-pump cardiac surgery performed, extubation on the evening or night of surgery, age ≥ 18 years and a well-functioning PAC on the morning following surgery. Patients with atrial fibrillation, moderate to severe insufficiency of either mitral valve, aortic valve or tricuspid valve, intra-cardiac shunt or insufficient echocardiographic imaging on the morning following surgery were excluded. All participants provided written, informed consent. The protocol was published prior to study commencement (clinicaltrials.gov identifier: NCT03372863).

2.1 Study protocol

On the morning following surgery, patients were placed in the left-lateral position after verifying correct PAC position by pulmonary pressure waveforms. Adequate echocardiographic imaging quality of the apical 5-chamber view using the automated CO method was ensured and a 5–10 min pause ensued to achieve haemodynamic steady state as confirmed by stable CCO readings. Subsequently, patients underwent a first scan with application of the two different echocardiographic techniques for measuring CO: Automated CO and manual CO. The order of these scans was determined by means of the randomisation module in the REDCap data capture tool [10]. The first echocardiographic method applied was followed by the second echocardiographic method separated by approximately 30–60 s with the patient in the same position and performed by the same observer. CCO measurements were registered simultaneously with echocardiographic assessments, thus spanning both echocardiographic assessment periods (Fig. 1).

Fig. 1
figure 1

Outline of the study. Patients’ cardiac output were measured in two scanning sequences with three different methods: Continuous cardiac output with a pulmonary artery catheter (CCO), the automated software echocardiographic method and the manual echocardiographic method. VTI velocity time integral, LVOT left ventricular outflow tract

Full size image

A second series of CO measurements was performed 90–180 min after the first scan to evaluate the ability of the methods to track CO changes. In the interval between scans, patients had breakfast, were mobilised to a chair and received physiotherapy in accordance with standard care. Haemodynamic steady state was ensured before initiation of the second series as described.

2.2 Methods to determine cardiac output

2.2.1 Automated echocardiographic CO

The Auto VTI Tool® software is integrated in the Venue R1 ultrasound system (GE Healthcare, Horten, Norway) which was fitted with a 3Sc cardiac probe. Upon presentation of the apical 5-chamber view, a square region of interest (ROI), termed the “Auto VTI ROI”, is placed manually in the LVOT. The software automatically tracks pulsed wave spectral Doppler curves within the Auto VTI ROI and yields a colour code indicating image quality. Image quality is automatically detected and adequacy is conveyed to the operator by the Auto VTI ROI turning green. A 2.3 mm sample volume is automatically displayed within the Auto VTI ROI. The spectral Doppler curve analysis updates with short, regular intervals and a CO value is given from the preceding period. The CO is calculated from an average of the automatically tracked VTIs visible in the top-right corner of the screen, average heart rate and LVOT diameter (Fig. 2). For further technical details, please see [11].

Fig. 2
figure 2

Automated software assessment of cardiac output (CO) from the apical 5-chamber view. An “Auto VTI region of interest” (green square indicates good spectral flow quality) is placed in the left ventricular outflow tract (LVOT) and CO is calculated from the averaged tracings of velocity time integrals (VTI) (top right corner), heart rate (HR) and a previously calipered LVOT diameter

Full size image

Prior to definitive data acquisition, an insonation angle ≤ 30° between the ultrasound beam and the LVOT was verified. In case the sample volume fluctuated its positions within the LVOT, imaging was further optimised to provide the best conditions for automated sample volume positioning close to, but not into the aortic valve.

2.2.2 Manual echocardiographic CO

A Vivid S6 (GE Healthcare) with an M4Ss cardiac probe was used for manual CO following principles previously described [9]. In the apical 5-chamber view, a 5 mm pulsed-wave sample volume was placed in the left ventricular outflow tract (LVOT) close to, but not into, the aortic valve. Data from five successive flow curves were stored. Off-line analysis was performed using Echopac software (GE Healthcare). Velocity time integrals were traced manually upon correcting for the insonation angle between the ultrasound beam and the LVOT. CO was calculated from the averaged velocity time integral, heart rate and LVOT diameter. A single observer blinded to the cardiac results of the other methods completed all off-line analyses. A second observer blinded to all other data repeated all analyses for between-observer analysis of the variation attributable to off-line analyses.

The LVOT diameter was measured in the parasternal long-axis view and calipered immediately adjacent to the aortic annulus, as this point in the LVOT is well-defined even with low image quality [12]. The LVOT diameter obtained with the study’s first echocardiographic method applied was re-used for the second echocardiographic method. If an LVOT could not be measured due to inadequate image quality, post hoc correction was performed at the time of data analysis using LVOTs imaged perioperatively as a part of standard patient perioperative transoesophageal- or transthoracic echocardiography. For patients undergoing aortic valve replacement, only postoperative echocardiographies were used due to the potential effect of valve surgery on LVOT dimensions.

Three physicians, all with a minimum of 4 years’ experience in both performing and teaching point-of-care echocardiography including cardiac output measurements performed all echocardiographic scans subject to availability.

2.2.3 Continuous cardiac output

Prior to surgery, a 7.5 F thermodilution PAC was inserted and attached to a Vigilance II monitor (Edwards Lifesciences, CA, USA) for CCO measurements. The catheter was left in situ overnight and an average of five consecutive CO readings in STAT mode was used for comparisons with both echocardiographic methods (Fig. 1).

2.2.4 Within-method variation

Ten randomly selected patients were used for estimation of within-method variation. For echocardiographic methods, intra-observer and inter-observer variations were quantified. A single observer performed intra-observer analyses, scanning the same patients twice with the same echocardiographic method, separated by a pause of approximately 30–60 s. The initial LVOT diameter obtained was reused in the second scan. Two blinded observers performed inter-observer analyses in a similar way, although LVOT diameters were not reused but determined by each observer. For CCO, the averages of two consecutive sequences of 5 min-wise readings were calculated.

2.3 Statistical analyses

A sample size calculation for the agreement between the automated echocardiographic method and CCO was performed. We assumed that the agreement would be better than or equal to published data comparing manual echocardiographic CO to CCO in a comparable population showing percentage error to be 25% [13]. In order to ensure that the outer 95% confidence interval (CI) for the percentage error did not exceed 30%, often advocated as acceptable [14], 46 patients were required according to the approximate formula (outer 95% CI ~ percentage error + 1.96 × √(3 × standard deviation2/n) [15]. In order to compensate for data excluded during offline analyses, we decided to include 50 patients.

2.3.1 Between methods analyses

A mixed model analysis incorporating the effects of method, time point, patient and the interactions between patient and time (patient#time) and patient and method (patient#method) was applied to utilise the entire information of the repeated measures data set and report bias and 95% limits of agreement (LoA) for each comparison of methods [15]. CIs for LoA were estimated as described above. The mixed model adjusts for correlations between repeated measures [16]. Prior to analysis, basic assumptions for each pairwise comparison was evaluated, including normal distribution of CO differences as evaluated by Q–Q plot inspection. Pearson’s correlations with repeated measures taken into account were calculated to assess a systematic influence of CO on variation [17].

A priori, interchangeability between automated CO and the other methods was defined as a bias (accuracy) < 0.5 L/min and a precision ≤ the reference standard’s precision. Precision was calculated as 95% LoA for repeated CO measurements and divided by the mean CO. From this, the expected percentage error for method comparisons were calculated from intra-observer coefficients of variation (CV) as expected percentage error = 1.96 × √CV2expected  = 1.96 × √(CV2Method1  + CV2Method2 ) [14, 18].

2.3.2 Trending analyses

Trending ability was reported with four-quadrant plots and the corresponding concordance rate using an exclusion zone of 15% change in CO. Polar plot analysis was reported according to the initially proposed method by Crichtley et al. [19] with a 10% exclusion zone. In polar plot methodology, all data points in the four-quadrant plot are transformed into an angle (measured with respect to the unity line) and a radius (the average of the two changes). The exclusion zone criterion (10%) is then applied to the calculated radius. The polar plot methodology thus incorporates additional information on the magnitude of change rather than just the directionality of change. The transformation from the four-quadrant plot to the polar plot is well-explained and shown graphically in the original publication [19]. Due to criticism of this method put forward by Saugel et al. [20], the results of a modified polar plot analysis is reported. The primary issue with the original polar plot methodology is the radius calculation because it can lead to exclusion of observations where two measurements strongly disagree. Yet, the radius calculation in polar plot methodology is not used for reporting of polar plot statistics, it is solely used to define the exclusion zone. Therefore, we propose that the exclusion zone used in the four-quadrant plot concordance statistics also defines which data points are included in the polar plot analysis. Thus, major changes in CO with similar magnitude but with opposite directionality are not excluded in the modified polar plot analysis.

A concordance rate > 90% and a mean angular bias within ± 5° with a corresponding 95% LoA < ± 30° indicated good trending [19].

All calculations were performed in R (version 3.2.5: R studio, version 1.0.136) using packages BlandAltmanLeh for basic Bland–Altman analyses, nlme for mixed models analyses and rmcorr for repeated measure correlation analyses. A p < 0.05 was considered statistically significant.

3 Results

54 patients were eligible for inclusion in the study of which four were excluded due to insufficient image quality. Patient characteristics are given in Table 1. Data was complete except for one CCO measurement and one automated CO reading, both due to technical issues. Image quality was sufficient for direct measurement of LVOT diameter in 24 patients.

Table 1 Demographic and perioperative characteristics of patients
Full size table

Precision was 35% for automated CO and 13% for manual CO, when measurements were repeated by the same observer. Precision for CCO was 21%. For inter-observer repetitions, the precisions for the automated and manual CO methods increased to 50% and 41%, respectively.

A summary of direct comparisons of all three methods for measuring CO is given in Table 2.

Table 2 Comparison of methods for measuring cardiac output
Full size table

3.1 Continuous cardiac output versus automated echocardiographic cardiac output

There was a significant correlation between CCO and automated CO measurements (Fig. 3a). A statistically significant correlation between mean CO and bias was seen (r2 = 0.056, p = 0.02). The mean bias in the Bland–Altman analysis was 0.72 L/min (95% CI 0.42; 1.02 L/min) with 95% LoA − 1.89; 3.33 L/min (Fig. 3b). The percentage error was 46%, exceeding the expected percentage error calculated at 41%.

Fig. 3
figure 3

Correlation- and Bland–Altman plots of cardiac outputs measured simultaneously with three different methods. Individual comparisons are: the automated software echocardiographic method versus continuous cardiac output (CCO) with a pulmonary artery catheter (a, b), the automated software echocardiographic method versus the manual echocardiographic method (c, d) and the manual echocardiographic method versus CCO (e, f). In the Bland–Altman plot the seamless red- and black lines denote bias and 95% Limits of agreement and the dotted lines their respective 95% confidence intervals. The first scan was performed on the morning after surgery. The second scan was performed 90–180 min later following routine physiotherapy and mobilisation

Full size image

The concordance rate was 47% (Fig. 4a). The conventional angular bias was 0.6° (95% LoA − 57.0°; 58.2°). When using identical exclusion zones for concordance rates and polar plots, the angular bias increased to − 11.3° (95% LoA − 115.0°; 92.4°) in the modified analysis (Supplemental material Fig. 5a, b).

Fig. 4
figure 4

Four-quadrant plots visualising differences in cardiac outputs (CO) from first scan to second scan. CO was measured simultaneously with the automated software echocardiographic method (automated method), the manual echocardiographic method (manual method) and with continuous cardiac output (CCO) using a pulmonary artery catheter. Individual comparisons are given in a–c. The 15% central exclusion zones demark data points not included in concordance rate calculation. The first scan was performed on the morning after surgery; the second 90–180 min later following routine physiotherapy and mobilisation. Data points included in the standard polar plots (Supplemental material Fig. 5a, c, e) are blue or green and data points included in the modified polar plot (Supplemental material Fig. 5b, d, f) are blue or red

Full size image

3.2 Automated echocardiographic cardiac output versus manual echocardiographic cardiac output

The echocardiographic methods were positively correlated (scatter plot in Fig. 3c) with no significant correlation between the mean CO and the mean bias (r2 = 0.001, p = 0.71) (Bland–Altman plot in Fig. 3d). The mean bias was − 0.06 L/min (95% CI − 0.33; 0.21 L/min) with 95% limits of agreement (LoA) − 2.33; 2.21 L/min, corresponding to a percentage error of 42%. The expected percentage error was 38%.

The concordance rate was 79% (Fig. 4b). The conventional angular bias was 0.2° (95% LoA − 39.8°; 40.1°). In the modified analyses, the angular bias was − 8.5° (95% LoA − 89.0°; 72.0°) (Supplemental material Fig. 5c, d).

3.3 Manual echocardiographic cardiac output versus continuous cardiac output

There was a significant correlation between the manual CO and the continuous CO measurements (Fig. 3e). There was a significant correlation between the mean CO and the bias (r2 = 0.056, p = 0.02). The mean bias between methods was 0.66 L/min (95% CI 0.36; 0.95 L/min) and the corresponding 95% LoA were − 1.90; 3.22− L/min, representing a percentage error of 44% (Fig. 3f). The expected percentage error was 25%.

The concordance rate was 44% (Fig. 4c) and the angular bias was 6.8° (95% LoA − 59.1°; 72.6°). The angular bias was − 13.2° (95% LoA − 118.7°; 92.3°) when the modified analysis was applied (Supplemental material Fig. 5e, f).

The between-observer variation attributable to off-line analyses of manual CO tracings was − 0.11 L/min (95% LoA − 0.59–0.38 L/min).

4 Discussion

This study demonstrated low accuracy, different precisions, wide percentage error and poor trending ability when comparing automated echocardiographic CO to the reference standard, CCO, in extubated patients on the day following open heart surgery. Agreement and trending were substantially better when automated CO was compared to manual echocardiographic CO.

Previous studies, utilising identical transthoracic, echocardiographic principles of measuring CO and comparing it to PAC CO in patients with critical diseases including pregnancy complications [21], ICU patients [13, 22], subarachnoid haemorrhage [23], advanced systolic heart failure [24] and pulmonary hypertension [25] have yielded results of substantial heterogeneity. Most studies show acceptable accuracy as assessed by mean biases, but the calculated LoAs range from narrow [13, 21, 24] to very wide and clinically unacceptable [22, 23, 25]. Thus, results vary from a median bias of − 0.1 L/min (95% LoA − 1.3; 1.2 L/min) [13] to a mean bias of − 0.75 L/min (95% LoA − 3.43; 1.94 L/min) [23]. Only a single study [13] has addressed the trending ability of the transthoracic manual LVOT method. In ICU patients mechanically ventilated due to respiratory- or haemodynamic compromise, 26 repeated measurements of CO were compared, and a concordance rate of 94% and an angular bias of 1.0° (95% LoA − 26.8°; 28.8°) was reported; superior to the results of the present study.

The precisions of all methods, ranging from 13 to 35%, were comparable with previous studies, indicating that random variation within the same subject using the same method was low-to-moderate. Nevertheless, LoAs were wide and exceeded what can be explained by the calculated expected percentage error [18]. Hence, the variation between methods was substantial. This may, in part, be explained by the patient population investigated. Although the conditions for echocardiography were standardised and patients were extubated and cooperative, post-cardiac surgery patients represent one of the most challenging populations in terms of acoustic windows due to impaired ultrasound penetration, shallow and frequent breathing and the abundance of interfering chest wall dressings, myocardial electrodes etc. Difficult echocardiographic imaging undoubtedly contributes to random variation resulting in wider LoAs and, by the current definition of concordance, lower concordance rates. Nevertheless, all patients were screened for sufficient pulsed wave Doppler signals in the LVOT as determined by the Auto VTI ROI of automated CO turning green. Of 54 patients screened for sufficient acoustic signals, four patients failed the test and were excluded according to protocol. Suboptimal acoustic conditions facilitated precise calibration of the LVOT diameter in only 24 patients and LVOTs derived from other perioperative echocardiographies, performed as part of standard care, were used for the remaining [26] in post hoc analyses. Using LVOT diameters captured under different haemodynamic conditions may have contributed slightly to the wide LoAs between CCO and echocardiographic CO measures, but not for trending as the fixed LVOT diameter cancelled out in before–after comparisons.

As software tracks VTIs, we would expect the automated CO to have superior precision as shown with other software-driven echocardiographic analyses [27]. However, the precision for automated CO (35%) was inferior to CCO (21%) and inferior to that of manual CO (13%). We attribute this difference to the automated positioning of the sample volume within the Auto VTI ROI. According to the manufacturer, this “depends on several flow parameters and the Doppler signal strength”. In practise, the sample volume fluctuates its position frequently, often every few seconds. The current software allows for suboptimal positioning in the LVOT which, despite the operators’ efforts, may increase variation.

A priori we expected automated CO to slightly underestimate CO in comparison with manual CO, as the latter incorporated correction for any insonation angle between the ultrasound beam and the LVOT. Yet, the bias between the two echocardiographic methods was minimal and the percentage error was 42%. This percentage error is unacceptably high for interchangeability but, put together, agreement- and trending abilities were far better for the echocardiographic methods than when comparing to CCO. From a statistical point of view, manual CO was preferable to automated CO due to the superior precision. From a clinical standpoint, this point may be outweighed by the immediate measurements provided by automated CO.

CO measurements were a 1.5–3 fold less precise when another physician performed the second scan. The sources of variation included LVOT diameter, basic 2-dimensional imaging of the apical 5-chamber view, alignment of the Doppler beam with the LVOT and placement of the sample volume within the LVOT. The results testify to the pronounced observer dependence of ultrasonographic imaging and quantification and discourages repeated measurements with different observers.

We additionally report the results of a modified polar plot analysis using the exclusion zone of the concordance plot thereby including data points excluded from analysis in the originally proposed polar plot methodology [19, 20]. Critchley and colleagues must have been unaware of this pitfall–exclusion of data points with opposite directionality and consequently strong disagreement [19]. The resulting radial LoAs are substantially wider, but in our opinion provide a more correct picture of the variation for tracking changes in CO. Still, this study does not identify a clinically acceptable radial LoA for the modified method.

This study has several limitations. Patients were not subject to systematic manipulation that either increased or decreased CO with certainty but were rescanned after routine activities. Therefore, the true directionality of CO changes addressed with trending was not provided. It therefore remains unclear to which degree the individual methods for CO measurement, including CCO, contributed to the trending results. Our design exposes a weakness in the use of exclusion zones when calculating concordance rates. If two methods correctly identify a participant with a change in CO < 15%, we believe the methods are concordant, but will by default be excluded from analysis due to the use of central exclusion zones and, hence, falsely reduce the concordance rate. In addition, LVOT diameters were reused from one echocardiographic method to the other as LVOT variation with scans 1–2 min apart is likely caused by the observer and not properties intrinsic to the ultrasound scanner. However, this theoretically reduces variation between the two echocardiographic methods. There was a slight (r2 = 0.056 and r2 0.056), but statistically significant effect of mean CO on bias in both comparisons with CCO. As this was caused by one or few data points (apparent leverage problem), we chose to accept this to allow standard statistical analyses and to facilitate understanding. Finally, we used CCO as our reference standard rather than bolus thermodilution as CCO is the method of choice in clinical practice. Bolus thermodilution may have a slightly superior precision [28], but studies comparing the two have shown excellent accuracy and acceptable percentage error [29,30,31]. Furthermore, whereas we prospectively chose CCO as reference standard, it could be argued that the reference standard for testing the performance of automed CO could also be the manual CO. In that case, automated CO would be in better agreement with the reference standard with percentage error still beyond 30%, but close to the expected percentage error, and the trending ability would be moderate.

5 Conclusion

Automated CO and manual CO and CCO did not meet the criteria for interchangeability with CCO following cardiac surgery. The bias between automated CO and manual CO was low, whereas percentage errors were unacceptably high across all comparisons. Trending ability was moderate when comparing automated CO to manual CO. The observer dependence of both echocardiographic methods discourages repeated CO measurements with different observers.