Introduction

Hand motion analysis is an evolving method of evaluating technical skill in medical procedures [1,2,3]. Prior studies have supported the presumption that expert physicians are more efficient and steady with their hands when performing a given procedure than trainees [2, 4,5,6]. This was done through the study of motion metrics such as the total distance the hands traveled to complete a task (path length), the number of discrete linear movements of the hand (translational movements), the number of discrete rotations of the hand (rotational movements) and the time to complete the procedure. In clinical practice, improved procedural speed can translate to reducing the operating room time, thereby decreasing the associated costs and improving overall health economics. Furthermore, longer procedure time has been associated with higher complication rates and mortality in patients undergoing bariatric surgery [7]. Motion economy has been highly correlated with the level of expertise and could be associated with clinical outcomes.

Given that experts are expected to perform a manual task faster, the question arises as to whether time alone can be used as a quantitative measure of technical proficiency in lieu of other calculated metrics of economy and smoothness of motion.

While it seems evident that a procedure can be performed quickly but in an inefficient fashion, this has not been clearly proven. This study was designed to determine if the relationship between time and hand motion metrics is always fixed, by asking trainees to perform a common interventional task (ultrasound-guided central venous access) both at their own pace and under time pressure.

This has the secondary benefit of evaluating the effects of time pressure on trainee hand motion. This common clinical scenario may lend insight into the biometric technical performance of trainees who are under real or perceived stress. Understanding these effects could tailor educational programming to match real-world expectations with respect to biometric performance in an in vivo procedure, which has not previously been a specific focus of training.

Methods and Material

Participants and Settings

With IRB (Institutional Review Board) approval, eight radiology trainees (PGY3-PGY6) were voluntarily recruited to perform a simulated central venous access on a standardized manikin (SimuLab Central Line System, Seattle, Washington), while their hand motion was recorded using an electromagnetic motion tracker (Polhemus Liberty, Colchester, Vermont). All participants had previous experience with ultrasound-guided central venous line placement (ranging from 11 to 50 procedures for juniors and 51–100 for senior trainees). Motion sensors (Teardrop Mini, Polhemus) were attached to the dorsum of the needle hand and the base of the ultrasound probe (Fig. 1). All the participants performed the procedure six times (trials). They were not told the purpose of the study other than to evaluate their hand motion in simulated central venous catheter placement. They were asked to perform the first three trials at their own pace without any prompt (control). Then, for the next three trials, they were asked to perform the same task as fast as possible (time pressure), with live feedback on the time taken displayed on the screen of a digital stopwatch for the entire trial.

Fig. 1
figure 1

Laboratory setup of the experiment: The first hand motion sensor placed on the dorsum of participant’s right hand (yellow arrow), and the second on the base of the US probe (red arrow). Note, actual recorded video was limited to the participant’s hands

Full size image

All the trials were also video-recorded. Two experienced interventional radiology attendings (7–10 years of IR practice, with previous experience in performance rating) evaluated these videos and independently scored the subject’s performance using an adapted global rating scale (attached) [8]. Raters were blinded to the identity of the participants, and this was achieved by asking all the participants to wear surgical gloves and positioning the camera such that only the subject’s hands were visible (Fig. 1).

Simulated Central Venous Access

For all the trials, a standardized manikin (SimuLab Corporation, Seattle, Washington) and a Butterfly iQ + ultrasound probe (Butterfly Network, Guilford, Connecticut) were used. The initial setup was also standardized by placing the guide wire on top of the manikin, the needle and dilator on the dominant hand side and the probe on the nondominant hand side.

Starting from a standardized position with hands on the manikin, participants were tasked with visualizing the carotid artery and internal jugular vein using the ultrasound probe. Subsequently, they were to use a syringe needle to access the jugular vein under ultrasound guidance, aspirating as they advanced. Upon successfully aspirating “blood” in the syringe, they were to detach the syringe from the needle and thread the guidewire through it. The participants were then asked to confirm its appropriate position within the jugular vein using the ultrasound probe, checking both the longitudinal and transverse axes. Following this, they were to thread a dilator through the wire until reaching the “skin” of the manikin, thereby marking the trial’s endpoint.

Motion Data Collection and Analysis

An electromagnetic motion tracking system (Polhemus Liberty, Colchester, Vermont) was used for hand motion recording with a sampling rate of 240 Hz. Raw data were exported to R 4.3.1 (R foundation for statistical computing, Vienna, Austria) and an institution-developed script was used to calculate the validated hand motion metrics as previously described [3].

Global Rating Scales

A five-item global rating scale for the assessment of central line placement was adapted from two validated global rating scales: Direct Observation of Procedural Skills (DOPS; The Foundation Program 2009) [9] and the Objective Structured Assessment of Technical Skills (OSATS; Reznick et al. 1997) [10]. Items on the original scales not applicable to simulator examination were removed [8]. All of the items were scored from 1 to 5 and a global performance score was calculated at the end.

Statistical Analysis

Data analysis was performed in R 4.3.1. The Shapiro–Wilk tests showed that all metrics were non-normally distributed so nonparametric techniques were used.

Linear mixed-effect methods were used (nlme package in R) to individually model the three metrics (path length, translational movements and rotational movements) against the time taken for the procedure (fixed effect), with the participants acting as random effects.

Controlling for time between the time pressure and control condition was performed in two different ways to ensure uniformity of the findings: (1) the ratios of motion metrics over time were calculated and compared; (2) the linear mixed-effects model was used (as done in a previous surgical study [11]), to predict the motion metric values at 60 s in both the time pressure and control conditions. This model was used to prevent pseudo-replication due to multiple observations on the same subjects.

Inter-rater reliability was assessed using the Cohen’s Kappa coefficient. Mann–Whitney U tests were used for independent samples and Wilcoxon signed-rank for paired sample. Correlation analysis was performed using Spearman’s rank correlation.

Results

Motion data were recorded for 48 trials from the 8 participants. One of the trials was excluded from the analysis because the participant did not fully execute the standardized steps (unable to secure access in the internal jugular vein). Therefore, a total of 47 trials were included in the analysis.

Time-movement correlation plots showed a high correlation for both strategies using the regression model (Fig. 2).

Fig. 2
figure 2

Scatter plots with regression lines showing the correlation between time and path length (Fig. 2A), translational movements (Fig. 2B) and rotational movements (Fig. 2C) during time pressure and control (linear mixed methods model used)

Full size image

Analysis of motion data showed that when placing the line under time pressure, the trainees completed the procedure faster (48.9 ± 19 s vs. 68.9 ± 19.7 s, p < 0.001) (Table 1). Comparing the motion metrics between time pressure and control using the first method (the ratios of motion metrics per unit time), there was a significant increase in the number of translational and rotational movements per second (p < 0.001) when performing the procedure under time pressure (Table 2).

Table 1 Comparison of the hand motion metrics between control and time pressure groups (median ± IQR)
Full size table
Table 2 Comparative table of the trainees’ performance under the different strategies using hand motion metric ratios (median ± IQR)
Full size table

When using the second method to control for time (the linear mixed-effects model), trainees move their hands 79 more centimeters, use 15 more translational movements and 18 more rotational movements at 60 s when performing the procedure quickly (Table 3).

Table 3 Predicted difference of motion metrics at 1 min (time pressure–control) using linear mixed-effects model (nlme package from R)
Full size table

The inter-rater reliability of the GRS scores between evaluators had a Kappa coefficient of 0.12 (95% CI = − 0.08, 0.33) reflecting no agreement. Wilcoxon signed-rank tests showed no difference in GRS scores between time pressure and control performance for both raters (Rater-1; p = 0.29 and Rater-2; p = 0.11).

Spearman’s rank correlation coefficient between GRS and motion metrics showed a high negative correlation for rater-1 and negligible correlation for rater-2. Using the mean GRS, it showed low to moderate negative correlation (Table 4).

Table 4 Spearman’s rank correlation coefficient test between hand motion metrics and GRS scores
Full size table

Discussion

Attending physicians perform procedures faster than trainees in addition to being more efficient with their movements, resulting in moving their hands a shorter distance with fewer total movements. This has been borne out in prior studies of hand motion analysis in medical specialties such as surgery [1, 12, 13] and, more recently, interventional radiology [3, 4]. While biomechanics can only be assessed using specialized equipment, the tight correlation between biomechanical expertise and decreased time to complete a task raises the question of whether using time alone is sufficient to determine proficiency. While there is a correlation, time alone is a blunt instrument. It is plausible to do a procedure in a fast but haphazard manner, making the distinction between time and motion relevant. This means that time alone, in theory, is not sufficient to determine technical proficiency.

Having trainees perform a simulated task routinely and then under time pressure allowed for analysis of the time-motion relationship in different circumstances. Performing the simulated procedure several times before applying time pressure has the potential to confound the results, allowing trainees to become more comfortable and proficient with the task through repetition and practice. Despite this advantage, trainees needed to move their hands a longer total distance and made more movements when controlling for procedure time. This means that biomechanically the trainees did worse when focusing on doing the procedure quickly. This helps quantitatively establish that while improved efficiency may typically correlate with time, they are distinct, and hand motion analysis provides a better picture of technical performance.

Separately, if hand motion analysis is to play a role in the future assessment of technical performance, its comparison to an observational standard assessment is useful. As was seen in the results, the agreement between two interventional radiology attending observers was poor. This speaks to the subjectivity of observation alone and reflects the real-world situation of trainees being evaluated “on the job” by the attendings who work with them. In this case, the observers were given a global rating scale that was adopted from an evaluation tool that was validated in surgery [8].

Aside from visual assessment being subjectively biased, the results could be due to having only two observers, the tool not being transferable to radiologic procedures, inadequate training of the observers on the rating questionnaire or the inherent insensitivity of observational assessment of fine motor skills compared to electromagnetic motion tracking equipment. Regardless, having several observers and specialized observer training to ensure uniformity of scoring represents a cost that should be weighed against the efficiency of developing hand motion metric standards that a computer and motion-tacking equipment can collect and process in an objective fashion.

While the task in this study was a simulated one, it bears mentioning that just asking the trainees to perform quickly and being aware of being assessed on speed resulted in a measurable decline in biomechanical performance. The decline in hand motion metrics may translate to any state of anxiety or pressure, such as being observed by an expert, the declining or critical nature of a patient’s health intraprocedurally or lack of patient cooperation with a procedure, among other factors. This could be an area of future study. Simulated practice may develop more comfort and result in a smaller degradation of performance under pressure in in vivo procedures. If so, this may help justify a simulation program in interventional radiology training curricula.

There are some limitations. This study only used one procedure with its own limited realism and translatability to other procedures or performance in real life. The time pressure was self-imposed, with each trainee applying varying levels of pressure to perform the procedure faster, which was not quantified with surveys or biometric data such as galvanic skin response or changes in vital signs. All participants performed the procedure faster under time pressure, indicating that some degree of urgency was experienced by every trainee. Additionally, the rating scales were performed on videos of the procedures rather than being present at the time they were performed, which may have allowed the observer to have different angles of the procedure. Lastly, it is hard to define the clinical benefit of better hand motion metrics—a procedure performed inefficiently could meet the same safety endpoints as one that was performed smoothly if the incidence of complications for the procedure is inherently low.

Conclusion

This study supports a limited amount of prior research, suggesting that time alone is not sufficient to determine technical proficiency and that hand motion metrics provide additional objective information about technical performance. The impact of time pressure on a trainee has the expected effect of degrading technical performance, as evinced by worsening hand motion metrics. Further research should focus on identifying the clinical impacts of different hand motion biomechanical profiles on patients’ outcomes and impact on overall health economics.