A metric is a quantifiable measure used to track, compare, and assess performance or a process. Metrics are tracked to record and analyze progress toward a set goal or key performance indicator (KPI). In business, a KPI may be a revenue target or sales quota. In health care, an example of a KPI would be compliance with perioperative bundles such as for pediatric strabismus surgical repair, and within this bundle, compliance with each recommended task addresses a unique metric.1 Metrics are employed in health care to measure, report, and ultimately improve the safety and quality of patient care.2 Performance and quality metrics provide valuable insight into the effectiveness of various processes, interventions, and implementations because they provide a quantifiable means for evaluating the quality of care hospital-wide and within specific units and departments.3 In pediatric anesthesia, patient safety and quality of care metrics are yet to be widely adopted, resulting in lost opportunities to introduce focused interventions as a means to improve the quality of patient care.4

The underuse of metrics in pediatric anesthesia to date is partly explained by the general nature of existing metrics, which often fail to capture the unique requirements and nuances of pediatric anesthesia care. In many cases, such metrics are developed with a one-size-fits-all approach, often from adult perioperative medicine, which may confer some degree of standardization but do not offer the level of specificity needed for specialized pediatric anesthesia outcomes. The current widespread implementation of electronic health records (EHR) presents an opportunity for clinical departments, hospitals, and health authorities to implement quality metrics that are automatically collected and reported to health care workers with advantages of rapid review and reporting turnaround.5,6

To address such potential gains, we developed a set of metrics specifically tailored to current and near-future patient outcomes pertaining to the delivery of patient care by the Department of Anesthesia & Pain Medicine at the Hospital for Sick Children (SickKids; Toronto, ON, Canada). This technical report documents the process of developing such metrics, provides clear definitions, describes specific use cases, and presents preliminary data on their implementation and effectiveness.

Methods

This study used a modified nominal group technique (NGT) to incorporate a wide range of perspectives and expertise into the development process. The NGT typically involves direct, in-person discussions and immediate results, focusing on stakeholder perspectives through a process of idea generation and consensus. In contrast to the classical NGT that uses in-person meetings only, we employed various methods including in-person meetings, virtual meetings, e-mails, and phone calls. For this reason, we refer to this methodology as modified NGT.7 This work was performed at the Department of Anesthesia and Pain Medicine at SickKids. Development of the metrics was part of a software implementation and did not require ethical approval as per the Government of Canada’s Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (2018).8

Metrics working group for defining metrics

We approached this work by creating two working groups, which worked sequentially rather than in parallel. The first group was brought together a year before EHR implementation at a time when the hospital encouraged numerous “breakout groups” separate to the institutional hierarchy responsible for high-level planning. Our department created a quality improvement (QI) metrics working group led by the department chief, director of QI (C. M. D.) and Informatics Lead (C. M.). The working group comprised leadership from various subspecialty teams within our department and included representatives for acute pain, chronic pain, cardiovascular anesthesia, spinal/scoliosis anesthesia, postanesthesia care unit (PACU), transfusion and blood conservation, satellite and off-site, intensive care units (ICUs), medication safety, and trauma and resuscitation. Through iterative e-mail communications and in-person meetings, we encouraged all members to submit metrics that contributed to the quality of care delivered, assurance of best practices (as led by existing enhanced recovery after surgery protocols, previous QI project outcomes, care bundles within the National Surgical Quality Improvement Program, and Solutions for Patient Safety Network). Over the course of six months, this list was refined and tabulated to reflect areas of overlap (e.g., many members each requesting pain data from the PACU). This list is submitted as Electronic Supplementary Material (ESM) eAppendix. The second technical working group comprised the three leads from the working group plus analysts, experts, and programming staff from Epic (Epic Systems Corporation, Verona, WI, USA), who worked alongside hospital staff to deliver as much functionality as possible beyond day-to-day clinical activities by time of Go-Live. Multiple meetings were held throughout this six-month period to transform metric requests into Epic reports that could be collated and disseminated to members of the anesthesia department. Where possible, single metrics were brought together to create “compound metrics” that reflected compliance with suggested care alongside outcomes of care provided. Representation of all such metric data are beyond the scope of this report; however, the success of this approach was previously published in quality improvement projects describing strabismus surgery in our department.1

The completed metric set was integrated into the Epic EHR during its hospital-wide implementation in June 2018. Epic is a complex health information system composed of integrated modules, including one for anesthesia care. Its advanced level of integration ensures that patient information from any health encounter is accessible to any provider viewing that patient's record. This facilitates the creation of metrics using information from any part of the patient's record in Epic. For anesthesiologists, it enables the generation of metrics spanning the entire continuum of a patient's perioperative/perianesthesia journey, with the start and end points defined by the specific metric of interest.5,6 In addition, the working group also identified metrics for the following two categories: quality metrics (quality of induction experience, anesthesia delivery, unanticipated events, and PACU stay) and provider performance (compliance, collaboration, and skills assurance).

Agile methodology for developing metrics

We used Agile methodology to iteratively develop and validate the metrics within Epic. The Agile development methodology, frequently employed in software development, promotes a tight-knit collaboration between the end-user and the software developer. It encourages review and feedback at every stage, including development, testing, and deployment. This approach has been linked with high levels of satisfaction among end-users and has been used successfully in developing pediatric anesthesia content.9,10,11,12 Over a period of nine months, we initially met with Epic analysts weekly to identify relevant EHR data-points, to validate and clean data using automated processes, and to apply algorithmic rules to compute various metrics. Figure 1 shows how we employed both NGT and Agile methodology for each metric defined. When the QI metrics working group finalized metrics for development, these were passed onto the technical working group for development using the Agile methodology. This period was followed by a hiatus of some months while Epic analysts built the metrics. We recommenced our meetings in the last month before implementation to rereview the build. This was followed by scrutiny of early data to confirm veracity prior to provision of metrics to staff anesthesiologists. Data on metrics were validated at multiple stages, with metrics identified as requiring additional work undergoing further review and amendment. All metrics were made available as detailed reports within the Epic Anesthesia Registry to encourage standardization of care delivered and to decrease heterogeneity of practice when unpredictable or undesirable patient outcomes were identified. In recognizing the growing importance of data visualization to make complex data understandable and actionable, timely data release was transitioned to Epic-based reports and a Power BI dashboard. Power BI (Microsoft Corporation, Redmond, WA, USA) is an interactive data visualization software product developed with a primary focus on business intelligence.

Fig. 1
figure 1

Timeline and methodology for development of quality metrics

Full size image

Results

We developed 45 metrics in total for quality and provider feedback within our department and institution. Table 1 displays QI metrics developed with definitions, calculations, and the supporting cases for their development.

Table 1 Quality metrics collected
Full size table

In addition to QI metrics, provider performance metrics were reported as a demonstration of technical clinical and technical skill competence as well as compliance with departmental and organizational KPIs (Table 2). In addressing quality of care provision, we created reports that analyzed group and (anonymized) individual contributions to individual metrics (e.g., pain scores in the PACU) that also link to compliance with the perioperative bundle developed for pediatric strabismus repair.1 Our work in the strabismus population has been previously published; however, for the purposes of this report, Fig. 2 shows that prior to implementation of Epic, we struggled to maintain bundle compliance above a mean of 60%. This compliance rate achieved marginal gains in PACU pain scores; however, upon implementation of the care bundle within Epic and compounding compliance-rates to outcomes in dashboard form (Fig. 3), compliance improved significantly as shown in Fig. 2 by the improvements in both compliance and PACU pain visible and sustained from Summer 2020 on.

Table 2 Provider metrics collected
Full size table
Fig. 2
figure 2

Bundle compliance and postoperative pain scores in strabismus surgical repairs. (A) Percentage of strabismus surgeries achieving full bundle compliance; (B) incidence of moderate to severe pain post strabismus repair.

Blue line = centre line; orange lines = upper and lower control limits

Full size image
Fig. 3
figure 3

Power BI dashboard for strabismus quality metrics. This Power BI (Microsoft Corporation, Redmond, WA, USA) dashboard reports intraoperative compliance with individual units, total bundle compliance, number of cases, and incidence of moderate to severe postoperative pain.

Full size image

Key performance indicator (KPI) outcomes are presented relative to group and individual compliance within recommended bundles of care as applicable. Key performance indicators might be an internal departmental priority, such as postoperative pain, or institutional safety priorities for hospital-acquired conditions, such as surgical-site infection. Physician access to data describing the number of central venous catheterizations or placement of thoracic epidural catheters is also a viable resource for scheduling certain rooms and cases to maintain equitable access to appropriate procedures to maintain clinical technical skills.

Dashboards

We also developed a Department of Anesthesia Power BI dashboard summarizing early priority key metrics. While a more exhaustive list of metrics and reports is presented in Table 1, our initial choice of “roll-out” metrics for dashboard reporting was based on two important factors. The first factor was which of the metrics from the department’s prior “paper-based” QI program lent themselves to early, rapid transformation to electronic collection and reporting. These decisions were based on choosing metrics that reported day-to-day clinical data points fundamental to any anesthesia information management system (e.g., oxygen saturations, temperature, pain, drug administration, postoperative disposition). The second factor was which metrics best represented the safety of care provided (e.g., incidence of cardiopulmonary resuscitation, severe laryngospasm requiring urgent escalation in care), quality of care provided (e.g., pain on arrival to PACU), professional behaviour (incidence of nonreconciled controlled substance documentation, bundle compliance), and impact of clinical outcomes (unanticipated admission to ICU). Table 3 shows the key metrics included in our department’s preliminary dashboards.

Table 3 Key metrics included in preliminary Power BI dashboards
Full size table

Figure 4 shows the homepage for our prototype dashboard in Power BI. This provides important data describing departmental productivity and safety of care through visual representations of metrics against time. Figure 5 shows a dashboard for some of our statistical process control charts (SPC). These report monthly rates of emergency calls in the operating rooms and unplanned ICU admissions across a 30-month period. Statistical process control charts also show the results of projects and interventions designed to improve the rates of such outcomes. In the fall of 2020, the unplanned ICU admission chart showed monthly data points above upper control limits which suggested special-cause variation, i.e., variations to a system or process that were not previously observed, and which disrupt predictable favourable functioning of that system. Review of the cases reported, in addition to identification of similar peaks in perioperative respiratory adverse events, suggested increased airway hyperreactivity in patients following upper respiratory tract infections. Accordingly, a strict four-week symptom-free period prior to elective anesthesia was implemented (six weeks for patients with positive SARS-CoV-2 results), which resulted in a return to “normal baseline activity” for the subsequent ten months.

Fig. 4
figure 4

Sample data used in dashboard prototype development

ASA = American Society of Anesthesiologists Physical Status; CPR = cardiopulmonary resuscitation; ERaS = enhanced recovery after surgery; GA = general anesthesia; ICU = intensive care unit; PACU = postanesthesia care unit; SSE = serious safety event

Full size image
Fig. 5
figure 5

GA = general anesthetic; ICU = intensive care unit; SEM = standard error of the mean

Statistical process control charts. The top chart displays urgent calls for help in the operating room, the middle chart displays urgent calls for help in the operating room because of laryngospasm, and the bottom chart displays monthly unplanned ICU admissions. All data points are per 1,000 GAs.

Full size image

Discussion

This technical report describes the development of quality metrics in pediatric anesthesia and their implementation into our institution’s anesthesia care quality program. Defining and implementing standardized pediatric anesthesia quality metrics begins and ends with the patient but must be relevant to the individual clinician, clinical department, and institution to reflect expectations of care and uphold institutional values and strategic priorities. As EHR user interfaces continue to evolve, new opportunities arise to capture and present metrics in innovative and insightful ways through graphically rich, interactive, user-friendly dashboards. These advancements not only facilitate the process of performance reflection but also empower health care providers by rendering complex data more digestible and therefore actionable. In our department, the transition from PDF-based feedback to Power BI dashboards, and the ongoing integration into Epic dashboards, exemplifies such evolution in action.

Most institutions have introduced tools such as failure mode and effect analysis, lean methodology and Six Sigma within their quality divisions. Such measures report significant cost reductions and improvements in patient outcomes.13 The specialty of anesthesiology continues to take innovative strides in the evolution and development of health care metrics tailored to patient care. Joseph et al. detailed their web-based custom perioperative dashboard, OR Watch, which is used primarily for operational management.14 Hensley et al. described a highly evolved, interactive series of dashboards designed to enhance quality in cardiac anesthesia at Johns Hopkins Hospital.15 They reported the performance of metrics evaluating transfusion practices, intraoperative opioid use, intraoperative dexmedetomidine use, lactate values, time to extubation, and many more. These in turn fed into KPIs such as length of stay, percent survival, time spent in rehabilitation facilities, and discharge home, which in turn fed global and smart aims of improving the overall quality of cardiac anesthesia care to facilitate both the quality of the patient experience, and patient survival. While there are ongoing national and international collaborations such as the Pediatric Regional Anesthesia Network and Pediatric National Surgical Improvement Program, we are currently unaware of reports that describe local metric and dashboard development for use within EHR in pediatric anesthesiology. While we are yet to integrate the metrics and dashboards we describe to the extent employed by Johns Hopkins cardiac anesthesia, we nevertheless describe a similar ethos of vertical integration of individual patient-centred metrics as delivered in the operating room (e.g., pain management, central line insertion), as experienced during recovery (e.g., pain, postoperative nausea and vomiting, temperature), as impacting upon hospital resources (e.g., unexpected overnight admission, unanticipated ICU admission within 24 hr of anesthesia stop), and as contributing to overall hospital KPIs and reports (e.g., surgical site infections, central line-associated blood stream infection). Like Johns Hopkins, we also emphasize on the importance of rapid turnover, interactive, graphic-based data reporting as a source of self-reflection, continued learning, competence assurance, and focusing of quality improvement initiatives and projects during times of limited resources both financial and personnel-related.

Compound metrics provide a comprehensive perspective on the quality of care delivered and experienced during surgical procedures and hospital medicine admissions. Since our work with the pediatric strabismus population, we implemented a similar process of bundled care to our pediatric tonsillectomy population. Compliance rates greater than 85% with this bundle delivered a 36% decrease in the incidence of moderate to severe pain in the PACU, a 55% decrease in rescue opioid administrations, and a 44% decrease in emergence delirium, all without increasing PACU length of stay, hospital length of stay, postoperative hemorrhage, or readmission rates after discharge (unpublished data). There has been a notable move toward team-based perioperative care, and compound quality metrics provide helpful data when requesting extra resources and funding.16,17 Such data can also be used to direct community-based approaches as we emerge from lockdown restrictions and address surgical waitlist times. Our institution has the capability to collaborate with other Epic users. Recent collaborations include multisite work on COVID-19 implications for pediatric anesthesia, impact of public health protocols on school teachers and students, and international appendectomy outcomes during the pandemic.18,19,20

The work described in this technical report was successful because of the environment in which it was carried out. SickKids is a member of the Solutions for Patient Safety Network, a data-sharing network that comprises 180-plus children’s hospitals across North America. Institutional performance on KPIs such as central-line associated blood stream infections, surgical site infections, falls, unplanned extubations, etc., is reported to the network. Individual hospitals subsequently see their own performance relative to group data in raw form and SPC formats. From this, future goals are created, and network-created bundles of care can be adopted and implemented. One observes one’s progress over time and such macrodata can be shared at a department and individual level to match such seemingly simple metrics as timely antibiotic delivery in the operating room to local and network surgical-site infection rates. SickKids committed personnel and resources to this program and to Caring Safely, a patient safety initiative expressly focused on eliminating preventable patient harm. After our institutional EHR launch, the first two to three months of data collection were time-consuming while our departmental Safety & Quality Lead (C. M. D.) worked through many “bugs” in the system with Epic experts to refine the functioning of reports. At its peak, this work consumed 16–20 hr a month but with increasing familiarity and optimization of reports, it fell to approximately four hours per month, which entailed spot data checks and audits, and data-transfer from Epic to Power BI.

While this technical report provides insights into the development and implementation of tailored pediatric anesthesia metrics within our local institution, there are some limitations. This study was conducted within a single institution and the developed metrics and outcomes may not be directly applicable or transferrable to other institutions with different patient demographics, clinical practices, or EHR systems. Nevertheless, the paucity of pediatric-relevant data and quality metrics “out of the box” during our EHR implementation suggests our institution is the first to develop quality metrics to this degree of fidelity that speak to both patient and provider alike. New metrics will continue to be defined as new data sources are created by innovative surgeries and like-minded collaborators. Further studies across multiple institutions will ultimately be needed to show full potential of the metrics presented in this report; however, the work described here serves as a solid foundation for any clinical department or institution embarking on developing their own metrics and dashboards.

In conclusion, we describe the process of developing, customizing, and implementing 45 pediatric anesthesia metrics addressing quality and provider feedback within a quaternary academic pediatric institution. These metrics were developed and integrated into our hospital's EHR to facilitate real-time monitoring and reporting. The framework, methodologies, and results of this study can be translated for use in other clinical specialties and other institutions.