Introduction

Morbidity and mortality (M&M) conference is a time-honored tradition in surgery, and in many centers, still serves as the primary forum in which adverse post-operative outcomes are discussed. With rising health care costs and decreasing reimbursements, hospitals are under increased pressure to reduce the incidence of adverse events and thereby minimize the costs associated with a given procedure. Quality improvement has become more of a priority than ever before. This climate has led to a rapid increase in the number of hospitals participating in the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), which offers the opportunity to improve quality and decrease costs by tracking surgical outcomes [1]. Both M&M conference and NSQIP are limited in their ability to track outcomes, however, because M&M generally does not capture all complications [2–4] and NSQIP samples only a portion of cases that are performed at a participating institution [5].

In addition to the shift in surgical care toward a focus on outcomes and quality improvement, maintenance of certification (MOC) by the American Board of Surgery now requires participation in an outcomes registry or quality assessment program [6], and the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) requires hospitals to demonstrate how they evaluate practitioners’ procedure-specific competence. There are currently existing national programs for logging cases such as the American College of Surgeons Surgeon-Specific Registry [7], but they do not adequately track clinical outcomes, and the data fields are not well-suited to pediatric patients. NSQIP was developed to enable hospitals to collect highly reliable clinical data to provide comparisons of their outcomes with those of other participants in the program and to identify areas for quality improvement. Unlike the adult NSQIP program, however, the NSQIP-Pediatric (NSQIP-P) data is collected across surgical specialties, thereby limiting the number of cases being assessed in each specialty and for each individual surgeon.

The electronic medical record (EMR) contains useful information which, if properly extracted, can automatically generate records that can be used for maintenance of certification and to identify targets for quality improvement at an institution. We present the My Personal Outcomes Data (MyPOD) program, an EMR-based tool that tracks surgical outcomes at an institution, using a combination of electronic record extraction and several prompts for self-reported data. In order to assess validity, we compared the outcomes reported in MyPOD to those reported in NSQIP-P at our institution during the same time period.

Methods

Development of the MyPOD program and case review process

We developed a program that uses the electronic medical record to identify cases and transfer them to a platform in which the individual surgeon is prompted to enter follow-up information and record occurrences. Cases are automatically downloaded to a surgeon’s peer-protected site from EPIC Op-Time, an operating room management system, two weeks after the procedure. Downloaded information includes demographic data, procedure(s), American Society of Anesthesiologists (ASA) class, and wound class. If there is an occurrence, the surgeon records a case summary including specific occurrences as defined by the NSQIP-P program (See Fig. 1). The figure shows the screen that opens up once a case is marked by the surgeon as having an occurrence. The surgeon enters free text to describe the hospital course and also selects the complication from a drop-down menu so it will be identifiable in a database search. If a case is free of occurrences, the surgeon has the option to designate it as such or to highlight it for educational purposes so it can be discussed at the divisional M&M conference. Cases become eligible for the operating surgeon to review in MyPOD two weeks post-operatively, and surgeons are encouraged to complete case review within 30 days of the operation. The MyPOD program is accessed from a link in the EMR using a password-protected login screen. All occurrences are reviewed at the divisional M&M (twice monthly) and scored by the group. A case with an event cannot be closed until discussed. A summary of the group discussion for each case is entered by the designated divisional reviewer under “assessment notes.” (See Fig. 2). After divisional review is completed, the cases are forwarded to the Hospital Director of Surgical Outcomes and Quality for final review. Any review can be accepted “as is” or sent back to the division for further discussion. Certain occurrences can also be trended going forward in order to determine whether a more focused review is indicated in the event of a string of repeated occurrences for the division or the individual surgeon (for example, a sudden increase in the incidence of wound infections). All completed cases can be subsequently uploaded to the ACS case log system. The MyPOD data provides a complete record of case details which can be used for MOC purposes.

Fig. 1
figure 1

Screen shot for surgeon data entry on MyPOD

Full size image
Fig. 2
figure 2

Screen shot for case assessment on MyPOD

Full size image

Case selection

We reviewed all pediatric surgery cases captured in MyPOD from January 1, 2012 through June 30, 2013. All cases were performed at a single institution by one of eight pediatric surgeons. All cases scheduled through EPIC, whether in the operating room, the pediatric intensive care unit (PICU) or the neonatal intensive care unit (NICU), were included. Cases done at outside institutions were not included. Procedures performed under sedation in day medicine, in the clinic, or in the emergency department were not captured.

Description of variables and comparison with NSQIP-P data

We evaluated post-operative occurrences including readmission and reoperation reported in MyPOD over the 18-month time period. We defined occurrences according to NSQIP-P methodology [8], but we excluded post-operative transfusions. We reviewed all cases that were selected for entry into the NSQIP-P database during the same 18-month time period in order to assess the validity and capture rate of occurrences in MyPOD. Cases are selected for inclusion in NSQIP-P using an 8-day cycle-based sampling method which results in about 35 cases per cycle [9]. The NSQIP-P sampling methodology limits inclusion of low-risk, high-volume surgeries such as hernia repairs. NSQIP-P samples cases not only from general pediatric surgery, but other subspecialties as well, including neurosurgery, urology, orthopedics, otolaryngology, and plastic surgery. We limited this analysis to general pediatric surgery since this specialty was the first division participating in MyPOD.

In the subset of patients who were included in NSQIP-P and MyPOD, we evaluated concordance between the two reporting mechanisms. We compared occurrence rates reported in NSQIP and in MyPOD in using the chi square test. We performed chart review to identify characteristics of cases with missed occurrences in MyPOD. The research protocol was reviewed by the Nemours human subject research internal review board and deemed to be exempt from review.

Results

There were 77 cases with occurrences reported in MyPOD from a total of 2359 cases captured over the 18-month time period, resulting in an overall 3.3% occurrence rate. There were 691 pediatric surgery cases entered in NSQIP-P during the study period, which represents about 29% of the total number of pediatric surgery cases performed at our institution. Review of the NSQIP-P data, after excluding transfusion as an occurrence, revealed 48 cases with occurrences (6.9% occurrence rate). Overall, the occurrence, mortality, readmission, and reoperation rates reported in NSQIP-P were higher than those reported in MyPOD (Tables 1 and 2). Certain occurrences do not meet any of the NSQIP occurrence definitions and are therefore listed in the table as “other” (e.g., Clostridium difficile (C. diff.) colitis, gastrostomy tube dislodgement, prolonged ileus, recurrent hernia, persistent chest tube air leak, and other procedure-specific complications).

Table 1 Occurrence rates in MyPOD compared to NSQIP-P
Full size table
Table 2 Reoperation and readmission in MyPOD compared to NSQIP-P
Full size table

When we evaluated the subset of patients who were included in both NSQIP and MyPOD during the study period (691 patients), we found that the NSQIP occurrence rate was significantly higher than the MyPOD occurrence rate (6.9% vs 5.1%, p < 0.0001). Only 19 occurrences were reported in both NSQIP-P and MyPOD (see Table 3). In these cases, the concordance rate was 95%, with the same occurrence being reported in 18 out of 19 cases. The only discrepancy was an organ space surgical site infection (SSI) in NSQIP-P which was reported as a superficial SSI in MyPOD.

Table 3 Concordance between MyPOD and NSQIP-P (for the 691 cases captured in both)
Full size table

MyPOD reported 2 out of the 6 deaths captured in NSQIP (33% mortality reporting rate) and 15 out of the 43 other occurrences captured in NSQIP (35% morbidity reporting rate). We reviewed the charts of the patients whose complications were captured in NSQIP but not MYPOD, and we identified several themes: 1) events occurred outside the two-week window, 2) the patient was off the primary surgeon’s service when the event occurred (e.g. patient in cardiac care unit status post gastrostomy and fundoplication reintubated after the surgery team signed off), and 3) the patient was evaluated by someone else (e.g., in the ED or clinic) and the operating surgeon was never notified of the complication (this scenario was most applicable to SSI).

The deaths that were not reported in MyPOD (4 cases) were all discussed formally in divisional M&M. They occurred soon after the MyPOD program was initiated and we believe that failure to fill out the form in MyPOD was related to a technical glitch which has since been repaired.

NSQIP-P did not report a total of 16 occurrences which were entered in MyPOD (Table 3). Some of these missed occurrences did fit into NSQIP-P occurrence categories, so it is unclear why these were missed, but several were unique to the type of procedure that was done and therefore would not have been captured in NSQIP-P because there is not a specific NSQIP definition for the type of occurrence.

Discussion

We have demonstrated that an automated EMR-based program can generate case lists for individual surgeons to review and enter outcomes data, and that this process provides infrastructure for regular M&M conference and divisional review. The data generated by this program can be queried to identify trends in occurrence rates and facilitate local quality improvement efforts. It can also be used as a platform for maintenance of certification.

The comparison between MyPOD and NSQIP-P revealed that about two-thirds of morbidity and mortality cases are not being reported by the operating surgeon. This finding shows improved self-reporting compared to other publications which have shown under-reporting rates of up to 75%, but there is obviously room for improvement [3, 10]. One previous report suggested that the higher the severity of an adverse event, the greater the likelihood of it being reported. This hypothesis was not consistent with our findings, as there were 2 cardiac arrests, 4 deaths, and 5 reintubations that were not reported.

Our findings also highlight the limitations of relying solely on NSQIP-P to track outcomes. First, NSQIP-P includes only a sample of cases performed at a given institution. Second, many of the operations that are performed in pediatric surgery can result in complications that are very specific to the operation or to the patient population undergoing the operation. NSQIP-P is designed to capture a common group of occurrences that may be applicable to all operations, but it misses procedure-specific occurrences such as stricture after trachea-esophageal fistula repair or gastrostomy tube dislodgement after fundoplication. In addition to the general outcomes captured by NSQIP-P, it is important to capture and discuss operation-specific occurrences to identify opportunities for improvement in these cases.

We discovered several themes when we reviewed the charts of those patients with occurrences that were reported in NSQIP-P, but not in MyPOD. Many of the missed occurrences happened more than two weeks after the patient’s surgery. At the present time, cases become eligible to be classified as having no occurrences on post-operative day #14. The MyPOD system was designed this way to allow surgeons to access their cases close enough to the time of the operation so that occurrences would be fresh in their minds. Our findings in this study will result in the following change: surgeons will still have access to cases in MyPOD starting at post-operative day #14, but they will not be able to mark them free of occurrences until post-operative day #30. We anticipate that this change will capture more occurrences.

We also found that many SSIs were not being reported. This often happened when a different surgeon or care provider saw the patient in the ED or clinic with a wound infection, treated it, and did not notify the operating surgeon. As a result, when the operating surgeon filed the case in MyPOD, it was marked free of occurrences simply because the surgeon was not aware of the SSI. Our medical record now has an automated field that must be entered before any post-operative outpatient encounter within 90 days of the procedure can be closed, which asks: “Did this patient develop a wound infection?” We are currently developing the capacity to link this information into MyPOD so that if an infection is reported by any care provider seeing a post-operative patient, this occurrence will be captured automatically in the MyPOD system. Extraction of data from the EMR is likely to play an increasing role in enhancing not only self-reporting mechanisms like MyPOD, but also NSQIP-P data entry [11].

In order to improve self-reporting, surgeons must be well-educated about the definition of a surgical occurrence. We have designed MyPOD so that the definitions of occurrences are derived from the same platform as NSQIP-P, and surgeons who are entering outcomes data into the system have to choose from a series of dropdown menus. One of the most under-reported occurrences in our study was reintubation, possibly because surgeons were not aware that their patients were reintubated, or that they considered reintubation to be more of a critical care issue than a post-operative occurrence. Since reintubation now appears on the drop-down menu, surgeons will be more likely to report this occurrence as the MyPOD platform becomes a more familiar working environment.

At our institution, the ability to track occurrences led to several systems-based changes in clinical practice. In response to an increased rate of SSI, we developed an operating room protocol for antibiotic prophylaxis for all services. When there were a string of occurrences in NICU patients including reintubation and inappropriate transfusions, we established an interdisciplinary weekly case review with the neonatology team to discuss patient care. We are tracking reintubations in surgical patients so that these events can be discussed even after the surgical team has signed off, and we are also developing transfusion guidelines. We are working to develop a similar multidisciplinary meeting with the PICU team to track events in surgical patients. In response to an increase in the reported occurrences of deep venous thrombosis (DVT), a review of PICC lines placed in Interventional Radiology (IR) identified an increased number of DVTs following the introduction of a new type of catheter. We met with IR, reviewed the data, and agreed to discontinue use of the catheter. This has led to a decrease in the DVT rate among our patients.

Conclusion

We have shown that MyPOD successfully generates individual surgeon case logs accompanied by self-reported outcomes data. This information is required for maintenance of certification by the American Board of Surgery and may be useful for divisional/departmental leaders and hospitals for credentialing and recredentialing purposes. In addition, MyPOD has facilitated quality improvement at our institution by tracking occurrence rates. The comparison of self-reported outcomes with those captured by NSQIP-P has shown that MyPOD fails to identify many post-operative occurrences. As a result, we have already made several changes to the MyPOD system and will continue to modify it to increase the occurrence capture rate.

Our study demonstrates the limitations of relying solely upon surgeons to record their own outcomes. Built-in EMR prompts have the capacity to identify and record occurrences in such a way that the data is accurate and can be queried to track trends over time. The development of an interface between the EMR and MyPOD could allow for occurrence-specific information, in addition to baseline patient and operation details, to be transferred from one to the other. This way when a surgeon logged on to MyPOD to review a case, some occurrences would be pre-populated and the surgeon would then have the opportunity to enter further details as necessary. With these developments, the MyPOD system would more accurately reflect both NSQIP-P occurrences and procedure-specific occurrences and become a more powerful tool for targeting and tracking quality improvement in pediatric surgical care.