Introduction

The National Clinical Database (NCD) of Japan was established in April, 2010 with ten surgical subspecialty societies on the platform of the Japan Surgical Society [1]. Registrations began in 2011, and 1,172,579 cases were registered in the first year. The NCD has the records of > 95% of the surgeries performed in Japan, with more than 7,000,000 cases registered in the first 5 years.

The gastroenterological section of the NCD works in collaboration with the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), which shares a similar goal of developing a standardized surgical database for quality improvement [2]. The core members of the Japanese Society of Gastroenterological Surgery (JSGS) and the NCD attended the meetings and seminars of the ACS-NSQIP and debated about the various aspects of clinical databases, such as data collection methods and public relations [2]. Moreover, the NCD implemented the same items as those used by the ACS-NSQIP to conduct international cooperative studies. In the gastroenterological section of the NCD, risk models of mortality for the major eight procedures were created, using approximately 120,000 cases registered in 2011. Each model has been accepted and published in peer-reviewed journals [3,4,5,6,7,8,9,10]. We compared the consistency and impact of the risk factors of the three major gastroenterological surgical procedures, using web-based prospective data entry systems of the NCD in Japan and the ACS-NSQIP in USA [11]. However, the gastroenterological section of the NCD has not evaluated the reliability of the data collected, as opposed to the established data audit system of the ACS-NAQIP, according to which, audits have been conducted annually since 2005 [12]. Among the clinical data systems in the NCD, the Japan Congenital Cardiovascular Surgery Database (JCCVSD) began the data verification activities of pediatric cardiovascular surgeries only in 2008 [13]. In 2016, the Japanese Society of Gastroenterological Surgery (JSGS) conducted data verification activities using the registration data from 2014 to 2015. Here, we describe the initial activities performed for evaluating the reliability of the data collected in the gastroenterological section of the NCD.

Methods

Data source

All (about 850) hospitals that participated in the data verification activities performed by the JSGS were required to register their gastroenterological surgery cases in the NCD. Between January, 2014 and December, 2015, 1,136,700 patients underwent a collective total of 115 procedures at 4374 hospitals affiliated with the JSGS. Data were collected using specialized data collection forms that contained approximately 250 variables, including demographics, preoperative risk, operative information, postoperative complication, and outcomes. These variables are almost identical to those of the NSQIP registry: the variable items of the JSGS in the NCD were based on the definitions of the variable items of the NSQIP. Data were submitted through an internet case form and automatically checked for the selected key items. Registration was closed annually on fixed data to allow no further entries. Each participating hospital was required to assign a data manager to be accountable for data traceability. The protocol for the NCD project was approved by a suitably constituted Ethics Committee of the institution in which the work was undertaken, and checked that it conformed to the provisions of the Declaration of Helsinki (as revised in Fortaleza, Brazil, October, 2013). Each hospital was also required to obtain ethical approval from its institutional review board for the entry of all patient information into the database.

Methods of data verification

To confirm the feasibility of data verification, twenty cases were randomly selected from each hospital, to ensure that there were a sufficient number of cases for comparison. Data verification was performed for the following major eight procedures: esophagectomy, total/distal gastrectomy, right hemicolectomy, low anterior resection, hepatectomy performed for more than one segment apart from the lateral segment, pancreaticoduodenectomy, and surgery for acute diffuse peritonitis.

The test audit was conducted between February, 2016 and March, 2016. From 842 certified hospitals, 13 (1.52%) were selected for test auditing of the registration data in 2014. This audit was the first trial for the JSGS and its main purpose was to establish the evaluation criteria; therefore, all 13 hospitals were selected from the hospitals belonging to the directors of the JSGS. A total of 258 cases were randomly selected. Seventeen hospitals belonging to the councilors of the JSGS from among the 842 certified hospitals (2%) were selected between November, 2016 and March, 2017. The first established audit for data verification was performed using the evaluation criteria defined using the results of the test audit conducted during the previous year. These 17 hospitals selected for data verification were limited to high volume hospitals such as university hospitals and highly specialized hospitals. For the accuracy check, 340 cases were selected randomly from among the 17 selected hospital to ensure that each hospital was represented adequately as per the statistical requirement.

Accuracy assessment involved variables, such as patient demographics, intraoperative information, and outcomes (Table 1). We established the protocol to evaluate these variables because they matched the variables in the evaluation items of the ACS-NSQIP and influential parameters to define the quality of surgery. We also compared our concordance rate for audited variables with those of the ACS-NSQIP, using their published data [12]. Identification of the variables in the existing medical records was relatively easy, and the definitions of the terms were unambiguous because they were standardized. During each site visit, we referred to the hospital records for the discharge summary and details on surgery and anesthesia administration as the source documents and checked the consistency of the variable values with those in the registered data. If the data submitted to the NCD matched those in the source documents, we judged the items as “consistent”. If any value was inconsistent, we sought additional information to identify the cause of the discrepancy. In the test audit (258 cases from 13 institutions), items were judged as being in concordance when the data in the NCD and the source material were in complete agreement. During the first half of the test audit (five institutions), we did not record the data of the source material. To discuss the judgment of each item, during the second half of the first audit (62 cases in eight institutions), we recorded the differences between the data of the source material and the NCD in items with disagreement. To clarify the judgement criteria and revised protocol after the test audit, the database committee members in the JGCS discussed the results of the test audit. The items in complete agreement were important for the selection of quality; for example, the operator, the day of surgery, and so on. Conversely, the items for which values could change, depending on the source, were evaluated, while providing some allowance to the criteria. For example, the preoperative body weight recorded by the anesthetist and the physician could differ by several kg. During the test audit, data were verified along the protocol by three or four doctors who were members of the quality management subcommittee or the database committee in the JGSG. Two or three staff members who were independent of the JGSG, not involved in any clinical practice, and who had general medical knowledge (nurse or health information manager) performed the first established audit along the protocol under the supervision of two doctors belonging to the quality management subcommittee of the JGSG. All results of verification by staff members who were not doctors were checked by doctors. All auditors were required to sign a written contract that bound them to strictly follow the confidentiality obligations for the hospital information, and they were allowed access to the data only for the purpose of verification.

Table 1 Concordance rate for audited variables
Full size table

Statistical analyses

The accuracy of data entry was expressed as a proportion of the consistent items per verified case. We also calculated an item-wise proportion of data consistency between the source data and the NCD data. We considered some items where we could not identify the original source in the unified method as indeterminable and recorded these as a disagreement. We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value for each of the complications, transfusion, and mortality measures at discharge, then after 30 days. The time taken to verify the data for each case was compared using the medical records (paper and/or electronic), the number of auditors, and the degree of accuracy. All statistical analyses were conducted using JMP ver.8.0 software (SAS Institute Inc., Cary, NC, USA).

Results

Accuracy of data entry

All source documents for the 338 cases (99.4% of the extracted 340 cases) were identified at the time of verification. Two cases were excluded due to errors of extraction. The median concordance rate of 45 items from all 17 hospitals was 98.2% (range 96.4–99.9%). The following items displayed an item-wise data accuracy of > 95% in both audits: date of birth, sex, date of admission, date of surgery, pre- and postoperative diagnoses, date of discharge, status at discharge, history of severe chronic obstructive pulmonary disease (COPD), current dialysis, sepsis, emergency case, unplanned reoperation, nature of the tumor, location of the malignant tumor, operation date, intraoperative blood loss, operative time, readmission within 30 days after the surgery, and mortality (Tables 1, 2). The data accuracy with all the postoperative variables was > 95%. Among these variables, status 30 days after surgery and mortality exhibited almost perfect matches between the registration and the original records. The items with an accuracy < 90% in both audits were the operative member, weight, anesthesia technique, and American Society Anesthesiologists (ASA) physical status score. The majority of data inconsistencies were attributable to typing errors.

Table 2 Concordance rate for audited postoperative variables
Full size table

Accuracy statistics for mortality and complications

Measures of mortality at discharge and after 30 days had 100% agreement rates when there was information sufficient for adjudication, yielding a sensitivity of 1.00 and a specificity of 1.00 (Table 3). Among the six complications studied, organ/space surgical site infection (SSI) was the most frequent among the 336 cases undergoing data adjudication, with 26 and 27 cases recorded in the database and source materials, respectively. Among the 27 cases of organ/space SSI in the medical records, 24 were correctly recorded in the NCD database, yielding a sensitivity of 0.89. Among those 26 cases recorded in the NCD, 24 were confirmed in the medical records, yielding a PPV of 0.92. Deep incisional SSI, pneumonia, and sepsis/septic shock were relatively rare, observed in less than 10 cases either in the database or source materials. Among the 16 cases of superficial incisional SSI found in the source materials, 11 were found in the database, yielding a sensitivity of 0.88 and a specificity of 0.99. Finally, the adjudication of intra-/postoperative transfusion showed that among the 16 cases recorded in the source materials, 11 were recorded in the database, yielding a sensitivity of 0.79.

Table 3 Reliability of mortality and complication records against the medical records review
Full size table

Time spent for data verification

The mean time spent for data verification per case was 15.2 min in the first established audit (range 3–67 min; Table 4). The mean time required for verification of the paper-based and electronic records was 14.3 min and 15.3 min, respectively.

Table 4 Time spent on data verification
Full size table

Discussion

The data accuracy of the cases of gastroenterological surgery recorded in the NCD was high, with an overall concordance rate of 98.33% in our first established audit for registration data in 2015. Furthermore, the overall concordance rate for each of the 17 hospitals was above 95% and only 4 individual variables among the total 45 variables were above the 5% disagreement threshold, most of which appeared attributable to simple data entry errors. Frequent disagreements were observed between the data submitted to the NCD and the value in the source documents, with respect to body weight and ASA scores (< 80%). Despite the establishment of evaluation criteria (allowing a difference of ± 3 kg) after the test audit, there was still disagreement about body weight in the second audit (83.9%). The most probable reason for the frequent disagreement about body weight was simple error in entering the post-surgical data values. The disagreement about the ASA was in keeping with previous reports that have also shown differences in the ASA scores between the NCD, Japan and the NSQIP, USA [11]. In this report, there were more Japanese patients than USA patients with ASA class 1 (30.8% vs. 2.1%, for right hemicolectomy; 40.1% vs. 2.7%, for low anterior resection; and 30.2% vs. 0.7%, for pancreaticoduodenectomy). Among the inconsistencies in ASA scores in our first audit, 62.5% of data entered as ASA class 1 in the NCD database were recorded as class 2 on the anesthesia record. Conversely, there were few disagreements with respect to ASA classes 3 and 4 that seemed to strongly affect the occurrence of postoperative complications. The results of the audit suggest that ASA classes 3 and 4 data should be used for clinical studies based on ASA data from the NCD before 2016.

To reduce the number of disagreements, in July 2016, we published guidelines on the JSGS website, recommending that the NCD items of body weight, ASA, and anesthesia technique be input by referencing the anesthesia records. Reference materials about the ASA class were also published on this website. Using these guidelines, the frequency of disagreements about the high-disagreement items on initial audits are expected to decrease in the registration data of 2017.

The concordance rate of postoperative variables, including complications and mortality, is very high. These concordance rates are acceptable when compared with those of the NSQIP/USA in 2008, 4 years after the first audit. The reliability of the data on postoperative complications and mortality are in agreement with our previous report about the risk of postoperative complications and mortality [3,4,5,6,7,8,9,10,11]. Among the six complications studied for the first time, organ/space SSI occurred most frequently, with a sensitivity of 0.89 and a high specificity of 0.99. The sensitivities for pneumonia and intra-/postoperative transfusion were 0.70 and 0.79, respectively, although these occurrences were too infrequent for robust assessment of accuracy. According to the data validation activity for large-scale registries in ACS-NSQIP, postprocedural complications of pancreatic surgery were under-reported, with a complication rate of 30% in the registry, whereas the medical record suggested 45% [14, 15]. Although our verification results indicated that complications were reported with slightly higher sensitivity, there is still a need for improvement in reporting these complications.

The median time spent for the data verification of 45 items was 15.2 min in the first established audit. This result was acceptable compared with the median time of 7 min for data verification of 25 items in the JCCSVD audit [13]. Although the first established audit was conducted mainly by medical staff, the time spent for data verification was relatively short, possibly because the audit criteria were established during the test audit. All audits of the NSQIP were conducted remotely via online communication by a trained surgical and clinical reviewer [12]. Our established audit was also performed by two or three surgical and clinical reviewers with medical experience, under the supervision of gastrointestinal surgeons. However, in Japan, remote online auditing is still challenging, with concerns regarding the security of personal information. We plan to conduct remote audits by enabling the submission of anonymized material.

The audit results of this study have some limitations. First, we did not confirm the completeness of the registration. We plan to confirm the completeness of the NCD registration in future work by matching data from the Japanese diagnostic procedure combination/per-diem payment system. Second, the hospitals considered for selection were limited to those with high volumes and those affiliated with the JGSG, owing to the initial experience of conducting an audit for the gastroenterological section in the NCD. Third, a small number of hospitals (< 3% of the participating hospitals) were audited. In addition to data vilification, the establishment of evaluation criteria and an audit system were important objectives; therefore, we limited the number of target hospitals to a few of those affiliated with the JGSG. We plan to conduct a third audit involving at least 40 hospitals, representing 5% of the participating hospitals.

In conclusion, we verified data of the gastroenterological section of the NCD and found high accuracy of data entry. The initial success in quality assurance of the data of the JSGS session in the NCD should be strengthened by further advances in the registration protocol, continued training of data managers and auditors, and rigorous expansion of the verification activities.