Abstract
Purposes
To evaluate the reliability of data collected from the gastroenterological section of the National Clinical Database of Japan (NCD), which began registrations in 2011 with ten surgical subspecialty societies.
Methods
During 2014 and 2015, 1,136,700 cases involving 115 procedures at 4374 hospitals were registered in the gastroenterological surgery section of the NCD. After a test audit using the 2014 data, 17 hospitals were selected for the first audit and data verification for 2015. The data accuracy of patient demographics, surgical outcomes, and processes was assessed using 45 items from the cases registered, in comparison with the medical records.
Results
In the first audit of the 2015 data, case registration accuracy verification involved 338 patients (99.4% of the extracted cases). The data accuracy with the maximum postoperative variables was > 95%. Accuracy of the mortality and status 30 days after the surgery was high (> 99%) with a sensitivity of 1.00 and a specificity of 1.00. Among the six complications studied, the recorded cases had high specificity but lower sensitivity (0.70–0.89).
Conclusions
We verified the data from the gastroenterological section of the NCD and found high accuracy of data entry.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
The National Clinical Database (NCD) of Japan was established in April, 2010 with ten surgical subspecialty societies on the platform of the Japan Surgical Society [1]. Registrations began in 2011, and 1,172,579 cases were registered in the first year. The NCD has the records of > 95% of the surgeries performed in Japan, with more than 7,000,000 cases registered in the first 5 years.
The gastroenterological section of the NCD works in collaboration with the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), which shares a similar goal of developing a standardized surgical database for quality improvement [2]. The core members of the Japanese Society of Gastroenterological Surgery (JSGS) and the NCD attended the meetings and seminars of the ACS-NSQIP and debated about the various aspects of clinical databases, such as data collection methods and public relations [2]. Moreover, the NCD implemented the same items as those used by the ACS-NSQIP to conduct international cooperative studies. In the gastroenterological section of the NCD, risk models of mortality for the major eight procedures were created, using approximately 120,000 cases registered in 2011. Each model has been accepted and published in peer-reviewed journals [3,4,5,6,7,8,9,10]. We compared the consistency and impact of the risk factors of the three major gastroenterological surgical procedures, using web-based prospective data entry systems of the NCD in Japan and the ACS-NSQIP in USA [11]. However, the gastroenterological section of the NCD has not evaluated the reliability of the data collected, as opposed to the established data audit system of the ACS-NAQIP, according to which, audits have been conducted annually since 2005 [12]. Among the clinical data systems in the NCD, the Japan Congenital Cardiovascular Surgery Database (JCCVSD) began the data verification activities of pediatric cardiovascular surgeries only in 2008 [13]. In 2016, the Japanese Society of Gastroenterological Surgery (JSGS) conducted data verification activities using the registration data from 2014 to 2015. Here, we describe the initial activities performed for evaluating the reliability of the data collected in the gastroenterological section of the NCD.
Methods
Data source
All (about 850) hospitals that participated in the data verification activities performed by the JSGS were required to register their gastroenterological surgery cases in the NCD. Between January, 2014 and December, 2015, 1,136,700 patients underwent a collective total of 115 procedures at 4374 hospitals affiliated with the JSGS. Data were collected using specialized data collection forms that contained approximately 250 variables, including demographics, preoperative risk, operative information, postoperative complication, and outcomes. These variables are almost identical to those of the NSQIP registry: the variable items of the JSGS in the NCD were based on the definitions of the variable items of the NSQIP. Data were submitted through an internet case form and automatically checked for the selected key items. Registration was closed annually on fixed data to allow no further entries. Each participating hospital was required to assign a data manager to be accountable for data traceability. The protocol for the NCD project was approved by a suitably constituted Ethics Committee of the institution in which the work was undertaken, and checked that it conformed to the provisions of the Declaration of Helsinki (as revised in Fortaleza, Brazil, October, 2013). Each hospital was also required to obtain ethical approval from its institutional review board for the entry of all patient information into the database.
Methods of data verification
To confirm the feasibility of data verification, twenty cases were randomly selected from each hospital, to ensure that there were a sufficient number of cases for comparison. Data verification was performed for the following major eight procedures: esophagectomy, total/distal gastrectomy, right hemicolectomy, low anterior resection, hepatectomy performed for more than one segment apart from the lateral segment, pancreaticoduodenectomy, and surgery for acute diffuse peritonitis.
The test audit was conducted between February, 2016 and March, 2016. From 842 certified hospitals, 13 (1.52%) were selected for test auditing of the registration data in 2014. This audit was the first trial for the JSGS and its main purpose was to establish the evaluation criteria; therefore, all 13 hospitals were selected from the hospitals belonging to the directors of the JSGS. A total of 258 cases were randomly selected. Seventeen hospitals belonging to the councilors of the JSGS from among the 842 certified hospitals (2%) were selected between November, 2016 and March, 2017. The first established audit for data verification was performed using the evaluation criteria defined using the results of the test audit conducted during the previous year. These 17 hospitals selected for data verification were limited to high volume hospitals such as university hospitals and highly specialized hospitals. For the accuracy check, 340 cases were selected randomly from among the 17 selected hospital to ensure that each hospital was represented adequately as per the statistical requirement.
Accuracy assessment involved variables, such as patient demographics, intraoperative information, and outcomes (Table 1). We established the protocol to evaluate these variables because they matched the variables in the evaluation items of the ACS-NSQIP and influential parameters to define the quality of surgery. We also compared our concordance rate for audited variables with those of the ACS-NSQIP, using their published data [12]. Identification of the variables in the existing medical records was relatively easy, and the definitions of the terms were unambiguous because they were standardized. During each site visit, we referred to the hospital records for the discharge summary and details on surgery and anesthesia administration as the source documents and checked the consistency of the variable values with those in the registered data. If the data submitted to the NCD matched those in the source documents, we judged the items as “consistent”. If any value was inconsistent, we sought additional information to identify the cause of the discrepancy. In the test audit (258 cases from 13 institutions), items were judged as being in concordance when the data in the NCD and the source material were in complete agreement. During the first half of the test audit (five institutions), we did not record the data of the source material. To discuss the judgment of each item, during the second half of the first audit (62 cases in eight institutions), we recorded the differences between the data of the source material and the NCD in items with disagreement. To clarify the judgement criteria and revised protocol after the test audit, the database committee members in the JGCS discussed the results of the test audit. The items in complete agreement were important for the selection of quality; for example, the operator, the day of surgery, and so on. Conversely, the items for which values could change, depending on the source, were evaluated, while providing some allowance to the criteria. For example, the preoperative body weight recorded by the anesthetist and the physician could differ by several kg. During the test audit, data were verified along the protocol by three or four doctors who were members of the quality management subcommittee or the database committee in the JGSG. Two or three staff members who were independent of the JGSG, not involved in any clinical practice, and who had general medical knowledge (nurse or health information manager) performed the first established audit along the protocol under the supervision of two doctors belonging to the quality management subcommittee of the JGSG. All results of verification by staff members who were not doctors were checked by doctors. All auditors were required to sign a written contract that bound them to strictly follow the confidentiality obligations for the hospital information, and they were allowed access to the data only for the purpose of verification.
Statistical analyses
The accuracy of data entry was expressed as a proportion of the consistent items per verified case. We also calculated an item-wise proportion of data consistency between the source data and the NCD data. We considered some items where we could not identify the original source in the unified method as indeterminable and recorded these as a disagreement. We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value for each of the complications, transfusion, and mortality measures at discharge, then after 30 days. The time taken to verify the data for each case was compared using the medical records (paper and/or electronic), the number of auditors, and the degree of accuracy. All statistical analyses were conducted using JMP ver.8.0 software (SAS Institute Inc., Cary, NC, USA).
Results
Accuracy of data entry
All source documents for the 338 cases (99.4% of the extracted 340 cases) were identified at the time of verification. Two cases were excluded due to errors of extraction. The median concordance rate of 45 items from all 17 hospitals was 98.2% (range 96.4–99.9%). The following items displayed an item-wise data accuracy of > 95% in both audits: date of birth, sex, date of admission, date of surgery, pre- and postoperative diagnoses, date of discharge, status at discharge, history of severe chronic obstructive pulmonary disease (COPD), current dialysis, sepsis, emergency case, unplanned reoperation, nature of the tumor, location of the malignant tumor, operation date, intraoperative blood loss, operative time, readmission within 30 days after the surgery, and mortality (Tables 1, 2). The data accuracy with all the postoperative variables was > 95%. Among these variables, status 30 days after surgery and mortality exhibited almost perfect matches between the registration and the original records. The items with an accuracy < 90% in both audits were the operative member, weight, anesthesia technique, and American Society Anesthesiologists (ASA) physical status score. The majority of data inconsistencies were attributable to typing errors.
Accuracy statistics for mortality and complications
Measures of mortality at discharge and after 30 days had 100% agreement rates when there was information sufficient for adjudication, yielding a sensitivity of 1.00 and a specificity of 1.00 (Table 3). Among the six complications studied, organ/space surgical site infection (SSI) was the most frequent among the 336 cases undergoing data adjudication, with 26 and 27 cases recorded in the database and source materials, respectively. Among the 27 cases of organ/space SSI in the medical records, 24 were correctly recorded in the NCD database, yielding a sensitivity of 0.89. Among those 26 cases recorded in the NCD, 24 were confirmed in the medical records, yielding a PPV of 0.92. Deep incisional SSI, pneumonia, and sepsis/septic shock were relatively rare, observed in less than 10 cases either in the database or source materials. Among the 16 cases of superficial incisional SSI found in the source materials, 11 were found in the database, yielding a sensitivity of 0.88 and a specificity of 0.99. Finally, the adjudication of intra-/postoperative transfusion showed that among the 16 cases recorded in the source materials, 11 were recorded in the database, yielding a sensitivity of 0.79.
Time spent for data verification
The mean time spent for data verification per case was 15.2 min in the first established audit (range 3–67 min; Table 4). The mean time required for verification of the paper-based and electronic records was 14.3 min and 15.3 min, respectively.
Discussion
The data accuracy of the cases of gastroenterological surgery recorded in the NCD was high, with an overall concordance rate of 98.33% in our first established audit for registration data in 2015. Furthermore, the overall concordance rate for each of the 17 hospitals was above 95% and only 4 individual variables among the total 45 variables were above the 5% disagreement threshold, most of which appeared attributable to simple data entry errors. Frequent disagreements were observed between the data submitted to the NCD and the value in the source documents, with respect to body weight and ASA scores (< 80%). Despite the establishment of evaluation criteria (allowing a difference of ± 3 kg) after the test audit, there was still disagreement about body weight in the second audit (83.9%). The most probable reason for the frequent disagreement about body weight was simple error in entering the post-surgical data values. The disagreement about the ASA was in keeping with previous reports that have also shown differences in the ASA scores between the NCD, Japan and the NSQIP, USA [11]. In this report, there were more Japanese patients than USA patients with ASA class 1 (30.8% vs. 2.1%, for right hemicolectomy; 40.1% vs. 2.7%, for low anterior resection; and 30.2% vs. 0.7%, for pancreaticoduodenectomy). Among the inconsistencies in ASA scores in our first audit, 62.5% of data entered as ASA class 1 in the NCD database were recorded as class 2 on the anesthesia record. Conversely, there were few disagreements with respect to ASA classes 3 and 4 that seemed to strongly affect the occurrence of postoperative complications. The results of the audit suggest that ASA classes 3 and 4 data should be used for clinical studies based on ASA data from the NCD before 2016.
To reduce the number of disagreements, in July 2016, we published guidelines on the JSGS website, recommending that the NCD items of body weight, ASA, and anesthesia technique be input by referencing the anesthesia records. Reference materials about the ASA class were also published on this website. Using these guidelines, the frequency of disagreements about the high-disagreement items on initial audits are expected to decrease in the registration data of 2017.
The concordance rate of postoperative variables, including complications and mortality, is very high. These concordance rates are acceptable when compared with those of the NSQIP/USA in 2008, 4 years after the first audit. The reliability of the data on postoperative complications and mortality are in agreement with our previous report about the risk of postoperative complications and mortality [3,4,5,6,7,8,9,10,11]. Among the six complications studied for the first time, organ/space SSI occurred most frequently, with a sensitivity of 0.89 and a high specificity of 0.99. The sensitivities for pneumonia and intra-/postoperative transfusion were 0.70 and 0.79, respectively, although these occurrences were too infrequent for robust assessment of accuracy. According to the data validation activity for large-scale registries in ACS-NSQIP, postprocedural complications of pancreatic surgery were under-reported, with a complication rate of 30% in the registry, whereas the medical record suggested 45% [14, 15]. Although our verification results indicated that complications were reported with slightly higher sensitivity, there is still a need for improvement in reporting these complications.
The median time spent for the data verification of 45 items was 15.2 min in the first established audit. This result was acceptable compared with the median time of 7 min for data verification of 25 items in the JCCSVD audit [13]. Although the first established audit was conducted mainly by medical staff, the time spent for data verification was relatively short, possibly because the audit criteria were established during the test audit. All audits of the NSQIP were conducted remotely via online communication by a trained surgical and clinical reviewer [12]. Our established audit was also performed by two or three surgical and clinical reviewers with medical experience, under the supervision of gastrointestinal surgeons. However, in Japan, remote online auditing is still challenging, with concerns regarding the security of personal information. We plan to conduct remote audits by enabling the submission of anonymized material.
The audit results of this study have some limitations. First, we did not confirm the completeness of the registration. We plan to confirm the completeness of the NCD registration in future work by matching data from the Japanese diagnostic procedure combination/per-diem payment system. Second, the hospitals considered for selection were limited to those with high volumes and those affiliated with the JGSG, owing to the initial experience of conducting an audit for the gastroenterological section in the NCD. Third, a small number of hospitals (< 3% of the participating hospitals) were audited. In addition to data vilification, the establishment of evaluation criteria and an audit system were important objectives; therefore, we limited the number of target hospitals to a few of those affiliated with the JGSG. We plan to conduct a third audit involving at least 40 hospitals, representing 5% of the participating hospitals.
In conclusion, we verified data of the gastroenterological section of the NCD and found high accuracy of data entry. The initial success in quality assurance of the data of the JSGS session in the NCD should be strengthened by further advances in the registration protocol, continued training of data managers and auditors, and rigorous expansion of the verification activities.
References
Gotoh M, Miyata H, Hashimoto H, Wakabayashi G, Konno H, Miyakawa S, et al. National Clinical Database feedback implementation for quality improvement of cancer treatment in Japan: from good to great through transparency. Surg Today. 2016;46:38–47.
Miyata H, Gotoh M, Hashimoto H, Motomura N, Murakami A, Tomotaki A, et al. Challenges and prospects of a clinical database linked to the board certification system. Surg Today. 2014;44:1991–9.
Watanabe M, Miyata H, Gotoh M, Baba H, Kimura W, Tomita N, et al. Total gastrectomy risk model: data from 20,011 Japanese patients in a nationwide internet-based database. Ann Surg. 2014;260:1034–9.
Kimura W, Miyata H, Gotoh M, Hirai I, Kenjo A, Kitagawa Y, et al. A pancreaticoduodenectomy risk model derived from 8575 cases from a national single-race population (Japanese) using a web-based data entry system: the 30-day and in-hospital mortality rates for pancreaticoduodenectomy. Ann Surg. 2014;259:773–80.
Kobayashi H, Miyata H, Gotoh M, Baba H, Kimura W, Kitagawa Y, et al. Risk model for right hemicolectomy based on 19,070 Japanese patients in the National Clinical Database. J Gastroenterol. 2014;49:1047–55.
Kenjo A, Miyata H, Gotoh M, Kitagawa Y, Shimada M, Baba H, et al. Risk stratification of 7732 hepatectomy cases in 2011 from the National Clinical Database for Japan. J Am Coll Surg. 2014;218:412–22.
Takeuchi H, Miyata H, Gotoh M, Kitagawa Y, Baba H, Kimura W, et al. A risk model for esophagectomy using data of 5354 patients included in a Japanese nationwide web-based database. Ann Surg. 2014;260:259–66.
Matsubara N, Miyata H, Gotoh M, Tomita N, Baba H, Kimura W, et al. Mortality after common rectal surgery in Japan: a study on low anterior resection from a newly established nationwide large-scale clinical database. Dis Colon Rectum. 2014;57:1075–81.
Nakagoe T, Miyata H, Gotoh M, Anazawa T, Baba H, Kimura W, et al. Surgical risk model for acute diffuse peritonitis based on a Japanese nationwide database: an initial report on the surgical and 30-day mortality. Surg Today. 2015;45:1233–43.
Kurita N, Miyata H, Gotoh M, Shimada M, Imura S, Kimura W, et al. Risk model for distal gastrectomy when treating gastric cancer based on data from 33,917 Japanese patients collected using a nationwide we-based data entry system. Ann Surg. 2015;262:295–303.
Anazawa T, Paruch JL, Miyata H, Gotoh M, Ko CY, Cohen ME, et al. Comparison of national operative mortality in gastroenterological surgery using web-based prospective data entry systems. Medicine (Baltimore). 2015;94:e2194.
Shiloach M, Frencher SK Jr, Steeger JE, Rowell KS, Bartzokis K, Tomeh MG, et al. Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg. 2010;210:6–16.
Tomotaki A, Miyata H, Hashimoto H, Murakami A, Ono M. Results of data verification of the Japan congenital cardiovascular database, 2008 to 2009. World J Pediatr Congenit Heart Surg. 2014;5:47–53.
Kneuertz PJ, Pitt HA, Bilimoria KY, et al. Risk of morbidity and mortality following hepato-pancreato-biliary surgery. J Gastrointest Surg. 2012;16:1727–35.
Epelboym I, Gawlas I, Lee JA, Schrope B, Chabot JA, Allendorf JD. Limitations of ACS-NSQIP in reporting complications for patients undergoing pancreatectomy: underscoring the need for a pancreas-specific module. World J Surg. 2014;38:1461–7.
Acknowledgements
We thank all individuals and academics from the JSGS who extended their cooperation as well as all of the data managers and hospitals that cooperated in these audit activities. We also thank all auditors below as the members of the JSGS database (Akinobu Taketomi, Akira Tangoku, Chikara Kunisaki, Harushi Udagawa, Itaru Endo, Tadahiko Masaki), the members of NCD data quality management subcommittee (Akira Sasaki, Fuyuhiko Motoi, Hideaki Kimura, Kazushige Kawai, Kinji Kamiya, Masaaki Yokoyama, Masaki Ueno, Shinichiro Mori, Shoichi Fujii, Takayuki Anazawa, Tatsuhiko Kakisaka, Yoshihito Furukita, Yoshiyuki Sasaki), and the members of NCD data quality management subcommittee working group (Hiroshi Hasegawa, Kenji Wakayama, Kunihito Gotoh, Kyohei Ariake, Satoshi Iino, Takashi Kosaka, Takahiro Yoshida, Yuji Akiyama, Zenichiro Saze).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
We have no conflicts of interest to declare.
Rights and permissions
About this article
Cite this article
Kanaji, S., Takahashi, A., Miyata, H. et al. Initial verification of data from a clinical database of gastroenterological surgery in Japan. Surg Today 49, 328–333 (2019). https://doi.org/10.1007/s00595-018-1733-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00595-018-1733-9