Introduction

Over the past two decades, there has been a dramatic increase in studies based on large databases such as administrative claims data and large multi-institutional tumor registries (Fig. 1). In the field of surgical oncology, the Surveillance, Epidemiology, and End Results (SEER) registry1 maintained by the National Cancer Institute (NCI) has been used extensively, both alone and in conjunction with administrative data from the Medicare program,2 to understand epidemiology, treatment, and prognosis. Additionally, the National Cancer Database (NCDB) maintained by the American College of Surgeons Commission on Cancer has become a popular source of data on surgical and multimodality treatment and outcomes at cancer-focused hospitals. These data have been used to explore various themes including epidemiology, oncology, surgical techniques, perioperative outcomes, and prognosis. The relative ease with which these data can be acquired, the large available sample sizes, their improved generalizability compared with institutional data and reliable long-term follow-up collectively make them attractive to researchers. Furthermore, the availability of long-term follow-up offers an attractive option to study long-term survival of cancer patients.

Fig. 1
figure 1

Trends in use of SEER and NCDB over the past few decades

Full size image

Although registry analyses have an important role in surgical research, the limitations of these data and the statistical methods used to analyze the data often do not receive sufficient attention in the surgical literature. A strong understanding of the strengths and weaknesses of these databases is necessary to allow surgeons to perform high-quality research and critically appraise the literature. This review seeks to provide an overview of common research themes with some well-described limitations and caveats specific to the analysis and interpretation of these data. Although examples from SEER and NCDB are specifically cited, the concepts reviewed are broadly generalizable to secondary data analyses using other databases.

Background

The Surveillance, Epidemiology, and End Results (SEER) database is maintained by the National Cancer Institute (NCI). The SEER database has grown to include 21 cancer registries, representing 28% of the US population. As compared to the general US population, the SEER population is slightly more urban and has a slightly higher percentage of foreign-born individuals. Available data include patient demographics, tumor characteristics (e.g., histology, grade, stage), treatment data (e.g., surgery, radiation), and survival. Some data elements (e.g., American Joint Commission on Cancer (AJCC) staging, details of surgical therapy, tumor size, lymph node involvement) are consistently available only in more recent time periods. The NCI staff work with the North American Association of Central Cancer Registries (NAACCR) External Web Site Policy and a number of collaborating organizations to guide all state registries to achieve data content and compatibility acceptable for pooling data and improving national estimates.

The NCDB is a joint project of the Commission on Cancer (CoC) of the American College of Surgeons and the American Cancer Society.3,4 The NCDB gathers information from approximately 1,500 CoC-accredited hospitals and includes >70 % of all newly diagnosed malignancies in the USA. Because of its reliance on CoC hospitals for data, NCDB is not, strictly speaking, a population-based database. Similar to SEER, NCDB contains specific details about patient demographics, tumor characteristics, treatment, and survival. NCDB also includes information on patient comorbidities, facility type and location, perioperative outcomes (e.g., length of stay, readmission), receipt of chemotherapy, and resection margins. In addition, Through linkage to tertiary data sources, area-based socioeconomic status and estimates of travel distance/time to the reporting hospital can be obtained based on each patient’s home zip code at the time of diagnosis. In addition, hospital type and structural characteristics can be determined through linkage with the CoC’s facility information profile system (FIPS) file of self-reported hospital structural features, resources, and services related to oncology.

Research Themes

Epidemiology

Assessment of cancer incidence has been a major strength of SEER, providing population level data to allow age-adjusted analysis. These include incidence and mortality rates across different cancers over time. A notable example is the annual cancer statistics report released by the American Cancer Society evaluating trends in cancer incidence and mortality across different cancers over time.5,6,7 The large numbers allow study of specific demographic subgroups and uncommon disease types and histologies. Registry analyses also help provide benchmarking data for quality improvement and assessment of disparities in access, quality, and outcomes.

Surgical Oncology

The NCDB allows for multimodal treatment to be characterized, while this is less so for SEER as they lack information on type of oncological therapy (e.g., chemotherapy, radiotherapy). For instance, NCDB provides type (single-agent vs. multi-agent), sequence (neoadjuvant, intraoperative, adjuvant), and duration of chemotherapy, which potentially allows more nuanced analysis compared to the SEER database, where these data are not available. However, it still remains difficult to separate chemoradiation and chemotherapy in the NCDB.

The NCDB codes all events relative to the date of diagnosis (the first date a statement of diagnosis is made by a physician, which may be based on results of imaging or confirmation via histologic testing). Therefore, the timing of various treatments or events (e.g., start of chemotherapy or surgery) relative to the date of diagnosis can be used to determine sequence.8 More complicated strategies (e.g., chemotherapy or radiotherapy before and after surgery) can be determined using specific sequence variables in the NCDB. Recent studies have used timing data in the NCDB to assess the association of delaying adjuvant chemotherapy on survival of colon cancer9 and lung cancer.10 Further, NCDB allows assessment of national practices across the USA on use of systemic chemotherapy or radiotherapy (RT) in specific cancer types and evaluate its long-term benefit. For instance, a recent study highlighted a modest survival benefit of adjuvant RT in patients undergoing margin-negative pancreatic cancer resections using appropriate propensity matching techniques to account for treatment selection bias.11

Owing to the availability of some information on surgical approach (e.g., open or minimally invasive), researchers have used NCDB in assessing the association of minimally invasive techniques on short-term and long-term outcomes across a spread of different cancers.12,13,14,15 These data are not available in SEER. Such data have allowed appraisal on the benefit of minimally invasive techniques beyond a few specialized centers. More recently, the adoption of robotic surgery has led to further analysis on the benefits of robotic over laparoscopic surgery in surgical oncology cancers.16,17,18,19 However, it is important to note that current appraisals on minimally invasive techniques may be limited as surgery types (e.g., two-stage, three-stage, transhiatal esophagectomies) and reasons for conversion to open surgery are not available to provide granular analysis, resulting in some degree of bias in interpretation of results.

Health Services Research

Common themes within health services research exploring care of cancer patients include determinants of short-term outcomes (e.g., teaching status, center volume), determinants of long-term outcomes (e.g., time to surgery, timing of adjuvant treatment), hospital-based variation in treatment and outcomes, and disparities in treatment and outcomes. While center volume has been widely reported to be associated with short-term mortality,20,21 data on its impact on long-term survival of cancer patients are less robust. Center volume data can be derived from the NCDB through available unique facility codes, but this is not the case for the SEER database (unless Medicare claims are used).22 Studies have demonstrated improved survival in patients treated in high-volume centers for glioblastoma,23 metastatic renal cell carcinoma,24 and prostate cancers,25 for example.

The impact of time to surgery on long-term survival has also been evaluated using these databases as a determinant of outcomes. Although some studies have demonstrated that a delay to surgery is associated with reduced survival26,27,28,29,30 due to theoretical increased risk in delay and hence progression of tumor, some studies have reported the contrary. These data may be useful for clinicians for counselling patients on the impact on delay to surgery. However, cautious interpretation is required as such analysis may not consider residual bias such as from potential delays from referral or patients’ receipt neoadjuvant therapy prior to surgery.

NCDB and SEER have been utilized to understand impact of hospital-level characteristics such as center volume and institutional type (e.g., academic vs non-academic). In particular, NCDB has been utilized to examine impact of center volume in short-term mortality following major cancer surgery, owing to availability of annual procedural volume. However, this is not the case for the SEER database. Studies from NCDB have demonstrated that high-volume centers were associated with significantly lower 30-day mortality rates compared to low-volume centers, despite similar complication rates.31,32,33 These data implicate failure to rescue as a major determinant of variation in hospital mortality rates. In other work, centers designated as “academic” were more likely to have better surgical outcomes than “non-academic” centers.23 It is difficult to assess the significance of such findings, as programs with academic research are highly heterogeneous. Whenever possible, more detailed and objective hospital characteristics should be used. Although details of these observations are not available within both databases, they provide impetus for future research to identify factors that may explain these findings.21,34,35

Furthermore, underutilization of surgery in early stage cancers has also been a popular topic within both databases to highlight disparities in allocation to curative surgery. Several studies highlight variation in trends of curative resection for early (stage I) cancers,36,37 for which resection is usually the gold standard treatment. These studies have three broad advantages: (i) evaluate and characterize the utilization of surgery for cancers; (ii) to identify factors predicting failure to undergo surgery for localized disease, and (iii) to evaluate the association of surgery with survival. Such studies help identify areas for improvement and target center- or population-level interventions to improve utilization of surgery across centers.

Perioperative Outcomes

Research questions around perioperative outcomes are particularly common with the NCDB (but not SEER) due to the availability of readmission, 30-day and 90-day mortality rates with the former. Recent studies38,39,40 have evaluated readmissions and 30-day mortality in across different cancers in identifying factors associated with readmissions as these Interventions aimed at decreasing hospital readmissions should target these high-risk patients. These studies are limited by the fact that reasons for readmission are not captured within the NCDB. This is important especially in the context of identifying areas for improvement within health systems.

Prognosis and Staging

In the theme of prognostication, SEER and NCDB have been commonly used in validation of the AJCC staging system across various cancers over the past decade. Such analyses have been useful in testing the relevance of new changes to these staging systems and assess performance in models.41,42,43,44 Although these studies identify that current staging systems are poor at discriminating long-term survival in patients undergoing curative resection, it helps clinicians in providing important information to patients. It is important to note that these databases can only assess factors that are collected by the databases, which are usually just the factors that are already in the AJCC staging system

Owing to limitations of existing staging systems for cancers, both the SEER and NCDB registries have been also been utilized in developing prognostic models, risk scores or nomograms in guiding discussions with patients to guide treatment decisions that may influence long-term survival. For instance, a scoring system was developed to predict risk of lymph node metastases in patients with early T1 esophageal cancers to guide clinicians on decision between endoscopic or surgical resection.45 The proposed scoring system seems to be useful in discriminating risk of nodal metastases in patients with T1a or T1b esophageal adenocarcinoma and may be useful in directing patients who received endoscopic resection to esophagectomy or careful follow-up. Similar studies have been performed in other cancers with NCDB or SEER.46 However, applicability of these risk scores in other populations remain unknown as these risk scores lack external validation.

Finally, both NCDB and SEER are useful in identifying patient-level, tumor-level and treatment-level characteristics associated with good long-term prognosis. This remains possible with NCDB and SEER due to good long-term follow-up of these patients. A recent NCDB study in patients with pancreatic cancers identified only 4% (431/11,081) of patients were alive at 10 year and significant predictors were lymph node ratio, administration of adjuvant chemotherapy, and pathologic T stage.47 These useful well-designed study will help clinicians in prognostication of patient following cancer resections. Wider applicability to a variety of different cancer types can aid decision making and planning of follow-up management.

Addressing Database Limitations

Generalizability and Missing Data

Although SEER and NCDB provide comprehensive evaluation across >1000 institutions across the USA, neither of these prospectively maintained registries are externally validated to ensure high case ascertainment and data accuracy. Lack of external validation is highlighted by the variation in missing data across data fields, an inherent issue widely recognized within large databases. As a general rule, data fields with >10% missing data threaten data validity due to risk of bias. Two options exists to account for missing data: (1) Listwise deletion or complete-case analysis48; or (2) multiple imputation.49 The former is subjected to lower available sample size and consequently results in loss of power. Multiple imputation is one way to ensure the sample size is maintained to ensure reliable results can be drawn from powered studies. This is especially important in diseases with low incidence such as cholangiocarcinoma50,51,52,53 or gallbladder cancers.54,55 However, even for multiple imputation to be used, missing data should be missing at random, a condition that often cannot be empirically proven. Studies using administrative data should report the extent of missing data, use appropriate methods to account for missing data in analyses, and discuss their potential impact on inferences and conclusions.

Data Quality and Depth

Across cancer registries, there remain issues to depth of data provided, limiting granular analyses. For instance, NCDB contains data on margin status, surgical approach (e.g., open, laparoscopic, or robotic), and adjuvant regimens (e.g., chemotherapy, chemoradiotherapy). However, NCDB has some limitations in this regard. Firs, margin status may be biased by variation in institutional assessment because definitions for margin-negative or margin-positive resections are not well-defined. This is particularly important for esophagogastric cancers where details of longitudinal or circumferential margins are not specified in the database. Second, the Charlson-Deyo comorbidity score is provided in the NCDB but is captured in a summative way that does not convey fitness for surgery. Furthermore, data on performance status are not available within these datasets. Third, another notable difference exists between cancer types. For example, comprehensive genetic information is provided for breast and colorectal cancers, but not for hepatobiliary and esophagogastric cancers. Fourth, there is no information on disease-free or disease-specific survival. Finally, both databases also lack clinical detail beyond staging, such as extent of vascular involvement in pancreatic cancer or details of multivisceral involvement of solid tumors. In liver cancer, for example, location of a lesion adjacent to critical structures and extent of underlying liver disease can affect treatment options but are not conveyed in the data.

Bias

Although external validity of large databases can be threatened if the study population does not accurately represent the clinical population of interest, internal validity can be threatened by biases.52 The three main types of widely recognized biases are information bias, selection bias, and confounding bias.56 However, treatment selection bias remains pertinent to surgical oncology research. Non-random treatment assignment to oncological treatment introduces selection biases. For instance, allocation to surgery is more likely in younger, less co-morbid patients or those with early stage disease. To deal with this, multivariable regression modelling is utilized to produce adjusted estimates of the explanatory variable of interest.57 Over the past few year, propensity score analysis58 to account for non-random treatment assignment has become an increasingly popular method to address treatment selection bias. Despite this, correct application of propensity matching within surgical literature is lacking, warranting further judicious use to ensure accurate and reliable conclusions are drawn.59 Furthermore, these biases may be overcome with the use of interaction and sensitivity analyses to support findings of the overall analysis as utilized in recent NCDB studies.11,60

Conclusion

Moving forwards, cancer registries will be useful to assess relevant questions pertinent to national clinical practice. Large database studies will be able provide a “real” world perspective on surgical practice and could aid in conducting well-designed randomized controlled trials (RCTs). There even have been initiatives to integrated the strengths of large database studies and RCTs by performing for example registry-based pragmatic RCTs.61 In addition, large database studies are able to reflect on research questions regarding the efficacy of health policy, access to health care, and trends and geographic variation in practice patterns, as well as the treatment of rare disease or patients subgroups, which would be impossible or very strenuous by using RCTs. Moreover, large nationwide datasets, provide the tremendous opportunity to benchmark surgical outcomes and subsequently improve quality of care.

However, judicious use of these databases is also important, as analyses using these data are susceptible to bias (Table 1). Therefore, we have summarized a checklist to assist researchers in the appropriate use of these databases (Table 2). Great emphasis on appropriate, well-designed studies and statistical methods are essential, but this does not obviate the inherent limitations of the data or their retrospective nature. Statistical methods should aim to explore and mitigate threats to the validity of the conclusions whenever possible. Researchers and reviews alike should focus not on the limitations of the data per se—there are many—but rather on how these limitations affect the inferences from the data and conclusions of the research.

Table 1 Summary of advantages and disadvantages of the NCDB and SEER databases
Full size table
Table 2 Summary checklist for research studies utilizing the NCDB and SEER databases
Full size table