Introduction

Pancreatic ductal adenocarcinoma (PDA) is one of the deadliest malignancies worldwide, and its incidence increased in the last decades [1]. Survival estimates improve after PDA resection combined with adjuvant chemotherapy [2], which highly depends on preoperative local and distant tumor spread [3]. An accurate staging of PDA on CT is the primary and most important step for appropriate patient management [3]. An inaccurate CT protocol for pancreatic tumors or an incomplete radiological report of PDA may lead to unnecessary laparotomy or major surgery in up to 19% of patients with high risk of residual disease following incomplete resection [4, 5].

Free-text narrative reports for PDA can be ambiguous and may lack critical information for preoperative tumor staging, resulting in inappropriate patient management [6]. A standardized radiological reporting template—which is currently recommended by NCCN guidelines for pancreatic ductal adenocarcinoma [3]—may overcome these limitations by providing well-coded key descriptors needed for staging and surgical planning and including all the vital information that define disease extent in a manner that is understandable to all members of the multidisciplinary team [7, 8]. However, there is still a reluctance to standardized radiological reporting for PDA among radiologists, mainly because it demands more time and energy. To date, only one study assessed the positive impact—i.e. improved reporting of key descriptors and improved surgeons’ confidence for treatment decisions—of the implementation of CT structured reporting for PDA compared to free-text narrative reports [9]. However, this study lacked intra-patient comparison of free-text and standardized reports, and did not compare inter-reader variability in the two types of reports. We hypothesize that the use of standardized radiological reporting of PDA affects intra-reader variability and lowers inter-reader variability—namely with readers with different level of expertise. These changes would ultimately result in increased quality and accuracy of the CT report to the ordering physician, reduced need of CT imaging review for preoperative assessment and greater clarity to surgeons.

The aim of this study was to evaluate whether a standardized structured radiology report improves the completeness of preoperative CT staging of pancreatic ductal adenocarcinoma compared to conventional free-text narrative reports.

Materials and methods

This retrospective, single-institution study was approved by the Institutional Review Board of University Hospital “Paolo Giaccone” in Palermo, and a waiver of informed consent was obtained. The authors had control of the data and the information submitted for publication. There was no industry support for this study.

Study cohort

A third year radiology resident (M.D.) retrospectively searched the departmental electronic database at our academic Institution for consecutive patients who were referred for pancreatic preoperative CT scan for untreated PDA staging with either resectable or unresectable disease between January 1, 2015 to December 31, 2018 and in whom a diagnosis of PDA was ultimately confirmed.

The search yielded an initial target population of 59 consecutive patients who were deemed eligible for inclusion in the study. Subjects were excluded for: (a) inadequate or unavailable CT images (n = 11); (b) other non-pancreatic tumors (bile duct, ampullary, and duodenal carcinomas) (n = 19); lack of reference standard (n = 2). Our final study population was composed of 27 patients (mean age, 64 ± 11.1 years [standard deviation]; range 41–80 years), including 15 women (mean age, 66.3 ± 9.7 years; age range 48–80 years) and 12 men (mean age, 61.1 ± 12.6 years; age range 41–78 years).

CT acquisition technique

All multiphasic CT examinations were performed using a 16-detector row scanner (GE Medical Systems, Milwaukee, Wis) (n = 20) or a 128-detector row scanner (Somatom Definition AS, Siemens Healthineers, Forchheim, Germany) (n = 7).

Patients received 110–150 mL of an intravenous non-ionic contrast medium (400 mg/ml Iomeprol, Iomeron 400, Bracco Imaging, Milan, Italy in 22 of 27 patients; 370 mg/dl Iopromide, Ultravist 370, Bayer Pharma, Berlin, Germany, in 3 of 27 patients; 350 mg/dl Iobitidrol, Xenetix 350, Guerbet, Roissy, France, 1/27 350 mg/dl Iohexol, Omnipaque 350, GE Healthcare AS, Oslo, Norway, in 1 of 27 patients), depending on availability and at radiologist’ discretion. The bolus of contrast medium was injected through an 18- to 20-gauge IV angiocatheter using a dual-chamber mechanical power injector (Medrad Stellant-Bayer) at a flow rate of 3–5 mL/s, followed by 30 mL of a saline chaser [7, 10]. All patients underwent a multiphasic contrast-enhanced CT protocol for pancreatic cancer, which includes an unenhanced acquisition from top of the liver to bottom of both kidneys, followed by pancreatic and portal venous phases at 35–40 s (using the bolus tracking technique) and 65–70 s after contrast injection, respectively [7, 10]. Scan parameters of the two CT scanners are summarized in Table 1.

Table 1 Scan parameters of the 16-row and 128-row CT scanners
Full size table

Imaging evaluation: free-text and structured report

Imaging evaluation did not consider the original official radiologist’ report because several different radiologists had interpreted the original studies with heterogeneous reporting style. Imaging evaluation was performed by two independent and blinded abdominal radiologists with different experience levels in abdominal imaging (Reader 1, G.S. with 16 years of experience in abdominal imaging, and Reader 2, S.P., a fellowship-trained abdominal radiologist with 1 year of experience) and two independent and blinded radiology residents (Reader 3, F.A. fourth year radiology resident, and Reader 4, D.C., second year radiology resident). The four readers were blinded to any clinical and laboratory data, and to the purpose of the study. Images were presented to the readers with the only generic indication of abdominal pain. To minimize recall bias, all personal data were removed from the images and images were randomized prior to all reading sessions. Two reading sessions separated by 4 weeks were performed by each reader. Readers were asked to report the images using a conventional free-text report at the first reading session, and structured report at the second reading session [11]. The structured report provided to the readers was based on the consensus statement on pancreatic ductal adenocarcinoma radiology reporting template endorsed by the Society of Abdominal Radiology and the American Pancreatic Association [11].

Pancreatic CT structured report consisted of four parts [8, 9, 11]:

  1. (I)

    Indication and technical/protocol information;

  2. (II)

    Morphologic evaluation (i.e. tumor size and attenuation, and associated gallbladder, biliary or pancreatic duct dilatation/abrupt interruption);

  3. (III)

    Vascular information (i.e. superior mesenteric artery (SMA), celiac axis (CA), common hepatic artery (CHA), left gastric artery (LGA), gastroduodenal artery (GDA) and splenic artery (SA) assessment including degree of solid and hazy attenuation/stranding contact, narrowing or contour irregularity and extension to their main branches, and arterial variants; main portal vein (PV), portal confluence, superior mesenteric vein (SMV) and splenic vein (SV) assessment including degree of solid and hazy attenuation/stranding contact, narrowing or contour irregularity, extension to draining veins, thrombosis and presence of collaterals) [11];

  4. (IV)

    Extrapancreatic evaluation, including lymph nodes (with size along the short axis and location) and metastases (e.g. liver, peritoneal/omentum), and ascites. Of note, the assessment of vascular involvement included the well-coded key descriptors “abutment” and “encasement” (Fig. 1) when tumor contact of the vessel circumference was less than or equal to 180° or more than 180°, respectively [7].

    Fig. 1
    figure 1

    Four different patients with pancreatic ductal adenocarcinoma and degree of vascular involvement of mesenteric vessels. Top row: Axial CT scan on pancreatic phase demonstrates abutment (a, arrow) and encasement (b, black arrowhead) of the superior mesenteric artery in a 78-year-old man and a 57-year-old woman with PDA, respectively; notice also thrombosis of the superior mesenteric vein (white arrowhead). Bottom row: Axial CT scan on portal venous phase demonstrates abutment (c, arrow) and encasement (d, arrowhead) of the superior mesenteric vein in a 57-year-old woman and an 80-year-old woman with PDA, respectively

    Full size image

Reference standard

Our reference standard was established by the same radiology resident selecting the target population, not involved in the imaging evaluation, who had access to patient records, including pathologic reports and all images obtained before and after the index CT examination. Pathological reports—i.e. biopsy (n = 5) or surgical (n = 22) specimens—with a diagnosis of PDA were assessed to reach the definitive diagnosis with the rationale to ensure a balance between resectable and unresectable PDA.

Statistical analysis

Statistical analysis was performed using SPSS software (Version 20.0. Armonk, NY, USA: IBM Corp). Continuous variables are presented as mean ± standard deviation and categorical variables as numbers and percentages.

First, differences in reported morphologic and vascular features in free-text reports compared to structured reports were assessed through McNemar Test. Statistical significance level was set at p < 0.05. Second, intra-reader agreement and inter-reader reliability for tumor vascular involvement and hepatic metastasis were calculated through Cohen κ test and intraclass correlation coefficient (ICC), respectively. Intra-reader and inter-reader agreement considered the following categories of reported tumor vascular contact: encasement, abutment, no contact, not reported contact or inadequate report. Intra-reader (k values) and inter-reader (ICC) agreement were categorized as poor (< 0.00), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), or almost perfect (0.81–1.00) [12].

Results

Morphological and vascular assessment

A total of 216 reports were completed by four different readers in 27 patients with PDA, including 108 free-text and 108 structured reports. Overall, 139 of 540 morphologic characteristics and 869 of 1188 vascular key features for PDA staging were only described in the structured reports, and not in the free-text reports (Table 2). Free-text more commonly included summary statements such as “lack of vascular involvement” without specific indication about peritumoral artery and veins compared to structured reports (all p < 0.001) (Table 2).

Table 2 Frequency of reported primary tumor imaging features, signs of vascular invasion, and distant metastases in the free-text report compared to structured repots by different readers
Full size table

Intra-reader agreement and level of experience

Table 3 reports the frequency of specific vascular involvement and intra-reader agreement by four different readers. Concerning the type of solid tumor contact with arteries and veins, the intra-reader agreement between free-text and structured reports was slight-to-fair (k = 0.012–0.281) for arteries and poor-to-moderate (k = − 0.018 to 0.511) for veins. Free-text narrative reports resulted in lack or inadequate information (including summary statements regarding one or more vessels) on specific peritumoral arterial or venous involvement in 70.4–100% for arteries and 59.3–100% for veins for Reader 1; 51.9–88.9% and 22.2–81.5%, respectively, for Reader 2; 66.7–100% and 70.4–77.8% for Reader 3; 70.4–92.6% and 66.7–92.6%, respectively for Reader 4 (Table 3, Fig. 2). Higher intra-reader variability in arterial involvement was noted for left gastric artery, gastroduodenal artery and splenic artery. Particularly, encasement of these three vessels was specifically described by the four readers in 0–3.7%, 3.7–11.1%, and 3.7–18.5% of cases using free-text reports, respectively and in 0–14.8%, 7.4–25.9%, and 18.5–29.6% using structured report. The lack or inadequate information in free-text narrative reports and the variability of vascular involvement resulted in low-intra-reader agreement for involvement of left gastric artery (k = 0.046), gastroduodenal artery (k = 0.034–0.086) and splenic artery (k = 0.033–0.216). Similarly, encasement of the portal confluence, superior mesenteric vein and splenic vein was reported by the four readers in 0–14.8%, 7.4–25.9%, and 11.1–25.9%, respectively, using free-text reports and in 18.5–33.3%, 22.2–29.6%, and 18.5–29.6%, respectively, using structured report, resulting in poor-to-fair intra-reader agreement (k = − 0.018 to 0.111 for the portal confluence; k = 0.081–0.322 for superior mesenteric vein; k = 0.121–0.291 for splenic vein).

Table 3 Distribution of vascular invasion and intra-reader agreement between free-text and structured reports
Full size table
Fig. 2
figure 2

Axial CT images on pancreatic (a) and portal venous (b) phases of a 71-year-old woman with pancreatic ductal adenocarcinoma. In the free-text report no specific descriptions were included regarding the gastroduodenal artery or superior mesenteric vein involvements. When using the structured report, two out of four readers described abutment of the gastroduodenal artery (arrowhead), while three out of four agreed with the abutment of the superior mesenteric vein (arrow)

Full size image

Regarding the level of experience, the highest intra-reader agreement for the assessment of both arterial (k = 0.046–0.281) and venous (k = 0.111–0.511) involvement was achieved by Reader 2. As opposed, Reader 4 had the lowest intra-reader agreement for most of arterial (k = 0.033–0.239) and venous (k = − 0.018 to 0.213) involvement. In patients that were deemed completely free of vascular tumor involvement, free-text report included a summary statement with all abdominal vessels (i.e. “no vascular involvement”) in 4/27 reports (14.8%) by Reader 2, in 4/27 reports (14.8%) by Reader 3 and in 2/27 (7.4%) reports by Reader 4, but never by Reader 1.

Hepatic metastases were reported in 22.2–25.9% and 22.2–25.9% using the free-text and structured report, respectively. Intra-reader agreement was almost perfect for Reader 2 (k = 0.910) and Reader 3 (k = 0.901), substantial for Reader 4 (k = 0.703), while it was slight for Reader 1 (k = 0.205) due to lack of specific mentioning in 66.7% of free-text reports.

Inter-reader variability

The assessment of encasement, abutment, no contact and not reported contact of major arteries (i.e. superior mesenteric artery, celiac axis, common hepatic artery and splenic artery) reached similar inter-reader agreement levels with free-text and structured reports (ICC = 0.780–0.937 vs. ICC = 0.702–0.944, respectively) (Fig. 3). Conversely, an improvement of inter-reader agreement was observed using structured report compared to free-text report from moderate to almost perfect for left gastric artery (ICC = 0.493 vs. ICC = 0.844, respectively) and portal confluence (ICC = 0.422 vs. ICC = 0.848, respectively), from moderate to substantial for gastroduodenal artery (ICC = 0.449 vs. ICC = 0.730, respectively), from substantial to almost perfect for portal vein (ICC = 0.638 vs. ICC = 0.847, respectively) (Figs. 3 and 4). Improvement in inter-reader agreement was noted also for superior mesenteric vein (ICC = 0.695 vs. ICC = 0.765, respectively), and splenic vein (ICC = 0.841 vs. ICC = 0.921, respectively) (Fig. 4).

Fig. 3
figure 3

Inter-reader variability and distribution of arterial vascular involvement (yellow: percentage reported as abutment; red: percentage reported as encasement) in free-text and structured reports by each reader. ICC intraclass correlation coefficient

Full size image
Fig. 4
figure 4

Inter-reader variability and distribution of venous involvement (yellow: percentage reported as abutment; red: percentage reported as encasement) in free-text and structured reports by each reader. ICC intraclass correlation coefficient

Full size image

The inter-reader agreement for the assessment of hepatic metastases increased from 0.837 when using the free-text report to 0.987 for the structured report.

Discussion

The aim of this preliminary, single-center study was to evaluate the impact of structured reports for PDA staging on CT upon overall quality, completeness, intra- and inter-reader variability. Our results confirmed our hypothesis that the use of structured reports for PDA staging significantly reduces the number of missing key features and inter-reader variability. To our knowledge, this is the first study analyzing intra- and inter-reader variability with structured reports compared to free-text reports. According to our results, structured reports for PDA allow to better fulfill the two main tasks of diagnostic radiologists including identifying and interpreting the information available from diagnostic imaging studies and communicating that interpretation meaningfully to the referring clinician [13].

Our study shows that the use of a structured report for staging of PDA yields higher inter-reader agreement in the assessment of the encasement or abutment of the portal vein, portal confluence, superior mesenteric and splenic vein compared to free-text report (ICC = 0.422–0.841 vs. ICC = 0.765–0.921, respectively). More importantly, the assessment of encasement or abutment of some arteries—including left gastric artery and gastroduodenal artery—shows a significant improvement of inter-reader agreement using structured report compared to free-text report (ICC = 0.449–0.493 vs. 0.730–0.848, respectively). This reduced inter-reader variability is of utmost importance for patient management because it allows readers with different level of expertise in abdominal imaging to achieve similar results in PDA staging. Of note, our results also demonstrate that free-text reports variability is influenced by readers’ experience and training—with the highest intra-reader agreement reached by the fellowship-trained abdominal radiologist—and that structured reporting lessens this variability. A radiological report of PDA not mentioning or underdiagnosing arterial or venous involvement may potentially lead to misdiagnosing a borderline resectable pancreatic cancer as resectable [3]. This false-negative CT staging would ultimately result in unnecessary surgery with high risk of residual disease and may prevent the oncologist from performing neoadjuvant therapy [3]. The oncological benefits of neoadjuvant therapy—including neoadjuvant chemotherapy and neoadjuvant chemo-radiotherapy—in borderline resectable PDA has been proven in recent studies. It allows both a potential downstaging of locally advanced disease [14, 15], and longer overall survival and 2-year survival rates [15,16,17]. It is therefore a duty of the radiologist to provide an adequate and complete radiological report independently from level of expertise, and the use of a structured report for PDA may allow for reproducible high-quality complete reports.

In agreement with the results of a prior study by Brook et al. [9], our results showed that key features for the preoperative staging of PDA were overlooked more commonly with free-text report compared to structured report. Specifically, lack of description of involvement of small peritumoral arteries may misguide the choice of the best surgical approach in resectable tumors [18, 19]. In some patients tumor invasion of specific vessels (i.e. splenic artery or left gastric artery) does not affect resectability but may change surgical planning [7]. The development of newer vascular reconstruction techniques has now made locally advanced disease with major vascular involvement potentially resectable [18, 20]. In our cohort of patients, the assessment of encasement or abutment of major arteries was adequate with both free-text and structured report, while encasement or abutment of smaller arteries (i.e. left gastric artery and gastroduodenal artery) was more commonly and adequately reported by each radiologist with the structured report (i.e. improved intra-reader variability). Therefore, a complete assessment of vascular invasion of PDA obtained with structured report may potentially improve surgical planning, ultimately resulting in reduced operative time.

In addition to the limited number of patients included, other limitations pertain to this study. First, we acknowledge that with a structured report there may be a potential to miss an unexpected finding not included in the template. Although we could not assess for this potential bias of structured reporting, the main aim of preoperative staging of PDA is to provide the oncologist and the surgeon information related to the tumor itself and incidental findings are likely benign and often have little or no clinical significance [21]. Second, considering that readers used the same images for structured and free-text reports, potential bias due to testing or learning effects cannot be ruled out. To minimize these effects, the two reading sessions were performed with a 4-week delay using random case order. Third, our readers had no time constraints or other pressures placed on them during testing which does not reflect routine practice; in addition, we did not assess the required time for reporting with the two reports. In respect to these points, our readers included less experienced radiology trainees and time constraints could have lessened excessively their performance thus potentially creating a bias for the assessment of inter-reader agreement. Fourth, the lack of a control group (e.g. patients with mass-forming pancreatitis)—which does not reflect clinical practice—did not allow for assessment of specificity in a laboratory environment. However, readers’ performance is likely not influenced by either lack of a control group or prevalence of 100% of PDA [22]. Fifth, our study lacked a gold standard for the assessment of vascular involvement since in 18% of cases the reference standard was not based on surgical specimen. Finally, although accuracy and completeness are crucial aspects of report quality, we did not assess whether referring physicians or radiologists preferred free-text or structured reports and the variability of tumor resectability assessment by the surgeons with the two type of reports, but these points were already investigated by Brook et al. [9].

In conclusion, the use of structured reports improves the quality of preoperative CT staging of PDA compared to standard free-text report, with lower number of missing key features independently of the years of experience. A significant improvement of inter-reader agreement in the assessment of vascular invasion with structured reports allows for more reproducible information obtained with preoperative CT staging of pancreatic ductal adenocarcinoma.