Introduction

Most children undergoing cancer therapy experience multiple symptoms [1,2,3]. Among subjective symptoms such as pain or fatigue, documented discrepancies exist between child and proxy and/or child and clinician symptom reports [4, 5]. Accurate symptom reports are necessary to appropriately monitor and manage symptoms in an effort to improve the quality of life for children undergoing cancer therapy. Pediatric patient-reported outcome (PRO) instruments help collect valuable information directly from children but require careful consideration of the factors that may influence data completion and quality, including the child’s cognitive abilities and developmental stage. Many PRO instruments require written questionnaires to be completed independently, which necessitates children/adolescents to have appropriate cognitive skills to read and understand questions and to select answers that match their experiences or perspectives. Emerging literacy skills in children involve a combination of alphabet knowledge, phonological awareness, spelling, and oral language skills encompassing both receptive and expressive vocabulary [6,7,8]. As such, the ages at which it is feasible to begin PRO data collection in children may vary depending on the type of information being collected and the data collection modality (written, pictorial or verbal).

To self-report, children must have self-awareness; be able to concentrate and pay attention; comprehend instructions, the questions being asked, and the response options; and understand time if questions involve a recall period [9]. PRO instruments, such as visual analog scales, have been successfully administered to children as young as 4 years of age [10, 11]. Children ages 5–7 years, with and without chronic illnesses, have demonstrated the ability to self-report health-related quality of life when questions were administered by an interviewer [12]. Overall, age is used to guide recommendations for when children are able to self-report and when children are able to transition to adolescent or adult versions of PRO instruments.

With growing interest in integrating PRO data into clinical research and healthcare delivery settings to inform decision making, it is critical that we use instruments with maximal ability to elicit the voice of children with chronic illnesses. Although many PRO questionnaires used in pediatrics start around 8 years of age, a greater understanding is needed regarding the extent that a child’s disease may impact their development and cognitive abilities. Among children with cancer, causes for impairments or delays can be pathophysiological in nature (e.g., brain tumor), treatment related (e.g., cranial radiation), or could potentially result from missing school while undergoing prolonged treatment [13, 14]. As such, age alone is an imperfect standard for determining the ability of a child with cancer to complete a PRO instrument.

Large studies, such as the National Assessment of Educational Progress, have documented disparities in academic achievement between white, black, and Hispanic children [15]. Over the past 20 years, racial differences in reading achievements have been noted with black and Hispanic students scoring significantly lower than white students [16]. As such, this study included race as a variable to investigate potential interactions with childhood literacy.

The objectives of this study were to assess relationships among literacy, chronological age, and race to determine a child’s ability to understand and complete the Pediatric Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (Pediatric PRO-CTCAE) questionnaire. Using cognitive interview data and literacy assessments, our three specific aims are as follows: (1) determine the association between child literacy level and understanding of the Pediatric PRO-CTCAE questionnaire, (2) determine the association between child literacy level and chronological age to assist with deciding when children can begin completing the Pediatric PRO-CTCAE, and (3) investigate if disparities are present in literacy levels by race/ethnicity.

Methods

This child literacy study was part of a larger study entitled Creating and Validating Child Adverse Event Reporting in Oncology Trials (NIH R01CA175759). The study had Institutional Review Board approval at eight sites: Children’s Healthcare of Atlanta/Emory University, Children’s Hospital Los Angeles, Children’s National Health System, Dana-Farber Cancer Institute/Boston Children’s Hospital, The Hospital for Sick Children, Palmetto Health Children’s Hospital, St. Jude Children’s Research Hospital, and the University of North Carolina at Chapel Hill. The goal of the larger companion study was to develop, refine, and validate the Pediatric PRO-CTCAE and Proxy version of the Pediatric PRO-CTCAE for use in pediatric oncology clinical trials [17].

Eligibility criteria

We included children and adolescents between the ages of 7–20 years, who were English speaking, and actively receiving treatment for any type/stage of cancer. Participants were recruited from inpatient and outpatient treatment settings. Participants, ages ≥ 18 years , provided consent for study participation, while children under age 18 had parent/guardian consent, in addition to child assent in accordance with each institution’s policy.

Study design

The Pediatric PRO-CTCAE includes 62 possible symptomatic adverse events that may be subjectively experienced during therapy. One-on-one cognitive interviews were conducted with participants to evaluate their understanding of survey items assessing the symptom’s frequency, severity, and interference with daily activities. Data collection was completed during a normal visit for treatment. As the age span for inclusion was diverse, cognitive interviews were grouped to represent distinct developmental stages (7–8, 9–12, 13–15, and 16–20 years). Interviews lasted up to 1 hour with participants completing the following activities (in order): (1) a paper copy of the PRO-CTCAE measure (Pediatric or Adult version), (2) a cognitive interview, and (3) a word reading activity from the Wide Range Achievement Test version 4 (WRAT) for those participating in the sub-study to assess literacy. Digital audio recordings of cognitive interviews and WRAT assessments were obtained with permission from the caregiver and child. After two rounds of cognitive interviews in the larger companion study, 12 additional children (ages 7–9 years) were recruited at Emory University to extend testing of the core symptoms in the Pediatric PRO-CTCAE in the younger age group. Core symptoms were defined as those most commonly occurring across all types of pediatric cancer therapies [18]. Interviewers at all sites completed training related to cognitive interviews and administering WRAT assessments prior to conducting their first interviews. Additionally, interviewers participated in weekly calls to provide feedback on the conduct of interviews and to address any issues. These activities promoted consistency across sites to enhance data quality.

Measures

PRO-CTCAE

PRO-CTCAE refers to the adult version of the instrument. The Pediatric version of the PRO-CTCAE is a library of items to assess up to 62 symptomatic adverse events (AE). Children and adolescents completed either the PRO-CTCAE or the Pediatric PRO-CTCAE based on age and round of cognitive interview (Fig. 1). Children ages 7–12 years completed the Pediatric PRO-CTCAE instrument. During Round 1, children ages 13–20 years completed the PRO-CTCAE [19] which included 55 original items and 7 new items to match symptoms captured in the pediatric version [17]. In Round 2, 13–15 year olds completed the Pediatric PRO-CTCAE. Instrument development and findings for the qualitative evaluation of the PRO-CTCAE [19,20,21] and Pediatric PRO-CTCAE [17, 18] have previously been reported. Within the Pediatric PRO-CTCAE library, each AE consists of one to three questions to reflect symptom attributes such as presence, frequency, severity, and/or interference with daily activities.

Fig. 1
figure 1

PRO-CTCAE instrument by age group for cognitive interviews

Full size image

During the first part of the interviews, participants completed the survey measures on their own and marked items as “Hard to Understand” (HTU) if they experienced difficulty with reading or understanding the symptom item or question. If a child could not read the questions independently, the interviewer read the questions to the child. During the interview, field notes were kept to document words or items that children had questions about, but did not explicitly mark as HTU, and/or to document when the child needed help reading all or part of the questionnaire. Additionally, the validity of participants marking items as HTU was confirmed through cognitive interview questions such as “Did you think it was generally easy or hard to answer most of the questions?” and a follow-up probe of “What was hard about it?” when appropriate. Questions or terms identified as HTU in round 1 of cognitive interviews were revised, and the new language was evaluated in round 2 of interviews.

Reading level of PRO-CTCAE instruments

Flesch–Kincaid grade reading level analyses of the individual questions of the Pediatric PRO-CTCAE were evaluated using Microsoft Word and ranged from 0.1 (less than a first grade reading level) to 11.1 (11th grade). Across all 130 questions, the mean grade level was at 4.5 (between a 4th and 5th grade reading equivalency level). Flesch–Kincaid grade level analyses for the 110 individual questions included in the adult PRO-CTCAE item bank had reading levels ranging from 0.1 to 15.8 (with the higher score translating to an estimated college reading equivalency level). Across all items in the adult PRO-CTCAE bank items, the mean reading level was estimated at a 6.5 grade level.

Wide Range Achievement Test (WRAT) 4

This study utilized the Word Reading subtest of the WRAT version 4 as a proxy for literacy [22]. The WRAT assesses the basic academic skills of word reading, sentence comprehension, spelling, and math utilizing a norm-reference that was standardized on a sample of 3,000 individuals between the ages of 5–94 years [22]. The Word Reading subtest of the WRAT (henceforth labeled as WRAT-WR) measures letter and word recognition and has two assessments forms (green or blue color coded word cards) which can be interchanged with comparable results [22]. The WRAT-WR subtest includes 70 items (individual letters and words), which children were asked to read out loud. The words are listed in order of increasing phonological complexity. Raw scores for the WRAT-WR subtest range from 0 to 70 as the instrument is scored by giving the participant one point for each letter or word correctly read. The assessment was discontinued when a participant incorrectly read 10 consecutive items or when the child asked to stop as the words became too advanced for them to attempt to pronounce. The internal consistency reliability coefficients for the WRAT-WR subtest range from 0.88 to 0.98 depending on age group and form version, with a Test–Retest reliability of 0.86 [22]. The WRAT4 also demonstrates an acceptable level of concurrent validity with like measures for Word Reading with a median correlation of 0.71 [22].

In this study, WRAT-WR subtests were scored by cognitive interviewers at the end of the interview, and raw scores were entered into a study database. After interviews concluded, one or two independent reviewers (JW and MM) verified the WRAT-WR subtest scores using available audio recordings in order to assess scoring accuracy. The raw WRAT-WR subtest scores were transformed to standardized scores using the WRAT4 manual conversion charts for age and word card color [22]. Standardized WRAT-WR scores have a mean of 100 with a 15-point standard deviation [22]. Standardized WRAT-WR scores were categorized using qualitative descriptions and the score ranges provided in the manual [22]. Calculated scores were categorized as “Average/Above Average” if standardized WRAT-WR scores were 90 or greater. Standardized WRAT-WR scores of 89 and below were categorized as “Below Average.”

Statistical analysis

WRAT-WR scores were examined as continuous and binary outcomes. For the binary outcome, WRAT-WR scores were dichotomized as Average/Above Average vs. Below Average. Using this dichotomized definition of word reading level, t tests were used to compare mean age (years) and mean child WRAT-WR scores across the two groups. Chi-square tests were used to examine unadjusted differences in cohort (gender, race, ethnicity) and clinical (inpatient/outpatient, cancer type) characteristics by literacy level. To evaluate the association between literacy level and comprehension of the Pediatric PRO-CTCAE, items reported as HTU were counted and grouped (0 HTU, 1 HTU, and 2+). Standardized WRAT-WR scores were utilized only when literacy scores needed to be compared across ages or categorized (under, average or above average) as previously described.

Aim 1: Evaluate the relationship between literacy level and understanding of the PRO-CTCAE questionnaires

To determine if the number of HTU items was associated with lower WRAT-WR scores, we examined unadjusted associations between the number of HTU items and the child’s raw WRAT-WR scores, using a simple linear regression model with continuous number of HTU items as the outcome. We then estimated multivariable linear regression models adjusting for the child’s age, gender, race, ethnicity, treatment setting (i.e., inpatient or outpatient), and cancer type (leukemia, lymphoma, solid tumor, brain tumor).

Aim 2: Evaluate the relationship between literacy level and chronological age

Unadjusted associations between literacy level and chronological age were examined using a simple linear regression model with child standardized WRAT-WR scores as the outcome. Standardized scores were used to compare findings across multiple ages. Adjusting for the child’s gender, race, ethnicity, treatment setting, and cancer type, multiple linear regression models were estimated. Additionally, logistic regression models were used to examine associations between chronological age and the dichotomized WRAT-WR scores.

Aim 3: Assess if there are racial/ethnic differences in WRAT scores

Using Chi-square tests, unadjusted differences between a child’s race/ethnicity and their literacy level (below average vs. average/above average) were examined. Multivariable logistic regression models using the dichotomized literacy level variable were estimated, controlling for the child’s age, gender, race, ethnicity, treatment setting, and cancer type. Adjusted odds ratios (AORs) and 95% confidence intervals (CI) were calculated. To determine if the relationship between race/ethnicity and literacy level varied by age, a sensitivity analysis that stratified models for children younger than 13 years and adolescents 13 years and older was conducted. The age of 13 years was selected for stratification to capture the beginning of the adolescent age groups where standardized WRAT-WR scores began to fall below the expected level for age with increased frequency.

Statistical analyses were performed in STATA Version 13.1 with two-sided statistical tests and a significance level of 5%.

Results

Participants

One hundred and forty children participated in the literacy sub-study between February 2014 and December 2016. Five children were excluded from the study as their WRAT assessment sessions ended prior to meeting established stopping rules for the instrument scoring. Participant demographic characteristics for 135 children are summarized in Table 1. The sample was well distributed across ages 7–20 years and diverse in demographics (e.g., 47% male, 47% non-white) and cancer type.

Table 1 Participant demographics
Full size table

Younger children, ages 7 or 8 years, were more likely to request assistance with reading part, or all, of the Pediatric PRO-CTCAE instrument (63% of 7 year olds and 12.5% of 8 year olds). For the children requiring assistance with reading, the average (raw) WRAT-WR score was 27 which loosely equates to a second-grade reading equivalency. Items that equated to higher literacy demand, contained more complex sentence structures, for example, questions that asked about interference with normal activities. Individual words included in the Pediatric PRO-CTCAE were generally understandable for most 7–9 year olds after words were changed based on initial cognitive interviews. Children who needed assistance with reading individual words often expressed understanding of the meaning of the word upon hearing the word verbally stated.

WRAT scores

No notable or statistically significant differences were identified between the original WRAT-WR subtest scores and the verified scores using a sensitivity analysis. Raw WRAT-WR scores ranged from 11 to 67 on a scale of 0–70. Standardized WRAT-WR scores ranged from 65 to 145 (mean 103; standard deviation 16.5). Table 2 provides a breakdown of standardized scores by age and descriptive score categories.

Table 2 Standardized WRAT reading scores by age (in years)
Full size table

Aim 1: Literacy level and understanding

The majority (87%) of younger children (7–9 years) had standardized WRAT-WR scores that were appropriate for their age. Within this age group, there was one outlier, an 8 year old that displayed an exceptionally advanced reading skill (raw score 59; standardized score 145) which roughly equates to a 12th grade reading equivalency. Similarly, 82% of children 10–15 years had standardized WRAT-WR scores that were average or above average for their age. However, only 52% of participants aged 16–20 years had WRAT-WR scores that were consistent with their chronologic age (Table 2).

When examining the association between literacy scores and understanding, higher raw WRAT-WR scores were significantly associated with fewer HTU items in both unadjusted and adjusted models (p = 0.017, p = 0.033, respectively). On average, a one-unit increase in raw WRAT-WR scores was significantly associated with a (0.0278) and (0.0342) decline in the number of items marked HTU, in unadjusted and adjusted models, respectively. In these models, a decision was made to exclude an outlier (8-year-old child with exceptionally high WRAT-WR score).

Aim 2: Literacy level and age

Literacy level and age were highly correlated among participants in our study. There was a statistically significant association between standardized WRAT-WR scores and a child’s age, before (β − 0.804, p = 0.045) and after adjusting for race, ethnicity, gender, treatment setting, and cancer type (β − 0.837, p = 0.042). When we examined unadjusted associations between age and the dichotomized literacy level, we found a 1-year increase in age was significantly associated with a 19% increase in the likelihood of being below the expected (average) WRAT-WR score (OR 1.19; 95% CI 1.06–1.33, p = 0.003). Once potential confounders (gender, race, ethnicity, treatment setting, and cancer type) were controlled, we continued to see a statistically significant association between age and likelihood of being below average (aOR 1.23; 95% 1.09–1.37, p = 0.001).

Aim 3: Race/ethnicity and literacy

There were no statistically significant associations between race/ethnicity and being below average for standardized WRAT-WR scores in unadjusted (p = 0.823) and adjusted models (p = 0.254). In models examining children < 13 years of age and adolescents 13 years and older, we did not observe statistically significant associations between race/ethnicity and being below average. These findings were consistent when we examined race as a 3-category variable (White, Black or Other) as well as when we dichotomized race (White vs. Other).

Discussion

Clinical trials include children of all ages making it important to have PRO measures that have been validated for use in children and in compliance with the Food and Drug Administration guidance for PRO measure development [23]. Measures which are valid in children are needed for use in pediatric oncology clinical trials to ensure accuracy of adverse event reporting, but are also needed in the clinical setting to appropriately monitor and treat symptoms with the ultimate goal of improving quality of life indicators. The findings from this study are therefore highly relevant as they provide evidence regarding the influence of literacy on pediatric PRO self-report. Overall, we observed that children with cancer have wide variations in literacy skills across ages. In general, younger children (ages 7–8 years) who needed assistance with reading had WRAT-WR scores that were at or below a second-grade reading level. This finding is consistent with the age ranges presented in a systematic review of instruments validated for use in pediatric oncology showing that a large percentage of instruments begin written data collection around 8 years of age [24]. It has also been noted that the reliability and validity of self-report PROs in children improve around this age [25]. However, age should not be the only indicator of a child’s ability to independently self-report as 63% of 7 year olds and 12.5% of 8 year olds in our study required interviewer assistance with reading.

Some young children (ages 7–9 years) demonstrated advanced reading skills that were above the reading levels of older adolescents. Lower literacy scores, but not chronologic age, were significantly associated with the number of PRO-CTCAE items that a participant marked as hard to understand (HTU). This finding suggests that difficulties in understanding the Pediatric PRO-CTCAE may be influenced by literacy. Other possible explanations for items being marked as HTU may perhaps be related to developmental issues such as concentration and/or self-awareness. Our finding suggests that the Pediatric PRO-CTCAE was able to be completed by children across a wide literacy span. This is important as measurement experts recommend consideration of literacy skills and developmentally appropriate vocabulary during PRO instrument development [25].

Older children (16–20 year olds) in this study were more likely to have standardized WRAT-WR scores below the expected scores for their age. This finding is not unique to adolescents with cancer. In the general population, 63% of 12th graders test below a proficient achievement for their grade level in reading, with 38% scoring at a “below basic” achievement level [26]. Previous studies found that although the rate of growth for reading fluency increases with grade level, the rate of growth decreases with age, especially after second or third grade [27]. However, we found that older children, even those with lower WRAT scores, had little to no trouble independently completing the adult version of the PRO-CTCAE instrument. This observation suggests that a developmental element, in addition to literacy skills, may be associated with reading and interpreting questions related to health status. Vocabulary knowledge increases with age/grade level and these skills are associated with reading comprehension, [8] which may help older children reason out the meaning of more complex questions. It is also plausible that exposure to a treatment-intensive illness, like cancer, provided introduction to medical terms and health-related vocabulary which would normally be absent during childhood. An expanded health vocabulary may have assisted with providing a contextual reference for the questions presented in the Pediatric PRO-CTCAE.

Results from multiple rounds of cognitive interviews led to the recommendation that adolescents younger than 18 years of age use the pediatric PRO-CTCAE instrument, but also noted that the adult version of the instrument proved valid and could be utilized, as needed, in adolescents as young as 16 years of age [21]. The findings from our literacy study support these recommendations. Increasing literacy with age may also help to explain why older children (16 years and up) with lower WRAT scores were able to read and respond to the adult version of the PRO-CTCAE instrument which had a higher literacy demand than the Pediatric PRO-CTCAE. A Flesch–Kincaid reading level analysis demonstrated a difference in overall reading level between the Pediatric PRO-CTCAE and the adult PRO-CTCAE instruments, with the adult version scoring an estimated 2 grade equivalency levels higher (mean 6.5 reading level equivalency) and displaying a higher variability in the range of literacy demand among individual questions in the item bank. Older participants, ages 16–20, had higher literacy skills in general (mean 8.1 grade reading equivalency) which supports their ability to transition to the PRO-CTCAE instrument without jeopardizing validity. Cognitive testing also demonstrated the ability of older children (ages 16+) to adequately understand the adult PRO-CTCAE instrument [21].

Prior studies reported decreased cognitive and/or academic abilities among children treated with cranial radiation, [13, 28] especially for those treated at an early age [29]. One study examining children treated for brain tumors reported that children performed better on tests focused on understanding of material (reading comprehension or word comprehension) as opposed to other tests examining spelling, reading speed, or basic arithmetic skills [30]. Other studies suggest that chemotherapy alone, especially with intrathecal administration, may be associated with cognitive impairment [31]. Although we did not observe notable differences in WRAT-WR scores by cancer diagnosis, our sample contained relatively few central nervous system (CNS) tumors (n = 5) and we did not stratify by ± cranial radiation.

Our study primarily included childhood leukemia patients. Prior studies in acute lymphoblastic leukemia (ALL) support academic and language outcomes post treatment that are comparable to healthy controls, especially in ALL patients treated on lower (standard) risk protocols without cranial radiation [28, 32, 33]. Our study did not observe any significant difference in literacy level by race/ethnicity (White vs. Blacks vs. Hispanics). This is not in agreement with results from other national studies which have found disparities in academic (reading and math) abilities by race/ethnicity in school age children [34, 35]. Some experts argue that socio-economic status accounts for over half of observed differences in racial achievement gaps [36] and should be evaluated, yet socio-economic status was outside the scope of this study. Other publications have reported differences in academic achievements by race, in healthy children, even when considering economic inequalities [16].The results from our study, however, can be interpreted to mean that race was not a factor in children being able to complete the Pediatric PRO-CTCAE.

Limitations

Several limitations are noted for this study. Bias in study enrollment could have occurred as participants with lower literacy or an aversion to reading may have declined to participate in a study that involved answering questionnaires. Also, the cognitive interviews took between 30 and 60 min so it is possible that the sickest children were excluded as they may have declined to participate in an interview of this length. Additionally, few other studies examining literacy in children actively undergoing treatment for cancer have been conducted and none have reported the WRAT-WR subtest scores separately. As such, we were unable to compare our findings to previously published results. It is also possible that cognitive effects and decreased academic performance are more common as a late effect of cancer treatment and all of the children participating in this study were fairly early in their therapy. Additionally, the numbers of children in minority racial/ethnic categories were relatively small which may have limited our ability to detect associations with literacy. Lastly, our study examined literacy in relationship to a specific PRO instrument and thus results may differ if repeated using a PRO measure that is more challenging for children to understand. Despite these limitations, our study generated new evidence related to the influence of literacy and age on PRO completion in pediatric oncology patients.

Conclusions

This study offers important insights into the relationship among age, literacy, and PRO completion in children with cancer. Our findings underscore the importance of considering literacy regardless of a patient’s age. The exploration of literacy skills in conjunction with cognitive interviews provides additional support for the validation of the Pediatric PRO-CTCAE and its use with pediatric oncology patients. Literacy consideration during instrument development is an important step and should be considered the best practice for fully evaluating the understandability of PRO measures in specific populations. As this study included children (7–20 years) receiving active cancer therapy, it provides valuable information related to changes in literacy with age. Children with less than a second-grade reading equivalence (most 7 year olds) requested assistance with reading some or all of the measurement. Children with emerging reading skills were still able to complete the Pediatric PRO-CTCAE instrument when reading assistance was provided suggesting that comprehension is influenced by vocabulary and prior health experiences. This finding implies that children with lower reading skills may benefit from having an audio version of the instrument available. As PRO measures provide data for use in clinical care (symptom management) and have important implications for drug labeling and toxicity reporting within clinical trials, it is imperative to consider the influence of literacy on child PRO measures which utilize written data collection.