Introduction

Health systems aimed at improving child health are facing a formidable challenge due to the rising prevalence of chronic conditions and complexity of patients with multiple comorbidities [1,2,3,4]. Patient-reported outcomes, and experiences, as well as their measures (PROMs/PREMs) are essential to health system improvement on the basis of child and family-oriented care [1, 2]. PROM measures capture the outcomes important to growing up well beyond biomedical indicators, such as mental health, mobility, and quality of life. Experience measures (PREMs) allow systems to capture the extent to which health care was perceived as family-centered, coordinated, and useful [5]. Both PROMs and PREMs are vitally important to quantifying the successes and failures of health interventions as well as resource planning and health policies [3, 6]. Yet PROMs/PREMs are rarely used in large scale studies, thereby rendering the majority of robust evidence on which health decisions are made, to have omitted child and family-important outcomes and experiences completely.

Omitting PROMs and PREMs from otherwise useful research studies (like clinical trials) excludes children and family voices from policy decisions but also serves as a barrier to shared decision-making [7,8,9,10,11]. For example, a family considering a high-stakes medical intervention such as epilepsy surgery, or medical device insertion, requires answers from health providers about the likelihood of good outcomes in order to make decisions [12,13,14]. In such cases, clinicians often have evidence on the outcomes of mortality and safety but not long-term functioning and quality of life [15, 16]. Thus, families and providers are forced to make important clinical decisions based on incomplete information. The lack of inclusion of PROMs/PREMs in the health services research thus also impacts routine clinical care [17, 18].

The degree to which PROMs and PREMs are used in health services and systems research is understood more than the relevance of that use to children and families from the first place. A PROM selected by researchers or clinicians might not be measuring the outcomes that are meaningful to the target population being studied [7, 19, 20]. For example, utility-based attributes relevant to general child population-health (walking, speaking) were incorrectly and unethically selected by researchers for decades to assess the quality of life of children with cerebral palsy who were never expected to walk or talk, but nonetheless had potential for functional mobility and communication [21,22,23]. Examples of the inclusion of children and families in the selection of outcomes and experiences that are relevant to them appear to be increasing in the literature [6,7,8, 18, 24, 25]. But it is unclear whether children and families are influential in the selection and development of the PROMs and PREMs that will be incorporated into the generation of knowledge needed for systems change.

In response to the need for appropriate inclusion of patients in the measurement process, the Food and Drug Administration (FDA) [26] and European Medicines Agency (EMA) [27] have created guidelines related specifically to the development and use of outcome measures [26, 27]. Both document the need for clear evidence that the content of outcome measures needed to correspond to important domains of concern in the target patient population and that the thresholds of meaningful change were influenced by patient input. These guidelines were one more assertion in the growing acceptance that inclusion of children and families in measurement selection and development is not merely ethical or value-based imperative but germane to rigor in health research.

Once the issue of meaningful child and family inclusion in the selection and development of PROMs and PREMs is recognized as an issue influencing rigor, a systematic understanding of the levels of child and family engagement that currently exist is needed. The International Association of Public Participation (IAP2) Spectrum of Public Participation is a framework that can classify various levels of patient participation [28]. It was chosen in lieu of other models of public participation due to its focus on categorizing levels of participation from low (inform) to high (empower) in place of the methods that could be used to engage patients [29,30,31] in the research selection and development process. This review aims to classify the current level of child or family engagement in the selection and development of PROMs/PREMs in child health research using the IAP2 Spectrum of Public Participation [28].

Methods

Search strategy

We systematically reviewed three online databases: PsycINFO, MEDLINE, and EMBASE. The search consisted of keywords linked by Boolean terms pertaining to ‘selection’ AND ‘outcomes’ AND ‘patient-reported outcome/experience measures’ AND ‘children/families’ AND ‘patient engagement.’ Search terms were expanded based on these key themes and synonymous terms were identified (Online Appendix A). A Queen’s University health sciences librarian worked with the primary reviewer to establish a comprehensive search strategy. Searches were limited to articles published in English. Articles were only considered if published between December 2009 and September 2018. December 2009 is in accordance with the publication of the FDA and EMA guidelines for PROMs. The review was interested to see at what level of engagement PROMs/PREMs were developed after the guidelines were created. Existing PROMs that were developed for the use in children were included in this review if they were created after the introduction to the FDA guidelines. Articles involving children, adolescents, or proxy reporting for this cohort were considered for inclusion. The review was conducted using the PRISMA guidelines [32].

Study selection

Covidence software was utilized throughout study selection [33]. Covidence is a platform used to streamline and optimize systematic review processes. Users of the software can import articles to be screened across the systematic review process. Covidence allows multiple authors to collaborate on the screening of abstracts against developed inclusion criteria. The primary reviewer screened article titles during phase one of study selection. Phase two of study selection involved screening of titles and abstracts; this was accomplished by the primary reviewer and two secondary reviewers. The primary reviewer screened 100% of titles and abstracts; two secondary reviewers initially screened a random selection of 25% of titles and abstracts. Discrepancies between reviewers were resolved via consultation of a third uninvolved co-screener. See PRISMA flow diagram for synthesis of included articles (Fig. 1).

Fig. 1
figure 1

PRISMA flowchart

Full size image

Inclusion criteria consisted of (i) articles pertaining to the selection of outcomes OR development of child and family-specific PROMs/PREMs AND (ii) pertaining to a health care setting. Both condition-specific and generic PROMs/PREMs were chosen to be included in the review as this study focused on engagement across the selection of outcomes and development of measures irrespective of conditions. Studies were either categorized as ‘Selection’ or ‘Development.’ Included studies that related to the identification or prioritization of meaningful outcomes/outcome sets were categorized as ‘Selection.’ Studies rooted in selection did not identify how outcomes should be measured. The focus of selection was to select and achieve a consensus of standardized outcomes which are deemed relevant to patient populations and should be utilized within measures. Furthermore, studies aiming to identify and select appropriate PROMs/PREMs for child and family populations were included in this category. Studies that included methods of developing measurement tools (PROMs/PREMs) to be used alongside children and families were categorized as ‘Development.’ Studies in the ‘Development’ category focused on the creation of these measures and the methods utilized across the development process.

This review focused on the use of PROs and PROMs used in pediatric populations; the development of instruments related to family function or caregiver health was not included. Articles were excluded if they were (i) not involving children or families, (ii) conference proceedings, or (iii) pertaining solely to the validity testing or translation of an existing measure. The IAP2 Spectrum and modified examples cannot appropriately be applied to studies which focus solely on validity testing or translation of existing measures. Conference proceedings were excluded as they do not provide enough information for engagement or selection to be accurately mapped on to the IAP2 Spectrum. The reference lists of included studies were searched manually by the primary reviewer for consideration of inclusion.

Data extraction

Defining engagement with the IAP2 public participation spectrum

Data extraction was carried out solely by the primary reviewer. This systematic review used a modified patient engagement spectrum published by the International Association of Public Participation (IAP2) to quantify levels of child and family engagement in the current literature (Fig. 2). The IAP2 is an international organization with the overarching mission of improving public participation across individuals, governments, and institutions [28]. The IAP2 regards ‘The Spectrum of Public Participation’ as a pillar of public engagement; it is an internationally used tool to assist in the selection of levels of involvement. The spectrum divides engagement into five distinct categories: Inform, Consult, Involve, Collaborate, and Empower. Modified examples were created for each level of engagement to be made applicable to the selection of outcomes and the development of measures.

Fig. 2
figure 2

IAP2 Spectrum of public participation

Full size image

Included articles were analyzed using the IAP2 Spectrum. A data extraction form was created based on the Cochrane Data Extraction and Assessment Template. Information obtained from each article included (i) article title, (ii) author, (iii) measurement name if applicable, (iv) study type, (v) selection of outcomes or development of measures, (vi) methods of engagement, (vii) stages of development/selection in which patients were engaged, and (viii) IAP2 level of engagement. Specific methods of engagement, such as the use of interviews or focus groups with participants were included in the data extraction form to ensure consistency throughout data analysis. The data extraction form allowed data to be coherently presented to the researchers and categorized according to the IAP2 Spectrum.

Studies included in the review were categorized according to IAP2 levels of engagement and a set of definitions and examples made specific to patient-reported outcome (PRO) selection and PROM/PREM development developed by researchers of this review (Online Appendix B). Engagement is categorized into five succinct categories, from the least to greatest level of participation (Inform, Consult, Involve, Collaborate, and Empower). The IAP2 Spectrum provides promise to the public in lay terms for each level of engagement to increase public understanding. The developed definitions range from no engagement (Inform) to children and families leading outcome selection or measure development (Empower). The remaining three categories range from limited involvement within one particular step of outcome selection or measure development (Consult), the iterative involvement of children and families across more than one step (Involve), and children and families having a clearly defined role as co-conductors of research (Collaborate).

Results

Included study characteristics

Thirty-six studies met inclusion criteria. Seven studies focused on the selection of outcomes and 29 studies pertained to PROM/PREM development. Table 1 provides a breakdown of each article in terms of first author, year, article context (i.e., select outcome or develop measure), IAP2 level of engagement, methods of engagement, and stage of engagement. Specific measure names are provided for articles that used them (see Online Appendix C). The most common IAP2 level of engagement across included studies was ‘Involve’ with 22 of the articles falling under this category, further defined by the iterative involvement of patient perspectives and concerns. Three studies were classified as ‘Inform’ with no engagement and seven studies were classified as ‘Consult’ with limited engagement. Four studies adhered to the criteria for ‘Collaborate’ in which children and families had clearly defined roles and were partners in the co-conduction of research. None of the included studies matched the criteria for ‘Empower’ in which children and/or families lead outcome selection or measurement development and act as final decision-makers. The categorization of studies is represented in Table 2. The systematic selection of articles is presented in a PRISMA diagram (Fig. 2). The most common reasons for the exclusion of included studies were as follows: (i) articles not pertaining to outcome selection or measure development and (ii) participants not involving the target cohort.

Table 1 Included article summary
Full size table
Table 2 Summary of levels of engagement across included studies
Full size table

Methods of engagement

Methods of engagement varied across included studies. The following engagement methods were assessed for use across included articles: (i) interviews, (ii) focus groups, (iii) cognitive debriefing, (iv) concept elicitation methods, (v) co-recruitment, (vi) co-data collection, (vii) co-data analysis, (viii) co-dissemination, (ix) family driven outcome selection, (x) included in multiple steps and stages, and (xi) key decision-makers across crucial steps.

Participants were most frequently engaged via semi-structured interviews or focus groups. Five of seven articles pertaining to the selection of outcomes utilized these methods to elicit patient preferences for meaningful outcomes [6, 40, 45, 49, 56]. Nineteen of 29 articles with the aim of PROM/PREM development involved participants through focus groups or interviews throughout initial concept elicitation [34, 35, 37, 45,46,47,48, 51,52,53, 55, 57,58,59,60,61, 64, 65, 67]. Concept elicitation interviews were conducted to identify common themes and concepts which were deemed meaningful to children and families. The data collected through concept elicitation interviews or focus groups served as the basis for item generation to be used in developed measures. Ten articles which developed PROMs/PREMs did not engage participants through initial concept elicitation and the first draft of items were constructed based on expert consensus or through a review of existing literature [36, 38, 39, 41, 43, 44, 52, 62, 63, 68].

‘Informing’ children and families

Two of the studies in this category utilized a review of the literature followed by Delphi consensus rounds with experts to select and prioritize outcomes [42, 66]. One study relied solely on existing literature to develop a measure and piloted the measure with the target population, without further refinement [44]. The studies under ‘Inform’ did not directly reach out to children and/or families to obtain their preferences or perspectives relating to the selection of outcomes or development of measures.

‘Consulting’ children and families

Studies that only engaged children and/or families within one stage of selection or development with limited implementation were categorized as ‘Consult,’ All studies categorized as ‘Consult’ lacked the iterative process of engagement across several steps. Experts led the research process and target populations had limited decision-making power. Seven studies included in this review were categorized as ‘Consult,’ six of these focused on the development of PROMs/PREMs [36, 38, 39, 41, 43, 57], and one focused on the selection of outcomes [40].

To ‘Consult’ children and families across the selection of outcomes, focus groups were conducted alongside populations to identify relevant health-related quality of life outcomes [40]. Further refinement and analysis of collected data were conducted entirely by experts [40]. Outcomes identified as important by children and families across focus groups were refined by experts and compared to existing measures to identify gaps in the literature [40].

Most studies at the ‘Consult’ level developed items based on existing literature and consulted children and/or families through methods of item reduction or item refinement [36, 38, 39, 41, 43]. One study engaged children in focus groups for concept elicitation and researchers completed the subsequent analysis of transcripts and generation of items [57]. Children and families were engaged in methods to further refine items and ensure clarity through cognitive debriefing. For example, children were instructed to read the developed items out loud to identify any areas of difficulty within items [43]. Other item reduction or refinement strategies at this level included statistical methods, such as the removal of items based on the frequency of missing responses when piloting developed measures [39], and Rasch analysis methods to determine the appropriateness of items [41].

‘Involving’ children and families

Twenty-two studies categorized as ‘Involve’ worked directly with children and families in an iterative process across several steps of outcome selection or PROM/PREM development. These studies clearly engaged children and families across two or more stages of the selection or development process.

The articles which focused on the selection of outcomes consisted of two stages of selection. Both articles involving children and families across outcome selection conducted focus groups to identify which outcomes were important to patients [45, 49]. Dyson et al. conducted a quantitative survey of outcomes with regular parent input to determine which outcomes were important for children with pediatric acute respiratory illness [45]. Outcomes included in the survey were determined by a literature review conducted by professionals [45]. Follow-up focus groups were conducted with the target population to reach a consensus on important outcomes [45].

Franciosi et al. used focus groups and interviews to elicit child and parent preferences regarding outcomes [49]. A priori outcome domains were established through a review of the existing literature to identify themes of focus group discussion [49]. Interviews were conducted with both parents and children and were open-ended in nature to prompt discussion [49]. Interviews were refined alongside parents when analyzing participant interview transcripts [49].

Most of the ‘Involve’ studies that engaged children and families through measure development utilized a combination of interviews or focus groups for concept elicitation paired with further cognitive debriefing for item clarification and refinement [34, 35, 37, 46,47,48, 50, 51, 53, 55, 58,59,60,61, 64]. This iterative approach allowed participants to determine common themes for items and ensured they were comprehensible for children and/or families after cognitive debriefing. When paired with concept elicitation, cognitive debriefing strategies allowed children and families to have increased decision-making power on what was to be included in final measures. Furthermore, it was common for patients to complete existing measures to identify concerns with widely used tools and apply findings to measure development.

‘Collaborating’ with children and families

Four articles were categorized as ‘Collaborate,’ in which authors co-conducted research with children and families across the selection [6, 56] or development [54, 67] process. The two articles which focused on the selection of outcomes collaborated with children and families to identify and prioritize meaningful outcomes to be measured. Both of these articles included children or families as co-applicants for funding and included these stakeholders in regular team meetings [6, 56].

Morris et al. is a peer-reviewed and published protocol that outlines an ongoing study that has begun recruitment [56]. Morris et al. described a protocol for conducting a systematic review of the literature followed by the rating of importance of identified outcomes [56]. The researchers will facilitate three Delphi consensus rounds with children and families to refine and prioritize outcomes [56]. Children and families will be allowed to suggest new outcomes to be included in Delphi rounds if they were not identified by the systematic review [56]. A final meeting, which will include multiple stakeholder groups will be used to further refine outcomes using small plenary discussions and card sorting techniques [56].

Allard et al. [6] conducted a qualitative study alongside children and families to select key health outcomes for children with neurodisability. A research advisory board was established with parents of child patients [6]. Focus groups and semi-structured interviews were conducted with children and their parents to identify their perspectives related to important outcomes [6]. Further interviews with parents included the completion and rating of current PROMs/PREMs and whether they were important [6]. Child participants were shown photographs of an imaginary child and were prompted to identify meaningful outcomes [6]. Children used a ‘Talking Mat’ approach consisting of cartoon images on cards, displayed on a board, to prioritize health outcomes in order of importance [6]. Interview rounds using this method were repeated until consensus was reached [6]. Professionals created a framework for analysis based on focus group transcripts and the youth version of the ICF [6].

Youth were engaged throughout the co-construction of items, co-refinement of data, and cognitive debriefing across the studies related to measure development. Development was divided into succinct stages with researchers acknowledging and acting upon the perspectives of children and families. Focus group rounds were employed across both studies pertaining to concept elicitation, the refinement of items, and cognitive debriefing to ensure item clarity [54, 67]. Both development articles conducted focus groups across at least three stages of measure development [54, 67]. The analysis of focus group data was typically conducted by researchers [54, 67]. One study analyzed data based on the content areas of an existing framework [67]. A different study analyzed data using an inductive framework which was created using child responses [54].

In one study, the collaboration with children and families across development was defined by the creation of a ‘youth team’ to co-conduct research with professionals [54]. Focus groups were conducted with children, by children and researchers. The perspectives of the youth team were acknowledged and considered across the stages of measure development [54]. Furthermore, researchers utilized creative methodologies to convey the perspectives of children and families [54]. Children were prompted to take photographs and record their daily routines [54]. Recordings were played back to children and they were further probed to discuss barriers to participation identified in the recordings [54]. The identified barriers were used to further shape items to be included in measurement tools.

One study engaged children and families on their preferences relating to potential formats and modes of administration of measures through focus groups [67]. Children were separated by age groups and preferences were elicited relating to presentation [67]. The final developed PREMs were pilot tested in a large hospital setting to identify real-time barriers to PREM completion [67].

‘Empowering’ children and families

No articles included in this review were categorized at the ‘Empower’ level of engagement because there were no methods described which involved processes where children and/or families led the selection or development process and had final decision-making power. Table 2 provides a summary of levels of engagement across articles pertaining to the selection of outcomes or development of measures.

Discussion

Patient engagement of children and families in self-report measures is currently at an ‘involve’ level, according to the IAP2 Spectrum. The ‘involve’ level was mainly operationalized through the use of qualitative interviews for measurement content development. It appears that since the publication of regulatory guidelines [26, 27], patient involvement in content development is being implemented in contrast to ‘expert-driven’ methods that characterized the development of earlier legacy measures [69, 70]. This positive shift towards improving the relevance of measures to families has not come at the expense of clinical indicators [71,72,73,74,75] but merely enhances them. The regulatory guidelines that led to this shift towards patient involvement was based in an onus by developers to demonstrate a strong overlap between measurement content, measurement thresholds, and target populations. The guidelines did not require or recommend patients to be involved in the selection of outcomes or in psychometric validation, which remains a missed opportunity to improve the rigor and quality of self-report measures on the basis of child and family participation.

Where there are no specific guidelines on the engagement of selecting PROs and developing their measures within the FDA and EMA guidelines beyond content development, there are opportunities to enhance relevance of research through the co-selection of outcomes between children and families and researchers so that the evidence generated is meaningful to them [7, 19, 25]. Yet this remains a rare exception rather than the rule in research that is used for decision-making such as clinical trials [11]. When outcomes are selected by researchers and clinicians, this does not occur, researchers may insert their own biases into the selection of outcomes used, which then appear as success markers in clinical trials with children [11]. On the development side of participation, there were missed opportunities for children or families to collaborate on areas that impact upon psychometrics such response options and overall formatting and data collection methods. When this was studied, the results were highly informative to scale structure and form [67].

There is an evidence-driven value and impact to collaborating with children and families during measurement [6, 54, 56, 67]; yet there are also barriers to their engagement. Engagement increases the number of steps involved in the development process and hence the costs [76]. Also, depending on the number and level of involvement of the target population, generating participation can be difficult when the topic of measurement seems nebulous relative to intervention. Some of these practical issues such as getting participants to participate may be overcome through the use of creative or novel methods that can promote receptivity. This review found that studies with higher levels of engagement utilized targeted methods to act upon the specific barriers to inclusion, such as arts-based cues for children [6], or photovoice for families [54], and in doing so, promoted receptivity [54]. Children and families can also provide suggestions for how to make their inclusion more feasible and compelling [76].

An outstanding issue remaining at the conclusion of this review is the question as to: “what level of patient engagement should be pursued in the measurement process and why?” ‘Collaborate’ was the highest level of engagement identified and its feasibility demonstrated by exemplars such as ‘youth teams’ who partnered at all phases of decision-making to children facilitating focus groups on their own behalf [6]. Empower levels of engagement in the selection of PROs and development of measures were not found in the literature likely due to the technical expertise required to select and develop PROMs/PREMs. Some examples of empowered measurement selection have been intimated in the adult literature as seen through the prioritization of research areas and the selection of outcomes for patient-driven trials [77, 78]. Yet the outcome of this review has highlighted the importance and value of engagement to enhance the relevance, rigor, and uptake of evidence generated by measures overall. An example of child and family collaboration that reduced the relevance, rigor, or uptake of PROMs and PREMs research was not found.

Conclusion

Reframing children and families from participants of outcomes research, to stakeholders with the potential to co-conduct research, is required to reach a consensus on accurate outcomes and to shape effective measurement tools. Research conducted “for children and young people, by children and young people” [54] has the potential to increase the impact of interventions when outcomes are in alignment with the needs and desires of children and families.

Findings of this review demonstrate that the engagement of children and families across the selection of outcomes and development of PROMs/PREMs varies. Engagement efforts were primarily focused on concept elicitation and item refinement. Although engaging with patients is often valued as integral to research aiming to capture the accurate perspectives of children and families, more collaborative engagement strategies during outcome development remain under-explored.