Introduction

The consensus of recent studies in the USA, Europe and in other parts of the world indicates a prevalence rate for autism spectrum disorders (ASD) of at least 1% (Durkin et al. 2010; Elsabbagh et al. 2012; Hansen et al. 2015; Christensen et al. 2016). Given this high prevalence, the World Health Organization (WHO) has declared ASD to be a global public health concern that requires appropriate prioritisation in member countries (WHO 2014).

The WHO resolution on ASD recommended that appropriate access to assessment and intervention should be made to families, emphasising and including psycho-education to parents and carers (WHO 2014). In addition, the resolution recommended contextually-relevant research on these aspects of ASD. The recommendation for psycho-education and support to families is based on the fact that the level of need in ASD families represents some of the greatest burden of all disabilities (Cutress and Muncer 2014). This type of parent support and intervention is therefore considered part of best practice, in particular, shortly after diagnosis, to form the foundation for future positive advocacy and empowerment of families.

‘Parent Training’ is a term used in the ASD literature to cover a range of interventions and supports aimed at parents (Aman et al. 2009; Oosterling et al. 2010; Beaudoin et al. 2014). Bearss and colleagues provided a useful framework and taxonomy and suggested that parent training could be divided into two categories, i.e. ‘Parent Support’ and ‘Parent-Mediated Interventions’ (Bearss et al. 2015). Parent support includes psycho-education and care coordination, where the parent/carer is the direct beneficiary, and the child with ASD the indirect beneficiary of the intervention. Parent-mediated interventions refer to intervention provided by parents to their children with ASD, with the child therefore being the direct beneficiary of the intervention. Parent-mediated interventions might target core features of ASD (e.g. joint attention, communication, imitation, turn-taking) or focus on maladaptive behaviours (e.g. disruptive behaviours, sleep, feeding or toileting difficulties). The majority of research investigating parent training, as defined by Bearss et al. (2015), has been on parent-mediated interventions to date (Green et al. 2017; Watson et al. 2017; Nevill et al. 2018). A range of research and reviews have targeted the efficacy of parent-mediated interventions (Brookman-Frazee et al. 2009; Medan et al. 2009), the impact of such interventions on specific outcomes (Lang et al. 2009), or on particular methodological approaches like randomized controlled trials (Oono et al. 2013). There is therefore significant on-going research exploring parent-mediated interventions. In contrast, parent support has received much less attention in the scientific literature in spite acknowledgement of the value of psycho-education, in particular its comparative cost-effectiveness (Schultz et al. 2011; Bearss et al. 2015).

To add a layer of complexity, many ASD-specific parent programmes have multiple goals that include both supporting parents/carers and providing them with education or skills. For this reason, many programmes might not fall exclusively into the ‘parent support’ or ‘parent-mediated intervention’ categories described by Bearss et al. (2015) and may therefore be regarded as ‘hybrids’. In a review of one such ‘hybrid’ programme, the EarlyBird/EarlyBird Plus programme developed in the UK (Dawson-Squibb et al. 2019a), the authors proposed use of the term ‘Parent Education and Training’ (PET) as a placeholder to refer to such programmes. They defined PET as the passing on of information or skills to parents using a range of modalities (e.g. didactic, role-play, discussions, video guidance) in a context where parents/carers and trained facilitators are the direct participants. The emphasis is on ‘knowledge transfer to parents/carers and the priority participants are parents/carers and not the parent–child dyad’ (Dawson-Squibb et al. 2019a, p. 2). This definition implies that the child is not present in the session/consultation room while the intervention is being delivered. Such a definition allows for the content of a PET programme to be broad ranging. It could include parent/carer-focused goals (e.g. parent well-being, self-efficacy, or knowledge), family-focused goals (e.g. family quality of life), child-focused goals (e.g. helping a child’s communication development) or specific skills training goals (e.g. managing of behaviours that challenge) (Dawson-Squibb et al. 2019b). PET programmes are regarded as important and empowering first steps to guide, support and educate parents, while fostering acceptance and providing skills.

Using the above definition research on PET in ASD is relatively limited and only a few pilot studies have examined PET as stand-alone interventions for ASD (Bearss et al. 2015). There have been some initial attempts to review and synthesise this literature. Schultz et al. (2011) reviewed ASD PET studies in the USA, and identified 30 such studies in the 1987–2007 period. Eighty percent (80%) of the PET were delivered in a one-on-one format and 20% in groups. Apart from its potential cost-effectiveness, group programs have the added potential of promoting mutual support and opportunities to share personal experience with other parents (Farmer and Reupert 2013). Existing ASD PET have a number of universal objectives, including increasing parental knowledge, enhancing competence in advocating for the child, decreasing parental stress, and reducing parental sense of isolation (Tonge et al. 2006; Farmer and Reupert 2013). Available evidence to date has shown marked reductions in parental stress levels in response to parent education (Koegel et al. 1996), and increased parental knowledge and skills in the areas of managing behaviour and teaching their children communication and social skills (Charlop-Christy and Carpenter 2000; Solomon et al. 2007).

A recent scoping review of all PET programmes outside the USA identified 37 programmes in 20 countries from all continents except South America (Dawson-Squibb et al. 2019b). The programmes were highly variable in characteristics and a range of research methods and outcome measures were used. Few robust conclusions could therefore be drawn. Despite this, and in line with conclusions from the Schultz et al. review (2011), the majority of studies (86.4%) reported positive results in relation to core outcomes. The review highlighted that few of the studies focused on implementation-related factors, including cultural appropriateness, trainer fidelity, manualisation and cost. Whilst there has therefore been some research on PET, the scoping review highlighted that, in comparison to parent-mediated interventions, the evidence-base for PET is positive, if limited, and that much more robust research is required to fill in the key knowledge gaps highlighted in the review. One of these key gaps is the lack of a pragmatic evaluation framework that could be used to determine the suitability of specific PET programmes for specific setting.

Although different PET programmes generally have similar broad objectives, their structures, content, location, modality of teaching, duration, frequency and approaches are often diverse and adapted to suit different contexts or environments (Steiner et al. 2012). Examples range from a two-week inpatient model for parents in India using experiential approaches (Panchal 2017), to a highly manualised, weekly 3-month outpatient programme in the UK that includes a range of teaching modalities and home visits (Shields and Simpson 2004; Gillespie-Lynch and Brezis 2017). As commented on by Schultz et al. (2011) and Dawson-Squibb et al. (2019b), all PET reviewed reported positive outcomes. However, outcome measures for programmes to date have typically been programme-specific and have used simple pre-post evaluation with non-standardised measures (Schultz et al. 2011; Bearss et al. 2015; Dawson-Squibb et al. 2019a; Dawson-Squibb et al. 2019b). However, the level of evidence for the programmes was relatively low and no randomized controlled trials had been performed. In addition, there was relatively little evidence of research on other factors that may be important for implementation of the programmes, such as integration, expansion or demand (Schultz et al. 2011; Bearss et al. 2015; Dawson-Squibb et al. 2019a; Dawson-Squibb et al. 2019b).

The wide range of contexts where PET may be delivered and the range of formats, intensities and approaches raises the fundamental question about how to evaluate programmes in order to select the most appropriate programme for a particular setting. Programme evaluation has gained increasing interest from a variety of stakeholders including researchers, clinicians and policy-makers, keen to ensure that effective and implementable programmes are being provided to those that need them (CDC 2012; Moore et al. 2014; Kuravackel et al. 2018). To date no published or generally-accepted guidelines for the evaluation and selection of ASD PET have been developed.

Implementation science is described as a method of enquiry designed to assist investigators to determine whether interventions or methods can be implemented in real-world settings that may differ in a number of variables from the original setting (Damschroder et al. 2009). More recently the concept of ‘scaling-out’ has been introduced into the field of implementation science research, detailing when evidence-based interventions are adapted to new populations or delivery systems (Aarons et al. 2017). This is differentiated from ‘scaling-up’ where the same intervention is broadened to reach a larger but similar population. The developers of the scaling-out concept suggested that gathering certain empirical evidence about a programme is essential to determine whether it could be successful in a different population (Aarons et al. 2017). Bammer (2005) proposed that a key pillar of implementation science research is participatory methods. This recognises that a range of stakeholders have contributions to make in decision-making and understanding an issue that would be critical when considering factors like scaling-out (Bammer 2005). In line with this Dingfelder and Mandell (2011) have highlighted the importance of diffusion in ASD intervention and research. They emphasized how few efficacious treatments are adopted or implemented successfully in community settings. Efficacious treatments (‘efficacy’) are those shown to work under ideal circumstances and are distinguished from effective treatments (‘effectiveness’) that are beneficial under ‘real world’ settings (Kim 2013). As a solution Dingfelder and Mandel urged researchers to change their practices by collaborating with communities to ensure the adoption, implementation and maintenance of already developed interventions and in the development of new ones (Dingfelder and Mandell 2011).

A number of implementation science research models have recommended an emphasis on stages of implementation (e.g. exploration, initial implementation) and core implementation components (implementation drivers, competency drivers) (Fixsen et al. 2005; Blase et al. 2012; Chambers 2014). Metz et al. (2013) focused on the evaluation of early childhood systems and highlighted four key components of successful implementation, i.e. (1) purposeful selection of an effective early child and education programme, (2) identification of a conceptual framework that guides the implementation process, (3) continuous improvement and data feedback loops, and (4) organised expert implementation teams to ensure an accountable process.

There is consensus that ASD-specific PET programmes are both important and necessary (Schultz et al. 2011; WHO 2014; Bearss et al. 2015). Whilst there are broad implementation models as outlined, the global ASD community currently has no standardised and targeted pragmatic tool to guide the ‘purposeful selection of an effective early child and education programme’, beyond the emphasis on ‘efficacy’ of outcomes. That is, to date there are no literature where stakeholders in ASD PET were asked to identify and prioritise what they might look for in potential PET programmes in order to select a programme most suitable to a specific context. Given the fundamental role of community-based, participatory research in disability studies and in research in culturally-diverse settings, we set out to develop a evaluation framework for ASD PET programmes from the ‘bottom up’ using a multi-stakeholder participatory strategy (Kaholokula 2013). We predicted that multi-stakeholders would identify and prioritise not only ‘efficacy’ criteria, but also a range of implementation science-related criteria. We anticipated that such a pragmatic evaluation framework may have value for programme selection and evaluation across all environments but particularly so in low-resource environments such as in Low- and Middle-Income Countries (LMIC).

Methods

Study Design

We used a participatory research strategy and a consensus-building focus group approach incorporating a broad range of ASD stakeholders to develop a pragmatic evaluation tool for autism-specific parent education and training (PET) programmes.

Participants

Participants were recruited from multi-professional stakeholder groups using purposive sampling. We set out to identify participants with expertise in clinical aspects of ASD (psychiatry, psychology, speech and language, occupational therapy), in education of ASD (special education sector), in social care of ASD (social workers, family care workers), and in implementation science and health systems research. We aimed to have representation from the Health, Education, Social Development, Academic and non-profit sectors and included parents/carers of individuals with ASD. We also prioritised participants with expertise in low-resource settings. We aimed to recruit a group of n > 10.

Procedures

After appropriate ethical approvals from the University of Cape Town (HREC007/2016) participants were recruited and asked to provide informed consent for participation in a consensus-building stakeholder workshop.

A half-day workshop was held at the Division of Child and Adolescent Psychiatry, University of Cape Town, South Africa, where participants gathered and the format and purpose of the proceedings was described by the lead author. Following this, participants were asked to consider individually what they deemed important characteristics of an ASD specific PET programme shortly after diagnosis, particularly in a low-resource setting. They were also asked what barriers, challenges or problems they would foresee with implementing and scaling up such a programme, and were asked to write down their responses individually in the first instance.

Participants were next assigned to three smaller groups of 3–4 individuals each. They were divided into groups with a mix of professions and roles in each group. These groups were asked to discuss their individual responses and work towards establishing key themes. Each group nominated a scribe to collate their answers.

Two invited participants were not able to attend the main workshop. A separate meeting was therefore held by the lead author with these two participants. They were asked to complete the same questions given to the main group and a facilitated discussion between the two of them was conducted by the lead author, and data recorded.

Following the smaller group stage, all the stakeholders were assembled together. A facilitated discussion conducted by the lead author then allowed participants to give feedback both on their individual responses and the smaller group discussions and themes. The themes and discussion generated by the separate fourth group was fed back to the rest of the participants at this stage. During this final stage of the process participants were asked to work towards the generation of a consensus framework that could be used to evaluate ASD specific PET programmes. Through consensus discussions the participants developed a draft Evaluation Framework which all members present agreed upon. The draft Evaluation Framework was subsequently sent to participants after the meeting for comment and feedback to ensure a final consensus document.

Eleven months after the first workshop, a subset of experts who participated in the multi-stakeholder workshop participated in a follow-up workshop to review the framework. This workshop took place in the context of reviewing two ASD PET programmes. The comments and criticisms of the draft Evaluation Framework were discussed and consensus on a final framework was reached.

Measurements

Qualitative data were collected through written feedback for individual and small group stages. Participants were asked to complete written responses to a number of questions that had been developed for the study in their individual capacity (questions available from the authors). The large group consensus discussion also used written feedback, and the lead author collated information and themes generated during the workshop on a flipchart visible to participants and used to facilitate discussion and final consensus. In addition, the large group consensus discussion was audio-recorded and used to ensure accurate record keeping and summative analysis of discussions. No other measurements were used.

Analysis

The data from the workshop, including individually written responses, smaller group themes written by the scribes, written notes on the flipchart and audio recording of the larger group consensus discussion were collected. The Braun and Clarke (2006) 6-step framework was employed and the generated themes were used to do member checking to ensure a consensus interpretation of findings. The workshop process itself was a consensus building one and therefore independent coding by two coders as would be typical for standard qualitative analysis was not performed. Where data were not captured clearly, amendments were made and allowed the stakeholder panel to reach consensus through an iterative process. The data were summarised thematically by the authors as guided by the consensus-generated Evaluation Framework developed at the workshop. The themes that emerged from the multi-stakeholder process were mapped onto the implementation science literature after the data collection process.

Data triangulation was done by including stakeholders at different times in the process and introducing them to data from the groups (Denzin 1978). This allowed the stakeholders to modify and improve the Evaluation Framework.

Results

Participants

Fourteen participants from a broad spectrum of professions and disciplines were recruited to participate in the first workshop. These included two Child and Adolescent Psychiatrists, one Clinical Psychologist, one Educational Psychologist, two Speech and Language Therapists, one Occupational Therapist and a School Deputy Principal. Other participants included a Non-Governmental Organisation (NGO) founder, an NGO director, an NGO counsellor, two health systems researchers and an implementation science researcher.

In addition to their professions stakeholders represented two universities, two schools/ASD centres, two NGOs and three government departments (Health, Social Development, and Education). Two participants were parents/carers of individuals with ASD.

Of the 14, 11 of those participated in the follow-up workshop. Their professions included, one Child and Adolescent Psychiatrists, one Clinical Psychologist, one Educational Psychologist, two Speech and Language Therapists, one Occupational Therapist, and a School Deputy Principal. One NGO director, one health systems researcher and an implementation science researcher also attended.

The Evaluation Framework

A graphic representation of the themes and components of the Evaluation Framework generated is shown in Fig. 1. Three main themes emerged from the stakeholder workshop as guiding principles for the Evaluation Framework. These were ‘Outcomes’, ‘Processes and Procedures’ and ‘Implementation Landscape’. We will outline main findings under the themes below. Under these themes a number of components with several criteria were identified. Components under ‘Outcomes’ included Parent, Child, Family and Community outcomes. Under ‘Processes and Procedures’ components included Accessibility, Acceptability, Psychological Processes and Referral Pathways. Under the theme of ‘Implementation Landscape’ components, were Sustainability, Integration and Coordination, Scalability, and Monitoring and Evaluation. The full list of themes, components and criteria are detailed later in the ASD PET Evaluation Framework Checklist presented in Fig. 2.

Fig. 1
figure 1

Graphic representation of the themes and components of the evaluation framework

Full size image
Fig. 2
figure 2figure 2figure 2figure 2

Autism spectrum disorder parent education and training evaluation framework checklist

Full size image

Theme 1: Outcomes

Four key evaluation components were identified under theme 1. Parent-related outcome criteria included evidence of improvement in parental knowledge, beliefs and attitudes, emotional well-being (including stress reduction and increased hope), practical skills and an increased sense of empowerment, confidence and advocacy. Child-related outcome criteria included improvement in the child’s quality of life, in the ASD-related deficits (e.g. social and communication) as well as in parent-specified outcomes in the child (e.g. specific behaviours or activities). Evaluation criteria identified in relation to family outcomes included improved quality of life of the family, reduced sense of family isolation, and access to and mobilising support. Under the community component, stakeholders suggested as a criterion that evidence should be sought to determine whether the programme had a positive impact on the community, for instance, through reduced stigma or increased community knowledge about ASD.

Theme 2: Processes and Procedures

A wide range of components emerged under the processes and procedures theme. Criteria under the accessibility component included whether the programme is accessible in terms of the language of delivery, location of the programme, cost (to both parent and programme provider) and literacy requirements of participants. The acceptability component of the programme included criteria relating to cultural considerations, trainers, age range, and materials (e.g. whether the programme is culturally acceptable to the parents/carers attending and whether the trainers of the programme are acceptable to the parents/carers).

Under the psychological processes component, criteria included whether the programme structure was able to provide a psychologically ‘holding’ environment for parents/carers and whether it facilitated parent-to-parent support. An additional criterion examined whether there was evidence that the programme psychologically prepares parents/carers for next steps after the completion of the programme.

The final component under the processes and procedures theme was referral pathways and included two criteria: The first detailed if there was a clear referral protocol into the programme; the second asked whether there was a pathway for parents/carers to receive next-step interventions or support after completion of the programme.

Theme 3: Implementation Landscape

Under this theme, criteria were divided under four components. The first component, sustainability, included four criteria. Criteria included buy-in of the programme provider, local and national government, rated as key to the uptake and sustainability of the programme. Further criteria included evaluation whether appropriate decision-makers and stakeholders are included in the implementation of the programme, whether there was evidence that the programme could be sustainable in the intended context, and whether there was evidence of the necessary funding.

The second component, integration and coordination included four criteria. The first explored whether there was integration and coordination of the programme between relevant systems and departments. The second criterion examined whether there was agreement about who would provide oversight and coordination of the programme. The third criterion ascertained if referrers, professionals and parents were aware of the programme.

The third component scalability contained two criteria. These explore whether training for the PET programme can be scaled-up (e.g. whether there is a train-the-trainer system), and whether there is there evidence that the programme can be replicated across different sites (e.g. whether the PET programme is manualised).

The final component, monitoring and evaluation explored whether there is a system to monitor and evaluate the programme over time, in order to ensure ongoing adherence to the outcomes, processes and procedures and other aspects of the PET programmes, in order to raise the need for programme adaptation as and when needed. The proposed ASD PET Evaluation Framework Checklist is included below in Fig. 2.

After a final review of the draft evaluation framework, the multi-stakeholder group were asked for overarching comments about the consensus evaluation framework. These included firstly, that the framework emphasised that programme evaluation is about more than just efficacy (often used as the ‘gold standard’ in evidence-based medicine), and highlighted the importance of process and implementation as well; secondly, that the framework should be broad enough to be useful to researchers, policy makers, purchasers and providers; thirdly, that the framework could be useful at different stages of programme evaluation research e.g. as part of feasibility assessments, or for ongoing monitoring and evaluation. The panel, however, also raised the need to clarify or operationalise some of the identified variables (e.g. the role of trainers, stigma) and suggested that a shortened framework may be easier for implementation in real life.

Discussion

Given the range of ASD-specific PET programmes available, and the multiple and varied contexts in which they are provided (Dawson-Squibb et al. 2019b), there is a clear need to establish a standardised set of criteria by which to evaluate such programmes. In the absence of a standardised tool, this study sought to develop a pragmatic framework to evaluate ASD-specific PET programmes. Informed by implementation science research theory, a multi-stakeholder participatory approach was used to generate the evaluation framework tool.

One of the key findings of the multi-stakeholder evaluation framework was the importance of considering not only the primary outcome (e.g. parent-focused outcomes), but also to consider outcome in a broader context such as impact on child, family, siblings and the community. The focus of much of the outcomes-based research in ASD PET has been related to parents, for example, decreasing stress, or increasing knowledge (Tonge et al. 2006; Farmer and Reupert 2013). The individual approach is broadly in line with evidence-based practice which emphasises “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient” (Sackett et al. 1996). The framework generated here highlights the importance that PET might have a broader reach than just the individual parents/carers with possible positive outcomes that could be detected also in the individuals with ASD, their siblings and in the community.

The second key finding was the importance of considering processes and procedures in evaluating a PET programme. That is, even if outcomes are evaluated positively, there are cardinal process factors that need to be evaluated. For instance, acceptability and accessibility of a programme are potential barriers to the uptake of a programme. Implementation research literature considers identifying problems that hinder access to interventions and delivery of services as one of its core functions (Proctor et al. 2011; Dunn et al. 2012). The criteria listed under accessibility, for example, including language of delivery, literacy requirements, location and cost to both participant and provider are all potential barriers that may prevent parents/carers from attending despite the potential effectiveness of the programme. Cultural awareness and acceptability of a programme, particularly when developed in a different setting, is another important area to consider for implementation research (Cabassa and Baumann 2013). For example, individuals are less likely to access treatments they consider unacceptable regardless of their effectiveness and are more likely to access treatments viewed by them as acceptable (Eckert and Hintze 2000; Borrego and Pemberton 2007). As pointed out by Cabassa and Baumann (2013), the use of cultural adaptation models in implementation research can make evidence-based programmes more responsive to the needs and preferences of diverse populations. The significance of ongoing monitoring of the acceptability and accessibility of programmes as they are developed and adapted, has also been highlighted in the literature (Proctor et al. 2011).

The third key theme of the Evaluation Framework generated here, referred to as the ‘Implementation Landscape’, indicated further areas critical to the broader expansion of PET programmes. Regardless of the positive evidence for good outcomes and processes and procedures, this final group of criteria of the evaluation framework will determine if it has the potential for scale-up and sustainability, in particular. The key areas listed, including sustainability, integration & coordination, scalability, and monitoring & evaluation of the programme are all considered critical areas of focus for implementation research (Damschroder et al. 2009; Milat et al. 2012). The necessity for programmes to consider scalability at early stages of implementation along with related factors of replicability and sustainability has been described in the literature and this Evaluation Framework highlights their importance. The explicit consideration of funding, both at provider (e.g. for trainers, proprietary costs) and parent/carer level are related to the above. In the scoping review of EarlyBird and EarlyBird Plus described in the introduction the authors noted that implementation factors were often not considered in studies (Dawson-Squibb et al. 2019a).

The implementation science research emphasis on systems and the importance of stakeholder inclusion is further highlighted in the framework (Bammer 2005). The identification of leaders and coordinators of a programme, whether there has been buy-in by providers, local and national government, whether funding decision makers have been included in the process, and whether a PET programme has been or can be integrated into existing systems and services, all highlight the complexity of scaling-up and scaling-out a PET programme. While the goal of many PET programmes may be to scale-up and remain sustainable (as is the case with many health promotion interventions), such a goal requires consideration of these components from the early stages of development and implementation of PET (Milat et al. 2012; Aarons et al. 2017).

Given increasing use of technology for ASD, it may become feasible to provide PET through online or smartphone technologies. The focus of this study was specifically aimed at in-person training and the framework checklist was therefore generated from such data. We anticipate that many elements of the framework checklist may also be applicable to remotely-delivered PET, but it may be helpful to revisit the framework checklist with such modalities of programme delivery in mind.

As predicted, multi-stakeholders identified not only outcome-based or ‘efficacy-based’ criteria, but spontaneously also included a range of implementation-related criteria. The ‘community-generated’ evaluation framework therefore maps well onto existing implementation models that emphasise the importance of assessing and exploring different levels and stages of implementation (Fixsen et al. 2005; Chambers 2014). The work by Metz et al. (2013) outlined the need for a conceptual framework that guides a successful implementation process. This evaluation framework could be used in such an approach. The three levels of the framework align with the Metz et al. (2013) ‘stage matched activities’ that guide the implementation process (e.g. a focus on some of the Implementation Landscape components may be most relevant in the exploration stages). The framework could also assist implementation researchers in the selection of data to be collected and focused on, noting the concept of ‘continuous improvement and data feedback loops’ (Metz et al. 2013). The Evaluation Framework is distinct, however, from these broader implementation science research models in its specificity to ASD PET and in the pragmatic evaluation framework checklist generated for this specific purpose.

Interventions are often referred to as ‘evidence-based’ when one or two very specific pre-specified outcomes have been positively improved by that intervention (Titler 2008). Efficacy is therefore typically the primary goal of intervention research. Whilst necessary, it may not be sufficient for successful implementation in real-world settings. The evaluation framework generated here, emphasises the importance of additional factors that require consideration. It therefore underlines the fact that programme evaluation for all ASD PET requires balanced consideration of all three key themes in order to determine the most suitable programme for a particular setting at a particular time. As shown in Fig. 1 we used the image of three cogs to underline the fact that all three themes are required, and that problems in any of these may be associated with difficulties in real-life settings. The framework was developed out of a real-world problem in a low-resource environment. To our knowledge, this is the first multi-stakeholder generated evaluation framework developed for PET programmes in ASD. We hope that it may be useful not only in low-resource environments but in any setting where clinical, policy or purchasing decisions may need to be made about ASD PET programmes. In specific settings the Evaluation Framework may also be used in conjunction with Theory of Change which is used frequently in the development and evaluation of complex interventions (de Silva et al. 2014). Such an approach might assist users of the framework with directly mapping essential components of an intervention and understanding their relatedness. Theory of Change maps describe how and why a programme works and could use data from the Evaluation Framework to ensure a comprehensive and detailed conceptualisation (Breuer et al. 2016). We anticipate that it might also have potential use for other ASD-specific interventions, for instance parent-mediated interventions, and potentially for other neurodevelopmental intervention programmes.

Limitations

We acknowledge some limitations in our study. This include the fact that we did not incorporate individuals with ASD as stakeholders. However, given that the focus of our work was on parent/carer education training, we felt that inclusion of parents/carers was key. We also acknowledge that the evaluation framework did not generate a specific scoring system. Whilst we initially set out to develop a scoring system, the feedback from the multi-stakeholder participants suggested that the different needs, contexts and purposes of programmes may be better suited to a non-quantified evaluation framework ‘checklist’. That is, instead of proposing that a PET programme should score above a certain cut-off, evaluators can use the criteria as outlined to determine to what extent a potential programme may meet their specific needs. We also acknowledge that the framework was generated by stakeholders in a specific setting and that stakeholders around the globe may identify different or additional themes, components and criteria of relevance. With this potential limitation in mind, we included stakeholders who had experience of working or doing research in a range of countries, which we hoped would increase the likely universal application of the evaluation framework. A final limitation is that there is currently no accompanying document to assist users of the framework. Such a document will need to be developed and may incorporate guidance and strategies used by other models such as PASSING (Programme Analysis of Service System’s Implementation of Normalisation Goals) that use external evaluation teams (Wolfensberg and Thomas 2007).

Conclusion

ASD-specific PET programmes are considered an important component of support for parents/carers directly after diagnosis. There is large variability in the PET programmes available and the contexts in which they are delivered. In the absence of any existing framework, we used an implementation science, participatory research strategy to generate an ASD evaluation framework to guide selection of programmes best suited to specific needs and environments. The framework emphasises the importance of both implementation and process factors in addition to outcomes when evaluating PET programmes for use.