Several evaluators have explained the purposes of project evaluations. Taylor-Powell et al. (1996) stated that an evaluation should create greater understanding of the project being evaluated. Lewis et al. (2003) contended that project evaluations should help those implementing projects to make decisions, based on what is working and what is not. Peterson (2002) clearly stated that project evaluation must move beyond simply measuring program outcomes and provide accountability for those who implement the project. Perhaps most importantly, Scriven (1967) unequivocally declared that while evaluations may have many purposes the primary purpose is to determine “the estimation of merit, worth, value, etc.” (p. 5) of that which is being evaluated. We agree with all these contentions and chose the multiattribute utility evaluation (MAU; Edwards and Newman 1982) approach to evaluate the Parent-Implemented Communication Strategies (PiCS) Project, which was developed through a federal grant funded by the Institute of Education Sciences (IES). The MAU evaluation approach clearly adheres to the purposes of evaluation and to the theoretical foundations of the PiCS project.

This report includes a detailed explanation of the MAU evaluation of the PiCS project. The report is organized to provide an overview of the PiCS project, give detailed information on the steps conducted during the MAU evaluation, provide the results of the evaluation, and offer discussion of the benefits and concerns that we encountered when using the MAU evaluation approach.

1 Overview of the PiCS project

It is important that evaluators understand the underlying foundations of a project to make a judgment about its effectiveness. The PiCS project is based on two theoretical foundations: (a) social-pragmatic communication skills are vital for children with developmental disabilities (DD) and their families to maximize their quality of life and provide appropriate communication interactions that foster the development of meaningful relationships and (b) social-pragmatic development is fostered in natural environments and parents are their children’s language teachers in natural (i.e., home) environments. Specifically, effective social-pragmatic communication skills can be described as communication exchanges in social environments that involve competent use of language or pragmatic language skills (Beukelman and Mirenda 2000). For all children, and especially for young children with DD, it is the social context within the home environment and the relationship with their parents/caregivers that sets the stage for early social-communicative competence (Beck et al. 1994; Bell and Harper 1977; Bruner 1975; Greenspan and Wieder 1998). The PiCS project incorporates these theoretical foundations by coaching parents in their homes on strategies that should enhance meaningful interactions with their young children with DD. In addition, the naturalistic and visual teaching strategies that parents were taught are evidence based and effective in enhancing social-pragmatic communication with children with DD (Arthur-Kelly et al. 2009; Dettmer et al. 2000; Gagnon 2001; Ganz and Flores 2008, 2010; Gray and Garand 1993; Halle 1982; Hancock and Kaiser 2002; Hart 1985; Hart and Risley 1975; Heflin and Simpson 1998; Quill 1995a; Quill 1995b; Wall and Gast 1997).

During the first year of the PiCS project we developed 15 procedural manuals, trained staff, refined procedures, and implemented the PiCS intervention package with five families. We trained and coached parents on four naturalistic teaching strategies (i.e., environmental arrangement, modeling, mand-model, and time delay) and three visual teaching strategies (i.e., visual schedules, visual rule reminders, and visual task analysis). We employed a research design and data analysis procedures derived from single-case or intrasubject research methodology (Kazdin 2010) to test the feasibility and effectiveness of the developing intervention package. We assessed the effectiveness of the parent-implemented naturalistic and visual teaching strategies using a single-subject multiple-probe design across strategies, within each family, and replicated across families. Prior to coaching, parents identified social-pragmatic communication goals appropriate for their children’s home routines (e.g., meal time and free play) in collaboration with the PiCS project researchers. We conducted two or three coaching sessions on each strategy weekly and we collected data on both parent and child behavior. We collected results of parents’ behavior (i.e., use of the teaching strategies with high quality) and children’s communication skills (i.e., Preschool Language Scale 4th Edition [PLS-4], Zimmerman et al. 2002) and analyzed language samples. In addition, we collected social validity data regarding the project goals, procedures, and outcomes through interviews and surveys with our family participants.

2 Purpose of the PiCS evaluation

The PiCS project principal investigators and project coordinator developed a mission statement which reflected the theoretical foundations and evidence-based practices that were the core of the PiCS project. The mission of the PiCS project is to (a) develop a home-based naturalistic and visual strategies intervention program that parents can personalize and implement to improve the social-pragmatic communication skills of their young children with disabilities; (b) evaluate the feasibility, effectiveness, and social validity of the program; and (c) disseminate a multimedia instructional program, including prototypes of all materials and methods that diverse parents can implement in their home settings. The MAU evaluation of the PiCS project was a formative one, measuring the first two goals, and was completed after the second year. The PiCS team had not begun to address the third goal, dissemination of a multimedia instructional program, at the time of this formative evaluation. The PiCS team desired a formative evaluation to determine the extent to which the PiCS project was effective before producing a final multimedia instructional program. To engage in a comprehensive evaluation we decided to include data from the five families who participated in the first year. See Table 1 for a description of the participating families’ demographic information. The PiCS project is ongoing and data continue to be collected on an additional six families.

Table 1 Family participants’ demographic information
Full size table

3 Overview of the MAU evaluation approach

The MAU evaluation approach has been applied to the evaluation of special education services, higher education programs, special programs in schools, and supported employment services (Edwards and Newman 1982; Johnson and Lewis 1994; Lewis et al. 1994; Lewis et al. 2003; Thompson et al. 2000; Thompson and Stoner 2001). MAU evaluations are especially useful for establishing a “clear, defensible, and manageable set of criteria against which to assess a program and obtain quantitative data that reflect the extent to which program goals have been achieved” (Thompson et al. 2000, p. 37). Basic terminology and corresponding definitions used in the MAU evaluation of the PiCS project are listed in Table 2.

Table 2 MAU evaluation terminology and corresponding definitions
Full size table

The MAU evaluation approach is goal based, yields summative and formative information, and was workable with the fixed and limited resources available for the PiCS project. However, most importantly, we selected the MAU evaluation approach since it was participant oriented, it was congruent with the underlying theoretical foundations of the PiCS project, and it allowed for parent representatives to have a voice in the evaluation. The MAU evaluation of the PiCS project is considered an internal, participant-oriented approach designed to provide the team with information for making decisions about the effectiveness of the PiCS intervention.

4 Seven steps of the MAU evaluation of the PiCS project

There are seven steps in a MAU evaluation: (a) identify the purpose of the project, (b) identify relevant stakeholders (these individuals will help make decisions about the goals and attributes and their importance), (c) identify appropriate criteria to measure each goal and attribute, (d) assign importance weights to the goals and attributes, (e) assign utility-weighted values to the measurement scales of each attribute, (f) collect measurable data on each attribute being measured, and (g) perform the technical analysis (Edwards and Newman 1982; Lewis and Johnson 2000). It is important to note that the initial steps of the MAU evaluation, identifying the purpose, relevant stakeholders, goals, attributes, and assigning weights to goals and attributes were completed at the end of the pilot study. However, the current report includes data from an additional three families from the first year and, consequently, the data are from a total of five families and include data that were collected 6 months postintervention. Structuring the MAU evaluation within this time frame gave us the opportunity to involve parent representatives from our pilot study who had experienced the PiCS project, and evaluate the program with data collected from all five families who participated during the first 2 years of the PiCS project.

4.1 Step 1: Identify the purpose of the PiCS project

The principal investigators and the project coordinator developed a mission statement from the conceptual foundations of the PiCS project and the mission statement formed the basis for the MAU evaluation. The purpose of the PiCS MAU evaluation was to determine if and how well the project was meeting the goals set forth in the mission statement.

4.2 Step 2: Identify relevant stakeholders

The MAU evaluation holds, as one of its basic tenets, stakeholder involvement. The PiCS team strove for involvement from representatives of each group of stakeholders. Stakeholders were the principal investigators, project coordinator, speech and language consultant, multimedia consultant, family consultant (a parent of a child with disabilities who works in the role of a family advocate), graduate assistants, and the two mothers and one father involved in the pilot study, representing the family participant perspective. Our parent participants actively participated in many aspects of the PiCS project, such as communicating their vision for their children, setting individual goals for their children prior to each coaching session, and providing feedback on their own performance after each coaching session; it was, therefore, vital that they be represented and contribute to the evaluation process.

4.3 Step 3: Identify appropriate goals and attributes and set the criterion or criteria to measure each attribute

The group of PiCS stakeholders met for approximately 3 h at a large Midwestern university. The parent participants were encouraged to bring their children, since both of the participating mothers did not work outside the home. Toys, snacks, and supervision were provided in the large room where the stakeholders met, and we felt this was important, since it served to show the parents how much we valued them and their children. The PiCS project mission statement was presented to the stakeholders and after an opportunity to discuss and ask questions, all stakeholders agreed that the goals for the MAU evaluation would be taken directly from the mission statement. Each stakeholder was given a goal sheet and the stakeholders were asked to “spend” 100 points across the three goals. The goal sheets were collected but not tallied at that point due to the time allotted with the stakeholders.

We then began the brainstorming process for the stakeholders to establish the attributes that would measure the three goals. We strongly stated that this was not the time to evaluate the attributes and urged each stakeholder to contribute, cautioned against critical comments that could inhibit the brainstorming process, and reiterated the importance of “hearing each voice.” We divided stakeholders into three groups and each group was a combination of various stakeholders. For example, the members of one group were a principal investigator, the program coordinator, two graduate assistants, one parent, and the speech and language consultant. This process ensured that all stakeholders had a voice in establishing the attributes that would measure the goals.

There were three “goal tables” and each had large sheets of paper and markers. The group listed their ideas on how to measure each goal, and then moved to the next “goal table.” The purpose of this procedure was to give each stakeholder the opportunity to voice his or her own perspective and it lasted approximately 90 min. As we moved each group through this process, the stakeholders had the opportunity to read the other groups’ responses and at times they would ask for clarification to avoid duplicating an attribute. Even though this was a formal process, the interaction was casual and, at times, a bit raucous. Stakeholders would sometimes loudly ask, “Okay, what does this mean?” when reading ideas from other groups. Parents and other stakeholders would often relate incidents that had occurred during the training and coaching sessions to illustrate and clarify their ideas, and there was much discussion and laughter. At times, parents or graduate assistants would tend to the children, within a distance that allowed them to continue to participate. As the group moved from one goal table to the next we could hear and see the interactions among the stakeholders increase; all were actively engaged in the process and appeared to be enjoying it as well.

We concluded the group participation at the end of 3 h by listing all the attributes under each goal and asking for further clarification of attributes that were ambiguous. During this process some of the attributes were removed, if duplication occurred, or some were consolidated. We told the group that they would be receiving “attribute sheets” and that they would be asked once again to spend their 100 points across the attributes for each goal. For example, Goal 1 had two attributes and the stakeholders distributed 100 points across these two attributes based on their perspectives on what was most important to them. It is important to note that not all goals had the same number of attributes and this is perfectly acceptable in MAU evaluations since “it is important to identify the most essential attributes within each goal area, not to identify a set number of attributes” (Thompson and Stoner 2001, p. 4).

Approximately 3 weeks later the principal investigators and the project coordinator met to review all stakeholder responses. Attributes were condensed, consolidated, and organized under each goal; Goal 1 had two primary attributes, Goal 2 had five primary attributes, and Goal 3 had three primary attributes. We took great care at this point to reflect all stakeholders’ input, so that the process of condensing and consolidating did not eliminate any of the stakeholders’ viewpoints.

As we organized the attributes under each goal we found that the attributes themselves had several components that were identified by the stakeholders. For example, Attribute 5 under Goal 2 was, “The PiCS intervention program will be socially valid.” Three components of this attribute were identified: (a) socially valid intervention program goals, (b) socially valid intervention program procedures, and (c) socially valid intervention program outcomes. Consequently, we simply labeled the attributes 5a, 5b, and 5c, and we applied this process to all attributes. When we developed the attribute sheets to distribute to the stakeholders we organized them by goals and attributes so that the stakeholders could clearly understand the progression from goals to attributes.

Once the goals and attributes were determined by the stakeholders we developed a value tree. Edwards and Newman (1982) described a value tree as a graphic representation that links measurable attributes to program foundations and goals. The value tree for the MAU evaluation of the PiCS project is shown in Fig. 1.

Fig. 1
figure 1

MAU value tree

Full size image

We then defined a measurable and observable criterion score for each of the attributes; the MAU process refers to this as the “ideal” score. The ideal score was set at either 100% or 80% for Goals 1 and 2 (Goal 3 was not measured in the evaluation). We set the ideal score at 100% when the attribute was vital and foundational to the framework of the PiCS project. For example, the ideal score for Goal 1 Attribute 1c, “The intervention will be based on best practices in early intervention,” was set at 100% since this was a foundational value of the PiCS project. Attributes that had an ideal score of 80% were performance based. For example, Goal 1 Attribute 2 d was “Recognition of parents’ learning preferences and needs” and was measured by the procedural fidelity of incorporating parents’ learning preferences and needs into training and coaching sessions. We felt we could meet and tolerate an 80% ideal score on some attributes but desired a 100% ideal score for those we felt reflected the foundation of the PiCS project. It is important to note that some of the ideal and performance scores are in numbers instead of percentages. For example, Goal 1 Attribute 1a is to provide quality materials and this was assessed by asking parents to rate this on a scale from 1 to 5. We set a score of 4, or an 80% ideal score, for this attribute, and consequently in the performance index the set ideal score of 4 appears rather than 80%. The ideal scores are then compared to the performance scores obtained in Step 6, Collect Performance Data. See Appendix A for the comprehensive goals, attributes, and data sources.

4.4 Step 4: Assign importance weights to the goals and attributes

The MAU evaluation approach blends the importance the stakeholders place on each goal of the program with the performance of the program. For example, with Goal 1 Attribute 2, “The development of the PiCS intervention program will be based on collaboration between parents and coaches,” some stakeholders, such as parents, may place more value and “spend” more of their points than the principal investigators who may place more value on Goal 1 Attribute 1, “The PiCS team will develop an intervention program that will provide home-based training and coaching of naturalistic and visual strategies.” We averaged the points spent from all stakeholders and determined the value weight assigned to each goal. The result was that the average for Goal 1 was 36 and Goals 2 and 3 each had an average of 32, which totals 100 and is representative of the average points spent by all stakeholders. At this point we divided all numbers by 100 to “normalize the numbers to sum to 1.00” (Edwards and Newman 1982, p. 74).

To determine the importance weight of the attributes we followed the same procedure with one slight difference. The attribute sheets listed the goals with the primary attributes and their components. As the stakeholders “spent” their 100 points they were spending them on the primary attributes and could clearly see the components of these attributes. We determined the weight of each attribute component by dividing the number of points spent on the attribute by the number of components. For example, Goal 1 had two attributes and stakeholders gave Attribute 1 an average of 51.65 points and Attribute 2 an average of 48.35 points, which totals the 100 points each stakeholder had to spend. However, each attribute had a different number of components; there were five components for Attribute 1 and six components for Attribute 2. To determine each attribute’s weight we divided the attribute average by the number of components; Attribute 1 had five components, so each one was worth 10.33 (51.65/5) and Attribute 2 had six components worth 8.06 (48.36/6) each. At the end of this process we divided each number by 100 to normalize the numbers to sum to 1.00.

4.5 Step 5: Assign utility-weighted values to the measurement scale of each goal

The next step in the MAU evaluation process is to provide a measure (utility weight) to compare each of the attributes. To determine the utility weight for each attribute we multiplied the goal weights and the attribute weights that were calculated in Step 4. The utility weight takes into account the importance the stakeholders assigned to both the goals and attributes. For example, Goal 1 had a goal weight of .36 and Attribute 1 had an attribute weight of .5165 with each of the five components’ weight at .1033. Goal 1 Attribute 2 had an attribute weight of .4836 with each of the six components’ weight at .806. To calculate the utility weight and provide a measure of comparison for all components we multiplied the goal weight by each attribute weight.

4.6 Step 6: Collect performance data

We identified data sources and determined how and which data would be collected to measure the attributes. The data used for this report were collected as part of the PiCS project procedures and included performance data on the parents’ use of the target teaching strategies, interviews with parents concerning the PiCS project, staff meeting reports, formal and informal language assessments, meetings with the parent consultant, and materials purchased. The performance scores were then compared to the ideal scores for each attribute. Performance data will continue to be collected throughout the project and used in the summative evaluation at the end of the 3-year project. An example of the primary performance data we collected from parents on their use of the teaching strategies is presented in Fig. 1.

Table 3 contains secondary performance data from the formal and informal assessments completed with the children.

Table 3 Descriptive statistics for PLS-4 assessments and language samples analysis
Full size table

4.7 Step 7: Perform technical analysis

Since the multimedia intervention package, represented by Goal 3, has not been completed, we were unable to calculate the complete MAU evaluation performance index. However, the performance data from Goals 1 and 2 were collected and utility points for each attribute were computed by multiplying the ideal score by the performance score. The weighted utility was then computed by multiplying the utility weight for each attribute by the utility points for each attribute. Ideally, the total weighted utility for all attributes under Goal 1 should equal .36, under Goal 2 should equal .32, and under Goal 3 should equal .32. These figures are the average number of points “spent” by the stakeholders for each goal. Since Goal 3 was not included in this formative assessment, the total evaluation weighted utility could not be computed. However, the results for each attribute under Goal 1 and Goal 2 are presented in the next section.

5 Results of the MAU evaluation of the PiCS project

Results of the MAU evaluation are presented in Table 4. Results were positive; 25 of the 28 attributes met the established criteria. To determine how well each goal was met we added the weighted utility calculations and compared them to the points the stakeholders had desired for each goal. For Goal 1, the sum of the weighted utility points for each attribute was .41—far above the .36 stakeholders had desired. For Goal 2 the weighted utility was .28, which is less than the .32 the stakeholders had desired and represents the three attributes that did not meet their established criteria.

Table 4 MAU Evaluation technical analysis of the PiCS project
Full size table

The three attributes that did not meet criteria under Goal 2 were: (a) 2b: “Program activities will facilitate parents’ implementation and maintenance of the strategies with high quality;” (b) 3c: “Children will maintain and generalize learned social-pragmatic communication skills,” which was not met since the expected increase in skills did not continue after parent coaching concluded; and (c) 4a: “There will be improved family quality of life as measured by formal measurements (Family Quality of Life Survey; FQOL; Hoffman et al. 2006),” which was not met by any family.

6 Discussion

Evaluating any program is an arduous task and must be undertaken carefully and systematically. Barnett et al. (1999) postulated that when evaluating early intervention programs the basic questions of the (a) efficacy, (b) acceptability, and (c) practicality of the intervention must be addressed. These authors further contended that in the natural setting answers to these basic evaluation questions are challenging due to the “heterogeneous nature of children’s characteristics and ecologies” (p. 177). In addition, Peterson (2002) challenged evaluators of programs that provide services to families to “conduct a careful analysis of what actually occurred during the intervention process” (p. 84). Since the PiCS project focuses on early intervention with families we heeded the advice of Barnett et al. and chose the MAU process to answer the questions of efficacy, acceptability, and practicality. We also feel that the MAU process is in concert with the advice of Peterson since it allows for a careful and multifaceted analysis of what actually occurred during the intervention. In addition, and perhaps most importantly the MAU approach was in keeping with one of the core values of the project: stakeholder involvement.

Program evaluation standards developed by the Joint Committee on Standards for Educational Evaluation (2010) include the standard of transparency and disclosure (i.e., Standard P3: Transparency and Disclosure). Transparency in evaluation is essential to ensure the integrity of the results (Fitzpatrick et al. 2011). An additional advantage of using the MAU approach for evaluating the PiCS project is that it is transparent; reviewers and other interested individuals can examine the evaluation process and understand how the judgments of the PiCS intervention project were made. All attributes under Goal 1 met the established criteria and only three attributes under Goal 2 did not meet the established criteria. The purpose of an evaluation is to render a judgment and results of this evaluation have deemed the PiCS project well developed and well implemented.

The purpose of this report was to present the MAU evaluation approach as used to evaluate a project that focused on increasing the social-pragmatic communication skills of young children with DD by training and coaching their parents to implement naturalistic and visual teaching strategies in their home environments. Parent and child participant responses to the PiCS intervention package will be provided in detail in other publications. This report focused on the MAU evaluation process and since we have already provided a detailed description and results of the MAU evaluation process we will focus this discussion on the benefits and a concern related to using the MAU evaluation approach to evaluate the PiCS project.

There were four primary benefits of using the MAU evaluation for the PiCS project. Specifically, the MAU evaluation (a) was based on the core values of the PiCS project; (b) engaged all stakeholders, including parents, in developing the evaluation framework; (c) provided a certain degree of objectivity and transparency; and (d) was comprehensive. Since we developed the goals from the mission statement, the evaluation had a strong theoretical base and was clearly aligned with the core values of the PiCS project. Similarly, stakeholders in the PiCS project are of primary importance, and their input and perspectives are highly valued. The MAU evaluation process incorporated parent perspectives into the development of the attributes which measured each goal. Parents were actively engaged during this process and appeared to appreciate the opportunity to have a voice in the evaluation plan and process. The MAU evaluation of the PiCS project is transparent and strives for objectivity. The full evaluation report clearly states the goals and attributes and how they were developed, the set criterion or criteria for each attribute, the performance data that were collected, and discussion of the results of the evaluation of each attribute. We strongly feel that readers of the complete MAU evaluation of the PiCS project could replicate our findings. Since we used evidence-based practices to develop the PiCS project and measured those evidence-based practices through performance of our participants, fidelity to our procedures, and reliability during our coding of participant behaviors, we feel that we have completed an evaluation with a concerted effort to reduce any internal evaluator bias. Finally, we believe the MAU evaluation of the PiCS project was comprehensive since we evaluated parent and child outcomes of the PiCS intervention, the social validity of the PiCS project, and the logistical issues involved with administering such a large and complex project. The MAU evaluation process offered us the opportunity to evaluate all these aspects and consider issues that we might have overlooked had the evaluation framework not given each stakeholder a voice.

The primary concern of using the MAU to evaluate the PiCS project was the length of time and labor required to conduct it. One of the principal investigators took the lead in planning the MAU evaluation, facilitating the stakeholder meetings, distributing the goal and attribute point distribution sheets, organizing the performance data, completing the technical analysis of the MAU, and writing the final evaluation report. The final MAU evaluation report of the PiCS project is comprehensive and includes specific results for each attribute along with an executive summary and recommendations. For the PiCS project the MAU evaluation worked well, primarily because it provided a framework to involve our participants. We contend that the MAU evaluation could be used with any stakeholders. It was not difficult to include our stakeholders in the process and it provided them with a sense of empowerment, perception that their opinions were valued, and belief that they were an integral part of the PiCS project.

The MAU evaluation may not be applicable for evaluating smaller projects since the time and labor required are significant. However, we recommend incorporating stakeholder input in the evaluation of any project. Too many evaluations neglect to obtain input from stakeholders and the MAU evaluation provides an avenue for eliciting and valuing stakeholder opinions. We strongly feel that the MAU approach met our needs for evaluating the PiCS project despite the time and labor required.

7 Recommendations based on the MAU evaluation of the PiCS project

The first recommendation based on the MAU evaluation is that the PiCS project should continue with the majority of the procedures already in place since most of the criteria were met. Additional recommendations were easily developed by examining the three attributes that did not meet the established criteria. Attribute 2b fell just short (93.75%) of the 100% established criterion and upon further analysis of the performance data, we found that while all parents met the performance criteria for implementing the naturalistic teaching strategies and the visual teaching strategies, one family did not use visual teaching strategies during all observation sessions. However, it is important to note that there may not have been an opportunity to use the visual teaching strategies during observation time (e.g., no opportunity to use the visual bedtime routine strategy when coaching occurred during the day). Therefore, no recommendation was made to address this attribute. Attribute 3c was not met since the expected increase in skills did not continue after the parent coaching concluded and this was addressed through the recommendation of offering booster sessions to all parents during the 6-month postintervention probe period. The last attribute that was not met was 4a: “Family quality of life will improve as measured by the Family Quality of Life (FQOL) formal assessment (Hoffman et al. 2006).” The issue with Attribute 4a was that the FQOL scores were high in the preintervention assessment and room for growth was minimal. Furthermore, the pre- and postintervention FQOL surveys were administered only 3 months apart, which may not have been enough time to measure any changes in family quality of life. The recommendation was to rely more heavily on parent interviews to determine whether or not the PiCS project improved family quality of life.

8 Conclusion

In conclusion, the MAU evaluation approach is a labor-intensive and time-consuming process, yet we found it appropriate for evaluating the PiCS project which was designed to train and coach parents of young children with developmental disabilities on the use of naturalistic and visual strategies to increase their children’s social-pragmatic communication skills. The MAU evaluation has yielded an evaluation of the PiCS project that is comprehensive, transparent, and easily replicated. We especially valued the framework of the MAU approach that required input from all stakeholders and provided a method of weighting their input. Several authors (e.g., Edwards and Newman 1982; Scriven 1967; Thompson et al. 2000) have stated that the purpose of an evaluation is to render judgment of the effectiveness of the program. Our experience with this MAU evaluation has given us that judgment. The PiCS Project is valuable (a) to the parent participants who increased their use of naturalistic and visual teaching strategies, (b) to the young children who benefited from parental use of these teaching strategies, and (c) to the field of special education since results have increased our knowledge about the effectiveness of parent-implemented naturalistic and visual teaching strategies. Wagner et al. (2003) contended that the “underlying goal of all parent education programs is universal—-for children to thrive” (p. 185). The MAU evaluation process gave us the confidence to state unequivocally that the PiCS project taught our participants strategies that met that universal goal.