Introduction

The past 20 years have seen an explosion of interest in Social and Emotional Learning (SEL; Greenberg et al. 2017). Research increasingly suggests that Social and Emotional Learning contributes significantly to important life outcomes such as better mental health, school success, college entry and completion, and later earnings (Caemmerer and Keith 2015; Greenberg et al. 2017). Research also shows that Social and Emotional Learning can be taught and nurtured in schools (Jones et al. 2017). Moreover, in addition to promoting positive outcomes, the enhancement of social-emotional competencies also acts as a buffer to the effects of exposure to risk factors (Domitrovich et al. 2017), such as substance abuse (Sandler et al. 2014), aggression, bullying, and disruptive behavior (Arsenio et al. 2009; Cook et al. 2010). Furthermore, an economic review of seven Social and Emotional Learning programs has also demonstrated that they yield good cost/benefit ratios (Belfield et al. 2015), contributing to further interest in this area. All these outcomes have led to an exponential increase in the development, implementation, and evaluation of Social and Emotional Learning programs and policies (Greenberg et al. 2017), that is evident in several countries (Coelho et al. 2015; Malti et al. 2011; Sklad, Diekstra, deRitter, Ben and Gravesteijn 2012; Wolpert et al. 2015).

However, despite the aforementioned impressive growth in the field, there are several issues that need to be addressed regarding the effectiveness of Social and Emotional Learning. Although there are several meta-analyses supporting the general effectiveness of school Social and Emotional Learning programs (Durlak et al. 2011; Sklad et al. 2012), Jones et al. (2017) concluded there was a lack of studies focusing on differential effectiveness (i.e., what works, for whom it works, and under what conditions). Therefore, in the current study we compared the same Social and Emotional Learning program (Positive Attitude) when applied in two different settings (regular school hours and after-school).

Social and Emotional Learning (SEL) Framework

Social and Emotional Learning is the process through which social-emotional competence develops (Domitrovich et al. 2017). According to Weissberg et al. (2015), through Social and Emotional Learning children and adolescents acquire and effectively apply the knowledge, attitudes, and skills necessary to understand and manage emotions, set and achieve positive goals, feel and show caring and concern for others, establish and maintain positive relationships, and make responsible decisions The Collaborative for Academic, Social, and Emotional Learning (2015) considered that the process of acquiring and developing social-emotional competence can be enhanced by well-designed, evidence-based interventions, i.e., Social and Learning programs.

Greenberg et al. (2017) concluded that Social and Emotional Learning programs lead to measurable and potentially long-lasting improvements in many areas of children’s lives. Also, several meta-analyses conducted in the USA (Durlak et al. 2011; Taylor et al. 2017) and Europe (the Netherlands; Sklad et al. 2012) have concluded that evidence-based SEL programs, when implemented effectively, positively impact mental health, academic achievement, pro-social behaviors and attitudes about self while also contributing to the reduction of antisocial behavior and substance abuse. Furthermore, research has also identified some key elements that increase the probability of Social and Emotional Learning programs achieving positive results. Among these elements are high-quality implementation (Durlak et al. 2011; Taylor et al. 2017), developmental adequacy (Jones et al. 2017) and the use of a set of practices (sequenced, active, focused, and explicit) identified by the acronym SAFE (Durlak et al. 2011).

There is much diversity among Social and Emotional Learning Programs (Coelho et al. 2016). Most Social and Emotional Learning programs focus on universal prevention—that is, preventing behavioral problems by enhancing social and emotional competence (Collaborative for Academic, Social, and Emotional Learning 2015). Evidence-based Social and Emotional Learning programs include a wide range of curricula and instructional strategies designed to promote diverse social and emotional competencies among students (McKown 2017). All this diversity has led to some mixed results concerning the effectiveness of universal Social and Emotional Learning programs, and not all implementation efforts have resulted in the expected positive outcomes (Wigelsworth et al. 2016). In England, a strategy to implement a national curriculum of social and emotional aspects of learning has been considered a failure (Wigelsworth et al. 2013), while in Switzerland, an independent large randomized trial of the implementation of a universal Social and Emotional Learning program (Malti et al. 2011) reported no significant effects on social and emotional competence. There are, therefore, still important issues to be addressed. Durlak et al. (2011) reported that 77% of Social and Emotional Learning program evaluations last <1 year, and Wolpert et al. (2015) concluded that programs implemented on a routine ongoing basis in schools are only rarely considered in the evaluation literature. A recent meta-analysis (Taylor et al. 2017) was unable to draw firm conclusions about which specific features determine how effective a Social and Emotional Learning interventions is, and several authors (Jones et al. 2017; McClelland et al. 2017) have concluded that research in this area should focus on what works for whom (differential effectiveness).

There are, however, issues that have not been properly addressed in the literature when considering differential effectiveness. Some authors (Coelho et al. 2015; Hurd and Deutsch 2017) claim that students’ individual characteristics should be given more attention when designing Social and Emotional Learning programs given that among the few studies that have done so, age and gender have been associated with differences in a program’s effects. Other authors (Jones et al. 2017) warn that few studies have measured classroom-level outcomes or features of the classroom environment. These authors thus recommended caution when interpreting the effects of programs where the data has been analyzed at the individual child level, but randomization occurred at the classroom or school level, which may result in overestimating program effects. Finally, none of the most recent meta-analyses (Durlak et al. 2011; Sklad et al. 2012; Taylor et al. 2017) have addressed the issue of how the setting or schedule for program delivery might impact program efficacy or effectiveness, even though there are studies that indicate Social and Emotional Learning program delivery formats does have some influence upon it (Coelho and Sousa 2017a).

The Influence of Individual and Classroom-Level Characteristics on Social and Emotional Learning programs

Gender differences

There is no clear consensus in the literature regarding gender differential impacts from participating in Social and Emotional Learning programs. While some studies (e.g., Durlak et al. 2011) have found no differential impact of gender from participating in universal Social and Emotional Learning programs, several studies (Coelho et al. 2015; Holsen et al. 2008; VanSchoiack-Edstrom et al. 2002) report differential impacts by gender. VanSchoiack-Edstrom et al. (2002), after the application of the Second Step program to 6th grade students, reported reductions in perceived difficulty of performing social skills, but only among girls. Holsen et al. (2008), using the Norwegian version of the same program, concluded that only 6th grade girls benefitted in social competence from participation in the program. In Portugal, Coelho et al. (2015) reported that girls gained more in terms of self-control and social awareness after participation in an upper middle school Social and Emotional Learning program, even though girls had higher initial levels for those competencies.

Age differences

Few studies analyzed the differential effectiveness of Social and Emotional Learning programs according to age. Sklad et al. (2012) concluded that there were no reasons to believe that programs carried out in middle schools had a different effectiveness than those carried out in elementary school, whereas Coelho et al. (2016) found that 4th graders benefited more than 7th graders, in social and emotional self-concept, from the participation in similar Social and Emotional Learning programs. Regarding social awareness, Coelho and Sousa (2017b) reported that 7th graders had higher initial levels than 8th and 9th grader and that teachers reported that 7th and 8th graders gained more than 9th graders in that key competence from having participated in a Social and Emotional Learning program.

Classroom-level variables

The environments in which students are embedded either facilitate or hinder skill development. Emotionally supportive and well-organized classrooms can improve student-level outcomes (Jones et al. 2017). There is a paradox in the fact that most programs are delivered in the classroom, but few studies have measured classroom-level features or outcomes. As a result, the current understanding of interventions is partially devoid of context (Jones et al. 2017). Holsen et al. (2008) reported classroom-level differences in all measured outcomes and suggested that the program’s success varied between classrooms. Accordingly, there is a need to understand the relevant and influential features of the students’ setting (classes, in this case) to better comprehend the mechanisms through which interventions may affect students’ competencies. One relevant feature to analyze is class size, given that Coelho and Sousa (manuscript submitted for publication) concluded that larger classes were associated with higher levels of self-control and responsible decision making. Thus, as Jones et al. (2017) call for, monitoring features of the classroom environment may help us better understand changes in students’ competencies.

Program setting

Some Social and Emotional Learning school programs are more effective than others. This has led several authors (Durlak et al. 2010; Wigelsworth et al. 2016) to call for studies that determine which program characteristics make them more or less effective. The Collaborative for Academic, Social and Emotional Learning has defended that, in order to be most effective, programs should be implemented during the normal school schedule (Greenberg et al. 2003), although several Social and Emotional Learning researchers argue that after-school programs are also an important venue for helping students develop and apply new skills and talents (Hurd and Deutsch 2017), given that their goals are often aligned with those of Social and Emotional Learning.

However, after-school programs, defined as adult-structured programs for students that are offered during the school year between the hours of 3 and 6 p.m. (Hurd and Deutsch 2017), face a number of barriers in promoting Social and Emotional Learning. The main issue is that participation is not mandatory and some of the Social and Emotional Learning after-school programs reported inconsistent attendance by participants (Hurd and Deutsch 2017). While sporadic attendance may dampen a program’s effects, Hurd and Deutsch (2017) claim that attendance alone is not enough to promote Social and Emotional Learning outcomes and that program quality and student participation are crucial.

Positive Attitude Upper Middle School Social and Emotional Learning Program

In Portugal, schools are organized into school groupings, which are administrated by a School Board, comprising a middle school, several elementary schools and kindergartens. Lower middle school (5th–6th grade) and upper middle school (7th–9th grade) share the same school building and have similar class sizes and teacher distributions (one teacher per subject). However, in upper middle school there are more taught subjects (e.g., Portuguese, History, English) than in lower middle school (12 vs. 9), and there are differences in teacher training (upper middle school teacher training is more subject focused, whereas lower middle school teacher training is more pedagogy focused).

Positive Attitude is a project developed in the municipality of Torres Vedras, a district of Lisbon, Portugal, which has been implemented since 2004 (Coelho and Figueira 2011). It includes three developmentally appropriate universal Social and Emotional Learning programs: 4th grade, lower middle school (5th and 6th grade) and upper middle school (7th to 9th grade). The programs were designed to improve children’s social and emotional competencies by helping them develop the five key competencies proposed by the Collaborative for Academic, Social, and Emotional Learning (2015): self-awareness, self-control, social awareness, relationship skills, and responsible decision making. All three programs are based on the Collaborative for Academic, Social and Emotional Learning´s theoretical framework, and employ the set of practices (sequenced, active, focused and explicit) recommended by Durlak et al. (2011). They are classroom-based (including all students in each class), delivered weekly by an educational psychologist (in the presence of the class director), and were (up until the beginning of the current study) integrated into the school curriculum as part of a school subject named Civic Education. The programs’ contents and activities were developed during the first 2 years of implementation and are described in the respective training manual (Coelho and Figueira 2011).

The Positive Attitude upper middle school Social and Emotional Learning program was delivered in a curriculum format, comprising 13 one-hour weekly sessions, according to each classes profile, by one of four educational psychologists (with at least 3 years of experience in the Project), in the presence of the class director, following a program manual, which contains detailed plans for each session. The first two sessions were dedicated to an initial assessment and ice-breaking activities, which allowed for an introduction of the psychologist, the program, and the students. The next five sessions are a unit more focused on self-awareness and self-management. In the five subsequent sessions, one unit (out of three possible) is delivered based on each class’s initial assessment results and a meeting with the Class Director. The three possible themes (five sessions each) were: self-esteem enhancement, social awareness, and relationship skills. The last session (13th) was dedicated to program evaluation. In total, 23 sessions were created (and described in the manual), of which 13 were implemented in each class, with each module focused more explicitly on one or two social and emotional competencies, although all of them were developed within that unit (as recommended by the Collaborative for Academic, Social and Emotional Learning 2015). None of the schools were committed to other Social and Emotional Learning programs. The current study describes results from the 9th to 11th years of implementing this program.

Current Study

Understanding for whom, and under what conditions, interventions work best can guide research, practice, and policy (McClelland et al. 2017). The evidence from gold-standard studies—in which one group is randomly assigned to receive an intervention while another is not—is ambiguous. Specifically, although there are extensive studies analyzing the effectiveness of school Social and Emotional Learning programs (Durlak et al. 2011; Sklad et al. 2012; Taylor et al. 2017) and after-school Social and Emotional Learning programs (Durlak et al. 2010), there is a lack of studies comparing results from the same Social and Emotional Learning programs when applied in both settings.

In the current study, changes introduced into Portuguese educational legislation raised the question of which program format was the most adequate for maintaining the effectiveness and sustainability of an established ongoing program, Positive Attitude. Given that it is classes, and not students (whose participation was mandatory if their class was assigned to receive the program), who are assigned to receive the intervention, the design for the current study removes any potential bias introduced by self-selection into an after-school program (that is, students who sign up for and attend after-school programs could differ in important ways from those who do not).

Therefore, this study analyses the differential effectiveness of a Social and Emotional Learning program applied to upper middle school Portuguese students, specifically comparing the impact of two program delivery formats: curriculum and after-school. Based on previous results (Coelho et al. 2015), we hypothesized, that after accounting for age, gender, condition and class size, students who participated in either version of the Positive Attitude program (intervention groups) will show improvements in their social and emotional competencies when compared with students in the control groups (Hypothesis one), and that girls will display added gains in self-control and social awareness (Hypothesis two). We also posed two research questions, after accounting for age, gender, class size and condition, are there differences in the impact of the program due to program setting? (Research Question one). Also, given that previous results (Coelho and Sousa, manuscript submitted for publication) indicated that class size influenced initial social and emotional competencies level, we posed another research question, after accounting for age, gender, class size, and condition; Does class size influence program results? (Research Question two)

Methods

Research Design

This study had a quasi-experimental design as sampling was not totally random: the school grouping boards did not accept randomly assigning classes to the intervention or control conditions. Therefore, school groupings nominated the intervention classes in which the program would be implemented and conceded to assign matching classes with similar characteristics (achievement levels, class size, and socioeconomic status). The intervention was implemented in two settings: curriculum (during school hours) and after-school (from 3 to 6 p.m.). Control groups were established in both settings with no Social and Emotional Learning content included in their classes. All groups were natural groups (i.e., full classes). It was not possible to include teacher reports due to a change of questionnaires made between Year 1 and 2 of the intervention.

Participants

The sample was a convenience sample, comprising 837 upper middle school students (Mage = 12.70; SD = 0.98), from 41 classes in five public school groupings (in the district of Lisbon). Participant characteristics are shown in Table 1. Intervention and control groups did not differ in terms of age (F(2, 834) = 2.09, p = 0.124), socioeconomic status (ranging from 36.8 to 43.8% of students receiving free or reduced lunches), gender, ethnicity, or school location (as seen in Table 1). All intervention group students were participating in this program for the first time.

Table 1 Student characteristics across groups and waves of implementation
Full size table

Attrition was homogenous across the 3 years of program implementation. Attrition from pre- to post-test was low, with only 10 students (three from the regular school group, five from the after-school group, and two from the control group) dropping out of the program because they changed or dropped out of school. We opted to remove these participants because they only participated in one (out of three) assessments; therefore, the final sample was reduced to 827. However, as displayed in Table 2, attrition from post-test to follow-up was higher (n = 79), mainly because of students being retained in the same grade (n = 62), but also because of students changing schools (n = 17). The attrition rate was homogeneous for the three conditions, χ2 (3) = 0.297, p = 0.861.

Table 2 Social and emotional competencies—descriptive statistics for control and intervention groups, pre, post-test, and follow-up
Full size table

Instruments

Social and emotional competencies

The Social and Emotional Competences Evaluation Questionnaire (QACSE; Coelho et al. 2015) was used. This self-report instrument for adolescents (11 to 16 years) consists of 39 items presented as statements to be rated on a four-point scale (A–never; B–sometimes; C–frequently, and D-always). The Questionnaire assesses six dimensions, four of which were used in the current study.

Self-control

This subscale assesses the ability to monitor and manage one’s own emotions and behaviors, and is composed by seven items (e.g., “When I want to talk, I wait for my turn”). The subscale’s internal consistency is adequate, with Cronbach’s α = 0.73,

Social awareness

This subscale evaluates the ability of understanding other people, empathy, compassion and norms, and it is also composed by seven items (e.g., “I am concerned when someone has problems”). This subscale has good internal consistency (α = 0.87).

Relationship skills

This subscale assesses the capacity of initiating and maintaining positive interpersonal relationships, and the level of communication skills. It is composed by seven items (e.g., “When someone is arguing I am chosen as referee or judge”) and has an adequate internal consistency (α = 0.71).

Responsible decision making

This subscale measures the level of reflexive consideration when facing different choices, where the student has to take into account his and others’ wellbeing. It is composed by four items (e.g., “I ponder several alternatives before making a decision”). This subscale has good internal consistency (α = 0.87).

Self-esteem

Assessed through the Global Self-Esteem scale of the Self-Description Questionnaire II (SDQ II, Marsh et al. 1983; Portuguese version; Fontaine 1991). This scale evaluates global self-worth, and is composed of ten items (e.g., “In general I have a lot to be proud of”), five of which are presented as negative statements and rated on a five-point scale (1–false, 2–mostly false, 3–nor true or false, 4–mostly true, and 5–true). The scale’s internal consistency is adequate, with Cronbach’s α = 0.88 (0.82 for the Portuguese adaptation).

Procedure

The educational psychologists who implemented the program were present in back-to-school meetings (mandatory for parents), to present and explain the program, as well as to answer questions. All school directors agreed to the implementation of the programs and schools used passive informed parental consent, because the program could be considered part of the school offering. The study followed the Portuguese Association of Psychologists (OPP) ethical standards and was approved by the Psychology for Positive Development Research Center.

Self-reports were completed at baseline, post-test and follow-up (7 months after the conclusion of the intervention), whereas demographic data was recorded only at baseline. In the intervention groups, questionnaires were administered in the first and last sessions of the program. Control groups were assessed in the same periods. If a student was not present during evaluation the questionnaires were administered in another class within two weeks (n = 27). In each format, four educational psychologists implemented both the curriculum and after school versions of the program.

Quality of implementation was monitored in weekly team meetings of all the program’s psychologists, during which each psychologist submitted class reports. A progress report was sent to the municipality every month. Overall, the psychologists reported a 96% implementation rate for the regular school setting and 91% for the after-school setting. However, for the after-school setting format program, in six of the 18 classes (33.3%) where the program was delivered, an extra session was needed to deal with all the material in the lesson plans for every session, whereas the same was needed in three of the 13 classes where the program was delivered during regular school hours. An extensive report, including an analysis of the results, was also sent yearly to the municipality.

Data Analysis

T-tests were used to compare initial levels of competence according to age, gender, and school location. A one-way ANOVA was conducted to analyze if there were differences in the initial levels of competence between control and intervention groups. Little’s MCAR test was used to analyze the patterns of missing data, and the value indicated that the missing values were MCAR, χ2(2) = 1.78; p > 0.05. Multiple imputation was thus used to deal with missing values. The imputed data set, composed of five imputations, was created and pooled results were used.

Multilevel linear modeling (MLM) with a repeated measures design was used (SPSS, mixed models) to evaluate the program’s differential effectiveness. There are several advantages to the MLM approach to repeated measures when compared to other analyses, the main one being that it controls for non-independence among the repeated observations for each individual (Heck et al. 2013). In this repeated measures study design, individual scores are nested within individuals, and nested data is more likely to correlate within the group (class). Originally, a three-level model was run to account for the three measurements nested within 837 students, and that these students were nested in 49 classes. However, for relationship skills and responsible decision making, the Intraclass Correlation (ICC; reported in Table 5) show that there was no need to include a 3rd level (class) in the models, since that there is not sufficient variance explained at that level ( < 0.05), following the suggestion by Heck at al. (2013).

No time-varying covariates were included in the model. After several analyses, the best fit was achieved with a linear measure of time (thereby creating model 1) and auto-regressive as the covariance structure for level 1. The intercept was used as a random effect in the models, and for models 2 and 3 individual (Level 2) and class (Level 3) predictors were added. Scaled identity was the covariance structure chosen for levels 2 and 3, given that it guaranteed the best model fit. The next model (model 4) included an interaction between time and condition (dummy-coded as a level 3 variable). In model 4a, condition was removed and replaced by group (modeled by adding dummy-coded level 3 variables). Two final models (models 5a and 5b) were estimated by including cross-level interactions between time (Level 1), gender (Level 2) and condition (Level 3) for model 5a, and cross-level interactions between time (Level 1), class size, and condition (Level 3).

IBM SPSS Statistics for Windows, Version 20 (IBM Corp, Armonk, NY) was used. Heck et al. (2013) note that when using the SPSS mixed model, the reference group for a variable entered as a factor is the last category.

Results

Preliminary Analysis

Descriptive data for the control and intervention groups are displayed in Table 2. There were significant differences between control and intervention groups for social awareness, F(2, 824) = 6.66, p = 0.001, with control groups presenting higher initial levels than after-school intervention groups. No significant differences were found in self-esteem, F(2, 824) = 1.63, p = 0.197; self-control, F(2, 824) = 2.90, p = 0.056; relationship skills, F(2, 824) = 1.30, p = 0.274; or responsible decision making, F(2, 824) = 1.54, p = 0.215. Girls presented higher initial levels of self-control, t(825) = −3.22, p = 0.001, social awareness, t(825) = −8.73, p < 0.001, and responsible decision making, t(825) = −2.80, p = 0.005, than boys. There were no significant differences observed in self-esteem, t(825) = −0.78, p = 0.439 and relationship skills, t(825) = 0.63, p = 0.530.

Effectiveness of the Program on Social and Emotional Competencies

To test the first hypothesis, the effects of condition and the interaction between condition and time were tested in a 3-level model. Significant interaction effects between condition and time identified for self-esteem (β = −0.97, SE = 0.18; t = −5.44, p < 0.001), self-control (β = −0.61, SE = 0.11; t = −5.55, p < 0.001), social awareness (β = −0.90, SE = 0.12; t = −7.26, p < 0.001), and relationship skills (β = −0.24, SE = 0.10; t = 2.36, p = 0.018), but not for responsible decision making (β = −0.18, SE = 0.10; t = 1.76, p > 0.05). Therefore, when both programs settings were aggregated, students displayed gains from participation in four (out of five) outcomes.

Program effects by gender

Further analyses were conducted to understand if the program had similar impacts for both genders, or if girls benefitted more in self-control and social awareness (as proposed in hypothesis two). Cross-level interactions between time, condition and gender showed that only one competence yielded a different impact by gender: social awareness (β = −0.31, SE = 0.14; t = −2.19, p = 0.029), with girls benefitting more from the participation in the program during the total time interval analyzed than boys. There were no significant differences in slopes between genders from program participation in self-esteem, self-control, relationship skills, and responsible decision making.

Program effects, by program setting

To answer the first research question posed, we tested if there were differential program effects according to program setting. The analysis by setting revealed differences in program impact across time between program settings in self-esteem, self-control, and social awareness, with the regular school setting having a more positive impact (as displayed, respectively in Tables 3, 4, and 5). Additionally, substituting condition (control vs. intervention) for group (control, after-school and regular school hours) further reduced Level 3 variances, for self-esteem (−19.7%), self-control (−14.6%) and social awareness (−12.4%). The different program settings did not show significant differences for relationship skills (β = −0.11, SE = 0.11; t = −0.99, p = 0.352) and responsible decision making (β = −0.02, SE = 0.08; t = −0.21, p = 0.834), as seen in Table 6. Although there was no variance to justify adding a third level into the multilevel analysis of relationship skills and responsible decision making, there were significant differences between regular school hours group and control groups in relationship skills (β = −0.30, SE = 0.12; t = −2.55, p = 0.011); and responsible decision making (β = −0.19, SE = 0.08; t = −2.26, p = 0.024).

Table 3 Multilevel model analysis models for self-esteem
Full size table
Table 4 Multilevel model analysis models for self-control
Full size table
Table 5 Multilevel model analysis models for social awareness
Full size table
Table 6 Multilevel model analysis models for relationship skills and responsible decision making
Full size table

Program effects by class size

Finally, to answer research question two we conducted cross-level interactions between time, condition and class size to test if class size had any kind of impact upon each of the condition. For self-control, we found significant interactions for all conditions: control (β = 0.09, SE = 0.03; t = 2.44, p = 0.015), after-school (β = −0.04, SE = 0.02; t = −2.30, p = 0.021), and regular school (β = −0.07, SE = 0.04; t = −1.98, p = 0.048). For social awareness we found a significant interaction for the after-school setting (β = 0.06, SE = 0.02; t = 2.67, p = 0.008). For relationship skills there was a significant interaction in the control groups (β = 0.07, SE = 0.03; t = 2.18, p = 0.029). Therefore, for self-control, students in smaller classes, regardless of the setting, benefitted more from the program while, for the control groups, students in larger classes had a more favorable development of self-control and relationship skills during the analyzed time. Additionally, in the after-school setting students benefitted more in social awareness when they participated in larger classes.

Discussion

Several meta-analyses (Durlak et al. 2011; Sklad et al. 2012) have established that universal school-based Social and Emotional Learning programs positively impact a range of behavioral, social and emotional outcomes. However, there is a lack of studies that analyze differential effectiveness, i.e., the effectiveness of the same Social and Emotional and Learning program when applied in different settings or identifying “what works for whom”. The current study analyzed the differential effectiveness of a universal middle school Social and Emotional Learning program when implemented in two different settings, as well as the relevance of including classroom-level predictors.

In the current study, we first analyzed program results independently of setting. In order to do so, we followed Hurd and Deutsch (2017) recommendation that program evaluation should focus more on seeking ways to make them work better, instead of embarking on a propose large scale evaluation that would encompass more years of program implementation. Even so it encompassed three years of program delivery, carried out under regular daily conditions. The results show that students participating in the Positive Attitude upper middle school Social and Emotional Learning program displayed gains for several outcomes—namely increased self-esteem, self-control, social awareness, and relationship skills—compared to control groups. No significant differences were found for responsible decision making. These results thus generally support our first hypothesis, and are in line with Coelho et al. (2015) who had previously reported that students reported gains in self-esteem, self-control, and social awareness from participating in this program.

We also analyzed how individual predictors (such as gender and age) influenced social and emotional competencies. Regarding gender, results showed that girls only displayed added gains in social awareness, not in self-control, partially supporting the second hypothesis. These results are in line with several authors (Coelho et al. 2015; Coelho and Sousa 2017b; Holsen et al. 2008) who had concluded girls gained more in social competence from a Social and Emotional Learning program than boys. Furthermore, they add to the number of studies that found girls benefited more from these programs in variables where they were displayed higher initial levels (Coelho et al. 2015; Coelho and Sousa 2017b; Holsen et al. 2008). However, for self-control the results contradict previous results using the Positive Attitude’s upper middle school data (Coelho et al. 2015), where girls also benefitted more in self-control. This is probably due to the changes made to the program, replacing some activities with other that were more active to better promote social awareness and self-control in boys. Additionally, no program setting format was especially effective for either boys or girls, adding to the conclusion that each of the Positive Attitude programs (Coelho et al. 2015; Coelho and Sousa 2017a), adjusted to developmental profiles, tend to display distinct patterns of results in terms of gender differences.

There were also developmental differences in the current study. Younger students reported higher initial levels of social awareness, a result that is in line with Coelho and Sousa (2017b). However, in the current study younger students also reported higher levels of self-esteem and self-control. Taken together these results expand the conclusions of Marsh and Ayotte (2003) regarding self-concept. These authors proposed that, as children get older they become more efficient in their self-evaluations resulting in self-concept drops. In the current study, the same mechanism seems to apply to self-control and social awareness. These results highlight the importance of accounting for age when analyzing the Social and Emotional Learning program’ results.

However, more central to our research, there were differences in the programs’ effectiveness according to implementation setting. Implementing the program within the school schedule led to larger gains in self-esteem, self-control, and social awareness compared to the after-school schedule setting (supporting hypothesis two). These results support Greenberg et al. (2003), who suggested school-based Social and Emotional Learning programs yield the most successful outcomes when they are embedded into the day-to-day curriculum and connected with other school activities. This is probably more important in middle school given that students make multiple transitions between classrooms each day (Collaborative for Academic, Social, and Emotional Learning 2015). The current study results, however, are in contradiction with Sklad et al. (2012) who reported that the positive immediate effects of a program on social and emotional competencies decreased substantially at follow-up.

Implementing the program within the school schedule seems to optimize its effectiveness in comparison with implementing it in an after-hour school setting. There are two likely explanations for this result. The first is the lower levels of attendance, which have been pointed out as a disadvantage in after-school programs (Hurd and Deutsch 2017), or in the case of the current study, less consequences associated with not attending, given that in the current study the program´s effectiveness tended to decrease in an after-school setting even though participants attended regularly. The second likely explanation is that after-school activities tend to be viewed as an accessory by students, teachers and parents because that they have no impact on grades. Students might, therefore, be less engaged in the program in this setting as they may see the activities as more entertaining than pedagogical.

The results also highlighted the importance of accounting for class size in the analysis of the program’s effectiveness. Class size did not influence initial levels of social and emotional competencies, but it did impact the program’ effect on some competencies. In the control groups, larger classes progressed more favorably in both self-control and relationship skills during the analyzed time. For programs applied in the after-school setting, smaller classes benefited more in self-control and larger classes in social awareness. In the regular school setting smaller classes also benefited more in self-control from participation in the program. The results are, therefore, in line with Holsen et al. (2008), even though in the current study they were not as extensive as the ones reported by these authors.

In general, the current study implies that different program settings may lead to different effectiveness results. For the upper middle school program Positive Attitude it was clear that the best results were achieved within the regular school setting. Therefore, the setting in which the programs are implemented probably aggregates a series of factors, beyond the participants attendance as suggested by Kataoka and Vandell (2013), that significantly impact the program effectiveness. These results underline the importance of further analyzing programs’ differential effectiveness in the most suitable settings and indicate that classroom-level variables, along with individual characteristics, must be included in such analysis.

Limitations of the Study

Although the current study employed a multilevel longitudinal design it still had several limitations that must be acknowledged. A limitation of the study arose from it being rooted in practice, as school directors did not consent to randomly assigning classes into control and intervention conditions, as they felt that doing so would not serve their needs. Future studies about the program’s differential effectiveness should employ full randomization to strengthen the validity of the results.

Another limitation arises from the results being based solely on student self-reports, as Wigelsworth et al. (2010) identified two main concerns regarding the use of self-reports for the measurement of social and emotional competencies: (a) children and youths may be more likely to give socially desirable responses; and (b) answers can be biased towards “the here and now” rather than summative judgements covering a period of time. Therefore, it would have been useful to use teacher reports, allowing for the triangulation of the assessments, as suggested by McKown (2017). However, this was not possible because during two of the implementation years a reduced version of the questionnaire was being implemented, compromising the validity of the findings.

Future Directions

After conducting this study there are several future directions that can be pursued to further increase our knowledge in this area. Future studies should include a more specific measure for students’ own perceptions of program quality, given that the benefit that students obtain from participation are not a function of the program alone—but of the fit between the program and students’ characteristics, and thus they may also be an important predictor of outcomes (Kataoka and Vandell 2013).

Also, future studies analyzing the effectiveness of the Positive Attitude program should focus on the analysis the cumulative results of Social and Emotional Learning programs (i.e., elementary school, lower middle school, upper middle school) when they are applied over several grades applied to the same students, in accordance with suggestions by Greenberg et al. (2017) that programs should be pluriannual. Additionally, future studies should try to detail the role of the implementer´s experience, as experience and mastery in program delivery could have a prominent role in the results achieved (Castillo et al. 2013).

Conclusion

Analyzing a Social and Emotional Learning program’s differential effectiveness, i.e., understanding for whom and in what settings an intervention works best, can help guide how we adapt existing interventions or develop new programs that meet the needs of children. Program developers should consider that the setting where a program is implemented may influence its effectiveness. Although research has shown that if after-school programs devote time to social-emotional development, and have a good quality of implementation, they can significantly improve students’ self-perceptions and positive social behaviors (Durlak et al. 2010), implementing a program in this setting may also result in the program being less effective. Furthermore, the results highlight the importance of the including not only individual differences (e.g., gender), but also classroom-level differences, as recommended by several authors (Coelho and Sousa 2017b; Conduct Problems Prevention Research Group 2010; Holsen et al. 2008). In the current study, class size influenced the program´s effectiveness. For self-control students in smaller classes benefitted more (for both settings), while for social awareness students in larger classes gained more (in the after-school setting).