Introduction

Up to 20% of children experience mental health problems any given year, with emotional disorders being the most prevalent cluster [1]. Thus, it becomes essential that efforts are concerted toward increasing access to effective prevention programs. Recently, online therapeutic or serious games were proposed as a promising prevention tool for emotional disorders in youths [2]. Indeed, therapeutic games capitalize on their attractiveness for the youths, that can promote engagement, retention and motivation, and also reward skills practice in simulated environments. Several such initiatives were dedicated over time to the development of the therapeutic games, with research documenting their positive effects on the mental health-related outcomes in youths (i.e., SPARX to approach depression in youths; [3]).

REThink is such a therapeutic game, unique due to its features, among which are its transdiagnostic framework used to approach emotional disorders in youths, and a tested cognitive-behavioral therapy (CBT) based prevention protocol (Rational Emotive Behavior Education—REBE—and Therapy [4]) incorporated in a complete action and adventure video online game [5–9]. The game was developed around the character of RETMAN, which guides the youths in saving the minds of the earth inhabitants from the powers of irrationalizer. Thus, the main focus of the game is, based on REBE, the cognitive change skills that allow youths to recognize irrational beliefs (e.g., I must succeed all the time) that are connected to emotional distress, and replace them with rational beliefs (e.g., I very much want to succeed but I understand that sometimes I can fail). The game has seven levels and each level is focused on a specific component of the cognitive-behavioral transdiagnostic protocol and emotion-regulation skill, namely emotion recognition, cognitive change, relaxation, problem-solving and cultivating positive emotions. While there are a few games at the international level focused on some of these skills (e.g., relaxation, positive emotions), there is none comprehensive enough to integrate them based on a transdiagnostic approach to emotional disorders, based on a tested curricula, in a standalone game-based prevention tool [2].

This study is completing the main trial presenting the post-test results in terms of the efficacy of the game in a sample of 165 healthy children and adolescents aged between 10 and 16 years, compared to a standard face-to-face cognitive-behavioral program and a wait-list [6,7,8]. Results showed that children in the REThink group obtained a medium-size significant reduction in emotional symptoms and additionally in conduct problems and improvements in pro-social behavior at post-test, while the other groups did not register these improvements. Furthermore, results indicated that youths in the REThink condition showed a significant decrease in a depressive mood, fear, attention, inhibitory control and emotion-regulation competencies (emotional self-awareness and emotional control) at post-test, as compared to the wait-list group. We also performed mechanisms of change analyses and we documented that changes in irrational beliefs mediated, the outcome improvements in the REThink intervention, in accordance with the cognitive-behavioral theory on which the game is based.

The present research, therefore, aimed to test if the gains documented after the REThink therapeutic game in a randomized clinical trial are maintained at a 6-month follow-up. Based on previous documented positive results of the REThink game as a standalone prevention program for improving the emotional symptoms and emotion-regulation in children and adolescents, we expected the documented improvements are maintained at a 6-month follow-up assessment. We hypothesized that the REThink therapeutic game will register similar maintenance of gains in these outcomes to the standard face-to-face intervention, relative to post-test assessment and an initial waitlist condition, which after post-test received care as usual (CAU). Moreover, since the role of parenting and childhood trauma have been well documented as a risk factor in child emotional disorders (e.g., psychological control and insecure attachment-related variables; see [10], we predicted that improvements in follow-up the outcomes for both the intervention groups will be lower for those youths presenting with risk factors, such as parental psychological control, insecure parent attachment or childhood trauma history.

Methods

Participants

One hundred and sixty-five children and adolescents aged 10–16 years were recruited voluntarily from one middle school in [Cluj county, Romania], out of which 142 participants completed the initial assessment and were included in the initial study With regard to drop-out, from post-intervention to follow-up, of the total sample of 142 participants at post-intervention (46-CAU, 48-REBE, 48-REThink), 137 participants completed follow-up measures (45-CAU, 45-REBE, 47-REThink). The total attrition rate was 3.5%. Given that an attrition rate of up to 5% is considered to introduce no- to very little bias [11], the 5 participants that were lost to follow-up were excluded from subsequent analyses. The mean age of participants was 12.84 (SD = 1.97), and 58.6% of them were females. Figure 1 presents the flow of participants over the course of the study.

Fig. 1
figure 1

The CONSORT flow diagram of the participants’ progress throughout the phases of the trial

Full size image

Procedure

The study protocol was registered at ClinicalTrials.gov: NCT03308981. Ethical approval for the study was obtained from the IRB at the author’s affiliated institution. Both interventions were based on dedicated protocols (see below) and their implementation was assisted by one experienced psychologist trained in CBT. The contents of each intervention are briefly presented below but can be found in the paper presenting the outcomes at post-test [6].

The interventions

The REThink game

REThink is a therapeutic online game, embedding the CBT/REBE protocol into a complete online mobile game, implemented at the time of this trial as an iOS applicationable to the function of Apple mobile devices (smartphones or tables). It is meant as a standalone prevention cognitive-behavioral program to train emotional competencies in children and adolescents. The user receives the mission of helping earth inhabitants to learn how to neutralize the irrational powers of the negative character, irrationalizer. They do this by learning the skills during the seven levels of the game (see Fig. 2): emotional recognition (Level 1), thinking and the well-being consequences (Level 2), the relationship between beliefs, emotions and behaviors (Level 3), neutralizing irrational thinking with rational alternatives (Level 4), problem-solving skills (Level 5), relaxation skills (Level 6) and happiness skills (Level 7). Each level was played twice by participants over four consecutive weeks, using an Apple iPad Air 2 (for details see [6]).

Fig. 2
figure 2

Illustration of the seven levels of the REThink game

Full size image

The standard face to face prevention program: REBE group intervention

The REBE group received a standard face-to-face preventive intervention based on the protocol of the curricula passport to success [12]. The intervention had a class lessons format, which had the same focus with each of the seven levels of the game and the same distribution of the modules delivered twice over four consecutive weeks. (for details see [6]). Thus, each lesson was focused on a specific emotional skill (e.g., emotion recognition, thinking-feeling connection) based on experiential activities, open discussions and setting actions.

The care as usual condition

The care as usual condition (CAU) was initially assigned to a waiting list, considered as an untreated comparison group during the study and participants taking part only in the three assessment stages during the trial. However, after the post-test assessment, this group received interventions as part of the school program during the following 6 months, until the follow-up assessment, that could consist of counseling, support groups, or prevention programs.

Measures

Participants completed assessment before the interventions, in the middle of the interventions, at the time of termination (with results being presented in [6]) and at 6-month follow-up. The following primary outcomes were considered

Emotional symptoms, measured with the Strengths and Difficulties Questionnaire-child version (SDQ; [13]) which is considered to have adequate psychometric properties [14] and was used to assess emotional symptoms as a primary outcome and, as secondary outcomes, the total level of psychological difficulties, conduct problems, hyperactivity, peer relationship problems, and prosocial behavior. The internal consistencies for the initial study were α = 0.75 for the emotional symptoms subscale, α = 0.80 for the total level of psychological difficulties, α = 0.65 for conduct problems subscale, α = 0.65 for the hyperactivity subscale, α = 0.63 for peer problems subscale, and α = 0.67 for prosocial behavior subscale.

• Emotion regulation key aspects, measured with the Emotion-Regulation Index for Children and Adolesents (ERICA; [15]). More precisely, we assessed emotional control (i.e., socially appropriate emotional expressions and responses) and emotional self-awareness (i.e., emotional recognition and flexibility, upregulation of positive affect and downregulation of negative affect). ERICA has been shown to have good psychometric properties in the previous studies [16], while the internal consistencies for our initial study were α = 0.70 for the emotional control subscale and α = 0.57 for the emotional self-awareness subscale.

• Depressive mood, measured with the corresponding subscale of the Early Adolescent Temperament Questionnaire-Revised (EATQ-R; [17]), and which was also used to measure as secondary outcomes the temperamental focused attention (the capacity to focus attention and shift attention), fear (affect related to anticipation of distress), and inhibitory control (the capacity to plan and to suppress inadequate responses). The EATQ-R demonstrated adequate psychometric properties in the previous studies [18], while the internal consistencies for our initial study were α = 0.48 for the attention subscale, α = 0.56 for the fear subscale, α = 0.52 for the inhibitory control subscale, and α = 0.64 for the depressive mood subscale.

With regard to secondary outcomes, the following were considered:

• Mental health symptoms (measured with SDQ and described above);

• Temperamental emotion-regulation features (fear, focused attention, and inhibitory control; measured with EATQ-R and described above); and.

• Functional and dysfunctional negative emotions, and positive emotions were measured with the corresponding subscales of the Functional and Dysfunctional Child Mood Scales—girls and boy’s versions (FD-CMS; [19]). The scale has 10 items rated on a ten-point likert scale for intensity. The functional negative emotions subscale sums the ratings for sadness, anxiety, and irritation, while the dysfunctional negative emotions for depression, fear, and anger, and the positive emotions subscales sums ratings for happiness, confidence, and calmness. Internal consistencies were computed on the current sample and yielded satisfactory coefficient levels (α = 0.80 for the functional negative emotions subscale, α = 0.65 for the dysfunctional negative emotions subscale, and α = 0.66 for the positive emotions subscale).

With regard to potential moderating variables for the outcomes at follow-up, the following were considered:

• Parental psychological control, measured with the Psychological Control Scale—Youth Self-Report scale (PCS-YSR [20]). The scale includes items capturing parent behaviors such as constraining verbal expressions (e.g., “My parents will avoid looking at me when I have disappointed them”), invalidating feelings, erratic emotional behavior, personal attack, love withdrawal, and guilt induction. Children and adolescents are asked to rate the extent to which items describe their parents using a 5-point likert-type scale from 1 (never) to 5 (always). The scale has adequate psychometric properties (α = 0.89; [21]).

• Parental attachment in youths, measured with the Inventory of Parent and Peer Attachment -Revised (IPPA-R; [22]). The IPPA is a 25 items scale that measures a youth’s relationships with parents (in this study mother, father; or guardians acting as main attachment figures), such as trust, quality of communication, and feelings of anger and alienation. More precisely, the total score reflects the quality of attachment to parents and peers [22]. The scale has adequate psychometric properties (internal consistencies α = 0.66–0.86; [21]).

• Childhood trauma, measured with the Child and Adolescent Trauma Screen (CATS; [23])—Youth Report. CATS is a screening instrument to measure of potentially traumatic events based on the DSM-5 criteria for Posttraumatic Stress Disorder (PTSD) in children and adolescents aged 7–17. We used the 16 items a criterion scale that screens for potentially traumatic events. The measure has adequate psychometric properties, with α ranging between 0.88 and 0.94 [23].

Statistical analysis

To estimate the results, we employed a 3 (intervention group: CAU, REThink, REBE) × 2 (assessments: post-intervention, follow-up) multivariate ANCOVA (MANCOVA), using the pre-intervention scores as covariates. This approach has been chosen for three main reasons, namely (1) to reduce Type I error rates associated with multiple outcome testing, (2) because using pre-intervention values as a covariate provides more statistical power and more precise confidence intervals with regard to intervention effects [24, 25] and (3) to minimize the statistical issues associated with Lord’s paradox [26], namely the fact that there are statistically significant differences between groups at post-intervention [24, 25]. The assumption of homogeneity of variances was tested by employing the Levene test and significant main effects and interactions were followed with Tukey-adjusted pairwise comparisons of estimated marginal means, this method being considered as the most preferable method when all pairwise comparisons are performed [27]. For the estimation of the effect size in the case of main and interaction effects, η2p was computed (to correct for the problem of overestimation of the population variance–[28]), while for significant pairwise comparisons, Cohen’s d was employed. Potential effects of moderation were also explored by checking for statistically significant interactions between the intervention group and the specific moderators, which were dichotomized depending on each moderator’s respective cut-off score, as averaged over time (i.e., from post-intervention to follow-up). All analyses were performed using the JASP version 0.14.3 [29] and the SPSS version 26 statistical software [30].

Results

Means, standard deviations and the number of participants at each stage and in each intervention group are presented in Table 1.

Table 1 Descriptive statistics: means (M) and standard deviations (SD) for each group, at each time point, for all variables
Full size table

With regard to the multivariate main analysis (MANCOVA), the within-subjects main effect (time effect) was statistically non-significant, Pillai’s Trace = 0.19, F (16, 102) = 1.56, p = 0.091. However, we identified a statistically significant between-subjects main effect (group effect; CAU vs. REBE vs. REThink), Pillai’s Trace = 0.42, F (32, 206) = 1.71, p = 0.014, η2p = 0.21. Moreover, we also identified a statistically significant interaction effect (time×group efect), Pillai’s Trace = 0.45, F (32, 206) = 1.88, p = 0.005, η2p = 0.22. Despite the overall test for time showing only a trend-level effect, we opted to progress with examining the significant univariate effects.

Following the statistically significant multivariate effects with univariate analyses, statistically significant univariate group effects were identified for emotional control, F (2, 132) = 5.42, p = 0.005, η2p = 0.07, temperamental focused attention, F (2, 132) = 6.40, p = 0.002, η2p = 0.08, and depressive mood, F (2, 132) = 5.03, p = 0.008, η2p = 0.07. Post-hoc Tukey-adjusted pairwise comparisons revealed that averaged over time, the participants in the REThink group had statistically significant higher scores than CAU and REBE for emotional control (REThink–CAU: p = 0.025, d = 0.22; REThink–REBE: p = 0.008, d = 0.25) and temperamental focused attention (REThink–CAU: p = 0.002, d = 0.29; REThink–REBE: p = 0.038, d = 0.21). Moreover, the participants in the REThink group had statistically significant lower scores than CAU and REBE for depressive mood (REThink–CAU: p = 0.039, d = 0.21; REThink – REBE: p = 0.010, d = 0.25).

Univariate analyses of interaction effects revealed statistically significant interaction effects for five dependent variables: conduct problems, (F (2, 132) = 3.41, p = 0.036, η2p = 0.04), functional negative emotions, (F (2, 133) = 7.64, p < 0.001, η2p = 0.10), emotional self-awareness, (F (2, 132) = 7.10, p = 0.001, η2p = 0.09), emotional control, (F (2, 132) = 8.59, p < 0.001, η2p = 0.11) and depressive mood, (F (2, 132) = 5.45, p = 0.005, η2p = 0.07). In the case of emotional self-awareness and emotional control, the interaction effect was explained by an increase in scores in the CAU group from post-intervention to follow-up (p = 0.043, d = 0.43 and p = 0.003, d = 0.56 respectively), while in the case of conduct problems, the interaction was explained by a decrease in scores in the CAU group from post-intervention to follow-up (p = 0.012, d = 0.50).

For functional negative emotions, the interaction effect was explained by both an increase in scores in the REThink group from post-intervention to follow-up (p < 0.001, d = 0.62) and a statistically significant difference between the REThink and CAU groups at follow-up (p = 0.033, d = 0.62), with the REThink group having higher scores than CAU. With regard to depressive mood, the interaction effect was explained by an increase in scores from post-intervention to follow-up in the REThink group (p = 0.004, d = 0.54).

Regarding potential moderators, we found several statistically significant group×moderator interactions. First, we identified a statistically significant interaction between the intervention group and the quality of attachment to parents, F (2, 128) = 3.96, p = 0.021, η2p = 0.05, on the hyperactivity outcome. Post-hoc Tukey-adjusted pairwise comparisons revealed that averaged over time, the participants in the REBE group with a high quality of attachment to parents had higher scores than participants in the CAU group with a low quality of attachment to parents (p = 0.034, d = 1.01). Moreover, participants in the REBE group with a high quality of attachment to parents had higher scores than participants in the REThink group with a high quality of attachment to parents (p = 0.014, d = 1.05).

Second, we identified a statistically significant interaction between the intervention group and child-self-reported parental psychological control, F (2, 128) = 4.75, p = 0.010, η2p = 0.06, on the hyperactivity outcome. Post-hoc Tukey-adjusted pairwise comparisons revealed that, averaged over time, the participants in the REBE group with low parental psychological control reported higher hyperactivity scores than participants in the REThink group with low parental psychological control (p = 0.008, d = 0.94). Moreover, we also identified a statistically significant interaction between the intervention group and child-self-reported parental psychological control, F (2, 128) = 4.08, p = 0.019, η2p = 0.06, on temperamental focused attention. Post-hoc Tukey-adjusted pairwise comparisons revealed that averaged over time, the participants in the REThink group with low parental psychological control had higher scores than participants in the CAU (p = 0.004, d = 0.94) and REBE (p = 0.004, d = 0.99) groups with low psychological control. The post-hoc analysis also revealed that the participants in the REThink group with low parental psychological control had higher scores than participants in the CAU group with high parental psychological control (p = 0.035, d = 0.92).

Third, we identified a statistically significant interaction between the intervention group and childhood trauma, F (2, 129) = 6.47, p = 0.002, η2p = 0.09, on the conduct problems outcome. Post-hoc Tukey-adjusted pairwise comparisons revealed that averaged over time, the participants in the REBE group that had childhood traumatic events maintained higher scores compared to participants in the REThink group (p = 0.002, d = 1.44) an CAU (p = 0.021, d = 1.31) that had childhood traumatic events.

Discussion

The aim of this study was to evaluate the efficacy of the REThink therapeutic game in inducing durable improvements, with regard to prevention and emotional skill-building, in a non-clinical sample of children and adolescents.

Our results showed that the REThink intervention had a durable impact on children and adolescents’ depressive mood. Participants in the REThink intervention outperformed both the participants in the REBE and CAU intervention groups with regard to depressive mood. The results highlighted the fact that the REThink intervention had a durable impact on the children’s ability to regulate their emotions, namely the ability for emotional control. Moreover, the scores in the REThink group significantly decreased from post-intervention to follow-up, with a low magnitude of changes. This result gives support in terms of the capacity of the REThink game intervention for producing longer-term improvements when taking into account the fact that in the primary study [6] the REThink intervention outperformed the CAU, but not the REBE intervention. These improvements are in line with few other trials documenting the maintenance of improvements in depressive symptoms and anxiety up to six months after the game-based treatment-based interventions [2, 31]. It might be that the simulated environment, reward-based tasks and challenges, that offer the opportunity to better consolidate the emotional skills, is in favor of larger, more generalized, and longer-term results than face-to-face traditional prevention. This study brings important contributions to the current knowledge regarding the prevention of emotional disorders in youths, by documenting for the first time the follow-up improvements of an online standalone completely game-based preventive and transdiagnostic mobile tool. Our results offer great promise in terms of implementing large-scale attractive and accessible prevention that can relief the burden of emotional disorders in youths. In addition to emotional regulation and depressive symptoms, we investigated the long-term effect of the REThink intervention on a series of secondary emotional characteristics [5, 8]. Firstly, we found that the participants in the REThink intervention outperformed both the participants in the REBE and CAU intervention groups with regard to temperamental focused attention. Similar to the case of the primary outcomes, this result also lends support to the premise that the effect of the REThink intervention is stable in time, given the fact that in the primary study [6] there was no significant difference between groups, only within groups with regard to this outcome. Current results showed that children and adolescents in the REThink intervention registered higher scores than the participants in the CAU with regard to functional negative emotions (d = 0.62), however, the scores significantly decreased in the REThink group from post-intervention to follow-up (d = 0.62). This result is in high accordance with the REBT theory (and supported by previous research [4]), which postulates the functional negative emotions, which can support adaptive reactions to negative situations, are not the main target of therapy but they can turn into disturbance when combined with irrational belifes.

Regarding potential moderators for differences between groups at follow-up, we found significant group × moderator interactions for the quality of attachment to parents, child-self-reported parental psychological control and childhood trauma on behavioral outcomes (hyperactivity and conduct problems) and emotion-regulation features, such as focused attention.

First, the quality of attachment to parents was a significant moderator for the difference between the intervention groups, with regard to hyperactivity. Participants in the REThink group with high quality of attachment to parents registered higher improvements from the intervention, reflected in their lower scores on hyperactivity symptoms compared to the participants in the REBE group with high quality of attachment to parents (d = 1.05). Second, child-self-reported parental psychological control was also a significant moderator for the difference between the intervention groups with regard to hyperactivity. Even when both groups had low levels of self-reported parental psychological control, participants in the REThink intervention registered lower scores for hyperactivity than the participants in the REBE intervention. With regard to temperamental focused attention, the REThink intervention was superior to both REBE and CAU, when the levels of self-reported parental psychological control were low (d = 0.99 and d = 0.94, respectively). Additionally, participants in the REThink intervention with low parental psychological control obtained significantly higher improvements in focused attention compared to their counterparts in the CAU group that reported high parental psychological control (d = 0.92), a result that also supports the efficacy of the REThink intervention, even when parental psychological control is low. Thus, better long-term effects of the REThink game-based intervention can be obtained in terms of hyperactivity and attentional focus for the youths with a low level of parent-related risk factors, such as secure attachment and low parental psychological control [10]. Our results suggest that children without additional parent-related risk factors are able to generalize their mental health improvements regarding behavior symptoms and cognitive emotion-regulation in the long run, following the RETthink game intervention.

Finally, childhood trauma was a significant moderator for between-group differences at follow-up concerning improvements in conduct problems. For those participants with higher levels of early trauma, children and adolescents participating in the REThink intervention registered significantly higher improvements, reflected in lower scores on the conduct problems outcome compared to those in the REBE group (d = 1.44). Given that REBE was an active intervention and traumatic events were similarly distributed in the participant population, it is promising that the REThink intervention was more effective in reducing conduct problems in the long-term.

Limitations

A potential limitation of this study is that the significant improvements that we observed in the CAU group on conduct problems, emotional self-awareness and emotional control could be owned to the fact that the participants in this group were exposed to other interventions during the post-follow-up period. We had no control over this potentially confounding variable. However, even when taking into account these improvements in the CAU group and the fact that we also employed an active intervention (REBE) as a secondary control group, most of the results in favor of the REThink intervention were maintained at follow-up, lending support to its longer-term efficacy. Another potential limitation is represented by the fact that certain overall effects were marginal (i.e., the MANCOVA time effect) and, because of this, further research is warranted to derive firm conclusions regarding the robustness of the long-term effects of the REThink intervention.

Conclusion and clinical implication

In this trial, we documented the follow-up effects of the REThink therapeutic game in improving emotional symptoms, emotion-regulation and the mental health of children and adolescents, compared to a standard prevention program and care as usual. Our results suggest that the REThink online game can bring relevant improvements in emotional symptoms, emotion regulation and mental health, which are maintained six months after the end of the intervention. The low dropout rate is an important strength of the REThink online game prevention. We consider that the gamified attractive format and also the school context of delivery contributed to maintaining the youth’s engaged. Thus, future developments of the game will include capitalizing on the attractiveness of the game that can support engagement (e.g., improving dynamics, multi-player functions) [32]. Future developments will also need to include specific challenges that can help the youths generalize early on the skills gains within the game to real-life situations.

An important finding of the current study is that the improvements get generalized to behavioral symptoms for the children without additional parent risk factors or early traumas. Thus the population that is most suited to benefit from the standalone REThink online game are the youths without additional family-related risk factors. Since in the current study we have documented the maintenance of emotional symptoms improvements at 6-months, future studies will need to document longer-term effects of the REThink online therapeutic game. Moreover, cost benefits ratio of the REThink online game compared to face-to-face prevention, and the characteristics that contribute to its efficacy beyond that of the traditional prevention programs. Future studies will also need to document the aditive effect that addressing parent factors [33] can have on improving the magnitude and long-term changes regarding the prevention of emotional disorders.