FormalPara Key Points for Decision Makers

Safe TALK training was feasible to implement within a school setting.

Students who attended safeTALK gained suicide-related knowledge, willingness and confidence from pre- to post-assessment up until follow-up four weeks later.

The economic returns are promising in high-risk communities.

1 Introduction

Suicide is a significant global public health problem and ranks among the top 10 causes of death in Western countries and top 5 causes of death among young people [1]. In Australia, suicide is the leading cause of death among Australian youth aged 15–24 years [2] with an annual rate reaching 11.6 deaths per 100,000 population [3] and over 302.9 hospital admissions for self-harm per 100,000 [4], resulting in an annual cost in excess of AU$549.3 million [3, 5, 6]. Many more young people attempt suicide or consider taking their own lives [7] with ratios that vary by age and gender and cannot be estimated reliably, largely, due to the often secret nature of these acts [8].

Evidence suggests that young people at risk of self-harm are less likely to seek support from parents or professionals and more likely to reach out to their peers for help [9]. Peer gatekeeper training holds promise in suicide prevention efforts in educational settings [10, 11]. Gatekeeper training is a key part of an integrated, regionally based approach to suicide prevention in Australia and one of the commonly adopted universal approaches to suicide prevention in secondary schools [12]. It teaches students to serve as peer gatekeepers, being able to recognise warning signs of suicide, respond to concerns and refer to appropriate help resources and care.

Research investigating the impact of educational interventions and specifically gatekeeper training in school settings is growing [10, 11]. A recent systematic review and meta-analysis of studies examining the impact of educational interventions estimated the pooled random effect risk ratio at post-intervention at 0.31 (k = 3, 95% CI 0.15–0.61, I2 = 0%) and 0.63 at follow-up (k = 3, 95% CI 0.42–0.96, I2 = 0%) [10]. In other words, the risk of suicide-related behaviour in young people who were exposed to an educational intervention was 0.31 times (0.63 at follow-up) the risk of suicide-related behaviour among those in the unexposed group. A review of 14 studies on the effectiveness of school-based gatekeeper training found evidence of improvements in gatekeepers’ knowledge; attitudes; self-efficacy; skills; and likelihood to intervene [11].

In 2017, Grapevine Group Mackay, through partner organisation Lifeline, delivered safeTALK, a universal gatekeeper training, to the secondary school students at Mercy College in Mackay. Mackay is a regional town with a population of 180,013 inhabitants [13]. Over the last decade, Mackay has been experiencing one of the highest rates of suicide in the region. A de-identified retrospective audit of suicide deaths extracted from the National Coronial Information System (NCIS) from 2001 to 2014, revealed the rate of 78.1 per 100,000 in Mackay in comparison to 10.2 in Queensland and 8.3 per 100,000 in Australia [14].

The safeTALK training included a 3-h workshop that was developed by Living Works Education in 2006. It included the use of presentations, videos, discussion and questions designed to help: (a) recognise warning signs of suicide; (b) move beyond common tendencies to miss, dismiss, and avoid suicide; (c) notice and respond to situations in which thoughts of suicide might be present, apply basic TALK steps (Tell, Ask, Listen, and KeepSafe); and (d) connect the suicidal person with suicide first aid help and further community resources. In the past, safeTALK was delivered to teaching and other staff at several Mackay region schools as well as at three mainstream secondary schools in Alice Springs [15]. The training is one of eight training programs in the USA, which is considered as a gatekeeper suicide prevention for high schools, higher institutions of learning, and the community at large [16, 17].

Despite safeTALK being implemented in numerous settings and demonstrated potential to change participants’ knowledge and skills in suicide prevention, research is dominated by proximal outcomes related to knowledge, attitudes, self-efficacy or perceived skills of gatekeepers following training [15,16,17,18,19,20]. Further work is warranted to determine how best improvements in knowledge and skills can be translated into behavioural changes and suicide rates [21] and where these programs exhibit economic viability. This study aims to address this gap and evaluate safeTALK training delivered within a secondary school setting to estimate its cost and effect on suicide-related outcomes (knowledge, attitudes, behaviours, including help-seeking and helping others at risk) and to model potential return on investment.

2 Methods

2.1 Analytical Framework

The analytical framework for this study was guided by the Kirkpatrick Model (KM) that favours a two-phase sequential approach to evaluation [22]. The evaluation consisted of two phases, ex-post and ex-ante, which corresponded to four levels in the Kirkpatrick Model: reaction (Level I), learning (Level II), behaviour (Level III), and results (Level IV). Phase I was a pre-post, follow-up analysis using a mixed-method questionnaire administered on three occasions: immediately prior to the training (Time 1), immediately after the training (Time 2), and about 4 weeks after the training (Time 3). In the second phase, a Markov model with an integrated Return on Investment (ROI) framework extrapolated potential impacts of safeTALK on incidents of hospitalised self-harm in a regional town of Mackay, Queensland state and Australia nationwide. These three settings differ by the rate of suicide and likely exhibit different return on investment.

2.2 Phase I: A Pre-post, Follow-Up Analysis

2.2.1 Study Setting

The safeTALK training was delivered to students in Year 10 at the Mercy College in Mackay in November 2017. The college was selected on the basis of convenience sampling. SafeTALK was offered to students of appropriate age (15–16 years). Students had a choice to opt in with the permission of their parents or guardians.

The training included the use of presentations, videos and discussions. A pocket card containing the suicide alert steps was provided to participants at the end of the training. The workshop was facilitated by one safeTALK and Applied Suicide Intervention Skills Training instructor (AM), and, for increased safety, Guidance Counsellors were present within easy reach. All students who attended the training and provided informed consent were eligible to participate in the evaluation. There were no other exclusion criteria.

2.2.2 Measures

A mixed-method questionnaire (Supplementary Appendix A), utilising both quantitative and qualitative means, was adopted from Bailey et al. [15] and Wyman et al. [23] to assess students’ reactions (Level I of KM), including subjective views and satisfaction with the training; and the degree to which students who attended safeTALK sensitised to their role as gatekeepers, which was assessed through learnings (Level II of KM), including changes in knowledge about suicide, suicide risk factors and attitudes toward suicide and suicide intervention. Lastly, a questionnaire assessed behaviour changes (Level III of KM), including help-seeking and helping others at risk.

Measures of knowledge, confidence, willingness, and likelihood of help-seeking have previously been applied in an Australian high school sample and described elsewhere [15]. Help provision was assessed using six questions selected from the survey of knowledge, attitudes, and gatekeeper behaviours for suicide prevention in schools developed and validated in the USA [23].

2.2.3 Data Collection

Participants completed the questionnaire on three occasions: immediately prior to the training (Time 1), immediately after the training (Time 2), and about 4 weeks after the training (Time 3). Items assessing demographic characteristics were administered with the pre-test. Training satisfaction accompanied the post-test assessment (Time 2). Participants were informed that the surveys would remain confidential.

2.2.4 Data Analysis

Data were analysed using paired samples t tests with a level of statistical significance at 0.05. There was no attempt to fit a model due to relatively small sample size and demographics being close to constant. All analyses were performed in Stata and SPSS and reported separately for Time 1, Time 2 and Time 3.

2.3 Phase II: Return on Investment (ROI) Analysis

2.3.1 Model Structure

A Markov cohort model with a three-month-cycle was used to estimate the impact of safeTALK training on incidents of hospitalised self-harm in a regional town of Mackay, a state of Queensland and across the Australian nation. The impact of the training was compared to no training option (status quo). The model followed a hypothetical cohort of students (aged 15) in one school year, for a 5-year period. After consultation with the school representatives, it was further assumed that 80% of the students would likely undertake the training during the first year as part of their curriculum.

The model used a matrix of transition probabilities between three health states programd in Microsoft Excel: general population, hospitalised self-harm and completed suicide. Results considered the potential ROI associated with a reduction in new cases of hospitalised self-harm and suicide (Fig. 1). The difference in costs over 5 years was divided by the cost of implementing safeTALK, to obtain the primary outcome measure—ROI ratio.

Fig. 1
figure 1

Markov model of health states and possible transitions during a 3-month cycle. The ellipses represent the possible states and the arrows correspond to transition probabilities

Full size image

Individuals in the target population either started the model in the general population state or hospitalised self-harm state at age 15. Applying the Australian Institute of Health and Welfare (AIHW) statistics, 3.20% of the target population started in the hospitalised self-harm state in Mackay model; 0.42% in the Queensland model and 0.34% in the Australian model and the remaining in the general population (Table 1). Within the next three months, individuals could remain in the general population state, hospitalised as a result of self-harm, or die from suicide. Those with previous hospitalised attempts had an increased risk of readmission within the next year (15.2%) as well as dying from suicide (1.7%) [24] (Table 1). Annual rates were adjusted to reflect three-month probabilities [25]. Death by suicide could occur at any time point during the model duration, and was not adjusted for mortality by natural causes, due to the low rate among this population group. The model terminated when the cohort reached the average age of 19 years, which is the age when most young people had left school.

Table 1 Model parameters for Mackay, Queensland and Australia
Full size table

2.3.2 Model Inputs

The parameters used in the model were obtained from numerous sources including: published literature, specifically a meta-analysis of hospital presenting self-harm, risk of fatal and non-fatal repetition [24]; expert opinion; the Australian Bureau of Statistics (ABS); AIHW and the National Coronial Information System (NCIS) (Table 1). In the absence of evidence related to hospitalised self-harm and completed suicide data among those in the non-self-harmed populations, assumptions were calibrated (Supplementary Appendix B) and tested in the sensitivity analysis (see below).

2.3.3 Cost Offsets

The model considered direct, indirect and intangible costs. The direct costs included ambulatory transport, attendance at emergency department (ED) and inpatient care (combined into ‘Immediate treatment for self-harm/suicide’ category in ROI model), ongoing psychological treatment, police, coroner activities, and funeral. Indirect costs or productivity losses related to the premature loss of life, and intangible costs, referred to as grieving costs, were considered in an alternative model and reported separately (Table 2). All costs were applied on a per incident basis in the cycle in which they occurred. The unit costs in AU$, using 2014 as a reference year, were derived from published literature and data from AIHW (Table 2). Costs were subsequently adjusted to reflect AU$2017–2018 using Health CPI index of 1.13 [26]. The AIHW dataset contained inpatient episodes with a reported ICD-10-AM principal diagnosis in the community injury range S00-T75 or T79; the first encountered external cause code in the intentional self-harm range of X60–X84 and a separation date between 1st January 2014 to 31st December 2014 (inclusive).

Table 2 Model costs (in Australian dollars AU$)
Full size table

2.3.4 Intervention Cost

The authors affiliated with Grapevine Group Mackay (LM, AM), estimated it cost AU$35 per participant to deliver the safeTALK training. This cost includes travel, printing of booklets and cards containing contact information for local healthcare services and non-clinical groups. Given the program was delivered by volunteers, other costs might include training facilitators and implementation. A likely incremental value of these costs of ± AU$10 per participant was added to the total cost and tested in a sensitivity analysis.

2.3.5 Relative Risk Reduction: Effect of safeTALK

The effectiveness of safeTALK in reducing hospitalised self-harm was assumed at 0.60 and based on the pooled random effect risk ratio for the signs of suicide (SOS) prevention program at post-intervention [37,38,39] (Fig. 2; Supplementary Appendix C). SOS is a universal training designed for middle-school (aged 11–13) or high-school (aged 13–17) students to prevent suicide by raising awareness of depression and other risk factors. Similar to safeTALK, the main teaching tool is a video that shows real-life stories about the right and wrong ways to react when a friend exhibits certain behaviours, followed by discussions about suicide. Due to the lack of the safeTALK data on effectiveness in reducing suicide attempts, as well as similarities in the design and delivery, SOS effectiveness was used as a proxy. After consultation with safeTALK facilitators, the training effectiveness was assumed to last for the first four cycles (Year 1) and nullified afterwards. This assumption was tested in a sensitivity analysis.

Fig. 2
figure 2

Forest plot of fixed and random effect models of three SOS trials

Full size image

2.3.6 Other Parameters

The model was gender neutral and used an average person as a unit of analysis. The model incorporated two perspectives—the narrower one, health and justice systems perspective, and the broader one, societal. Costs and cost savings were discounted at five per cent per annum, in line with the current Australian Health Technology Assessment guidelines [40]. Half cycle correction using the trapezoidal method was applied across transitions [41]. It was further assumed that the training was delivered to 80% of the target population, such as the costs and cost-offsets to the unreached 20% of the population would be unaffected regardless of the training status.

2.3.7 Sensitivity Analysis

Given the exploratory nature of the current analysis, a complicated multivariate probabilistic analysis was deemed inappropriate at this stage. Univariate analyses determining the effect of varying key assumptions on final results were undertaken. The cost of delivering the training per participant was varied by ± AU$10 in accordance with the experts’ opinion. Alternative discount rates were 7% and 0%. Probabilities of seeking immediate treatment, psychological treatment, police and ambulance attendance were varied by ± 15% as per consultation with experts. Probabilities of subsequent hospitalised non-fatal self-harm events and fatal suicide events were varied according to the 95% CIs 13.2–17.5 and 1.1–2.5, respectively. Similarly, the effect size was adjusted to reflect the upper and lower boundaries of the 95% CI 0.44–0.82; the training effectiveness was adjusted to last 6 and 18 months.

2.4 Ethics

The study received approvals from the Catholic Diocese of Rockhampton and the Central Queensland University Human Research and Ethics Committee (Reference number: 0000020781).

3 Results

3.1 Phase I: A Pre-post, Follow-Up Analysis

3.1.1 Sample Characteristics

Twenty-eight students completed the questionnaire at Time 1, 22 at Time 2, and 12 at Time 3. Comparisons between Time 1 and Time 2 are based on 22 participants. Twelve students completed the follow-up questionnaire at Time 3; however, two of these had not completed the Time 2 questionnaire. Time 2 to Time 3 comparisons are based on data from 10 students who had completed both. The main reason for participants not completing follow-up questionnaires was the timing. The follow-up questionnaires were sent out after students had completed their academic year, which significantly reduced the number of returned follow-up questionnaires.

Most (82.1%) of the participants were female; the mean age was 15.1 years (SD = 0.36). All but one lived with their parent(s) during term, with one living with friends; and all lived with their parent(s) during school holidays. All participants were in Year 10. Most (85.7%) were born in Australia (two in Philippines, one in Papua New Guinea and Zimbabwe), with similar statistics for their parents (82.1% of mothers and 85.7% of fathers born in Australia). Most (85.7%) spoke English as their main language at home, and none identified as Aboriginal or Torres Strait Islander.

3.1.2 Effectiveness

3.1.2.1 Knowledge, Confidence, Willingness, and Help-Seeking

Scores on all measures increased from Time 1 to Time 2, and remained stable from Time 2 to Time 3, with the exception of help-seeking, which decreased significantly from Time 2 to Time 3 (Table 3).

Table 3 Comparison of scores on measures of knowledge, confidence, willingness, and help-seeking as mean (SD)
Full size table

On average students named four warning signs of suicide (range 1–8) before the training (Time 1) and five (range 2–16) after the training at Time 2. Figure 3 depicts a word cloud of the warning signs identified by the students. The most commonly used words were depression; antisocial, as “aren’t as involved with the community as usual”, “not engaging with friends or family”, “don’t go out”, “spending more time by themselves”; self-harm or suicide experience; distancing from others; sadness; isolation; anxiety; and negativity, as “negative talking”, “negative attitude or approach to life”.

Fig. 3
figure 3

Word cloud “Warning signs of suicide” identified by students. Larger font size is an indicator of the prevalence of the word identified

Full size image

When asked how likely students were to seek help from different people in their lives, the most likely candidates were friends (not related to the person), parents, mental health professionals and phone helplines (Fig. 4). Note that a score below four indicates that, on average, students would be at least partly unlikely to seek help from this source. This is the case for boyfriend or girlfriend and teacher. The lowest score was for “I would not seek help from anyone” (M = 2.18, SD = 1.54). Scores for likelihood of seeking help from each of these categories did not change from Time 1 to Time 2, or from Time 2 to Time 3 (smallest p value = 0.075 for phone help line from Time 2 to Time 3).

Fig. 4
figure 4

Mean (and 95% CI) likelihood of seeking help from each source at Time 1. Reference line at 4 indicates midpoint of the scale

Full size image

At Time 1, the most commonly reported person that the students reported seeking help from in the past 2–4 weeks for advice and help for a personal or emotional problem were friends (79%), parent(s) (68%) and other relative/family member(s) (32%). Less than six of the 28 respondents at Time 1 reported seeking advice and help from any of the other sources.

3.1.2.2 Barriers to Help Seeking

At Time 1, the main barriers to help-seeking were being too embarrassed or shy, concern about what other people might think of you, concern that the person might tell others, and concern that the person might judge you (Fig. 5). The least important barriers were that the person is too far away, that it is hard to get an appointment, that nothing can help, concerns about what the treatment may involve, concerns that what the person is saying may be wrong, and the cost of seeing the person. The only barrier that changed over time was “concern about what the treatment could involve”, which became more important between Time 2 and Time 3, t(9) = 2.54, p = 0.032.

Fig. 5
figure 5

Mean (and 95% CI) rank of importance of barriers to help seeking. Higher scores indicate that the barrier is ranked as less important

Full size image
3.1.2.3 Help Provision

As seen in Table 4, 61% of respondents reported that they thought another student’s behaviour might have indicated that they were considering committing suicide, but only 21% of respondents reported asking another student about this. In contrast, 89% of respondents reported that they thought another student’s behaviour might indicate that they were very distressed or depressed, and the same proportion reported talking to students about their distress or depression. These questions were not asked of respondents at Time 2, and thus changes over time were not tested.

Table 4 Perceptions of, and discussions with, friends who may be considering suicide, or being very distressed or depressed [relative (%); absolute number (n)]
Full size table

Wilcoxon pairwise comparisons revealed that the most common reason for asking another student about their suicidal thoughts was when the respondent had a feeling that something was wrong (Table 5). The next most common reason was that the student seemed depressed, followed by the student having experienced a traumatic event, and finally the student seemed depressed (all differences statistically significant, lowest \(\left| Z \right|\) = 2.45, p = 0.014).

Table 5 Frequency of reasons for asking another student about suicidal thoughts in the last six months [relative (%); absolute number (n)]
Full size table

Wilcoxon pairwise comparisons also revealed the most common actions taken when respondents identified a student with suicidal tendencies during the past 6 months (Table 6). The most common response was to spend some time listening, followed by providing appropriate information, and then convincing the student to seek help. The least frequent options were not significantly different to each other in terms of frequency: notifying appropriate referral services, asking about the suicidal thoughts and taking the student to the school counsellor or other resource (lowest significant \(\left| Z \right|\) = 2.18, p = 0.029). These questions were not asked at Time 2, and thus changes over time were not tested.

Table 6 Frequency of taking action(s) for every student identified with suicidal tendencies during the past 6 months, valid % (n)
Full size table
3.1.2.4 Training Acceptability

Most students (82%) found the training very enjoyable. About half (55%) found the content somewhat upsetting, with the remainder finding the content not upsetting at all. Most (91%) reported that it was good having school teachers/welfare staff in the training workshop. Most (82%) reported that they were well prepared to help their friends after the training, with none reporting that they were not at all prepared; 77% reported that the training had equipped their friends to help them in the future. All reported that the training was very worthwhile, and all would recommend the training to others. Six out of 21 respondents who answered the question said that they would like to discuss their feelings about either the training or the questionnaire with a member of staff.

3.2 Phase II: Return on Investment (ROI) Analysis

The cumulative 5-year return per 1 AU$ invested into safeTALK program could potentially result in AU$1.45 return in Mackay, AU$0.19 in Queensland and AU$0.15 across Australia. This is a conservative assessment of the savings; as no impact on hospitalised self-harm was assumed beyond Year 1. From the broader perspective, which included indirect and intangible losses to families and friends, a ROI could increase to AU$31.21, AU$4.05 and AU$3.28 respectively (Fig. 6).

Fig. 6
figure 6

Return on investment (ROI) per dollar invested in safeTALK by location

Full size image

The largest return results from indirect cost avoided (ranging from AU$28.22 to 2.97 per AU$1 invested) and to health system (ranging from AU$1.31 to AU$0.14), followed by direct and intangible losses to families (ranging from AU$1.54 to 0.16) and to justice system (ranging from AU$0.14 to AU$0.01) across all locations (Table 7).

Table 7 Return on investment (in Australian dollars AU$2017–2018) by location and sector
Full size table

3.2.1 ROI Mackay

In Mackay, the model estimates a ROI of AU$1.45 at the end of the 5-year period from health and justice systems’ perspective (Table 8). From the broader perspective that included indirect and intangible losses, a ROI increases from AU$1.45 to AU$31.21. Almost all the ROI is due to productivity losses avoided. However, even from a narrower health and justice systems perspective a ROI would be AU$1.31 and AU$0.14 for every dollar spent on the program, respectively.

Table 8 Direct, indirect and intangible net costs in Australian dollars—AU $2017–2018, Mackay (target population 2561)
Full size table

3.2.2 ROI Queensland

In Queensland, the model estimates a ROI of AU$0.19 from health and justice systems’ perspective (Supplementary Appendix D, Table 1). When the model incorporates indirect and intangible losses, a ROI increases from AU$0.19 to AU$4.05, again largely due to productivity losses avoided. From a narrower health and justice systems perspective, a ROI for Queensland would be AU$0.17 from health system perspective and AU$0.02 from justice system perspective.

3.2.3 ROI Australia

Nationwide, cumulative 5-year return per 1 AU$ invested into safeTALK program results in AU$0.15 from health and justice systems’ perspective (Supplementary Appendix D, Table 2). From a broader perspective, a ROI increases from AU$0.15 to AU$3.28. From a narrower health and justice systems perspective a ROI would be AU$0.14 from health system perspective and AU$0.01 for every dollar spent on the program from justice system perspective.

3.2.4 Sensitivity Analysis

From a narrower health and justice systems perspective, uncertainty with respect to the effectiveness of safeTALK (as measured by RR and the duration of the effect), as well as associated cost of delivering the training had the strongest potential impact on ROI ratios (Supplementary Appendix E). The discount rate had a marked, but much smaller, impact on ROI ratios. The probabilities of undergoing psychological treatment, police and ambulance attendance had much smaller impact on ROIs. From a societal perspective, productivity loss beared significant uncertainty in line with the training effectiveness and its cost.

4 Discussion

This study evaluated the safeTALK program delivered to secondary school students in a regional town of Mackay. The evaluation provided the cost of delivery, before and after changes in suicide-related outcomes including, changes in knowledge and attitudes, changes in behaviour including help-seeking. Further, the study modelled potential impacts on incidents of hospitalised self-harm and estimated return on investment across three settings, Mackay, Queensland and Australia.

The findings confirmed prior research [15,16,17,18,19,20], suggesting that students who attended safeTALK training gained suicide-related knowledge, willingness and confidence from pre- to post-assessment up until follow-up assessment four weeks later. In line with other research [9], students sought help, or would likely seek help, from friends and families and less likely from teachers or professionals. The most important help-seeking barriers were being judged or feeling shy, whereas the least important barriers were professional help, such as cost or access. Likelihood of help-seeking behaviour improved from pre- to post-assessment, with a not significant drop at follow-up. Friends and peers remained as a “go to” support and help group. Students seemed reluctant to openly discuss suicide. They expressed willingness to support their peers but talking about suicide seemed to be a barrier. These findings highlight the significance of education and empowerment of those in close circle including friends and family as the first line of defence.

In the baseline model, from a health and justice system perspective, safeTALK administered to a 15- to 19-year-old population demonstrated a cumulative 5-year return of AU$1.45 per 1 AU$ invested in Mackay; AU$0.19 in Queensland and AU$0.15 across Australia. The additional costs of delivering the training were offset by a reduction in the costs to the health care and justice systems of treating future self-harm events. The ratio became stronger when taking account of productivity losses, associated with premature death, and grieving costs to families and friends. ROI increased to AU$31.21, AU$4.05 and AU$3.28, respectively. The largest return resulted in indirect cost avoided (ranging from AU$28.22 to AU$2.97 per AU$1 invested) and to health system (ranging from $1.31to $0.14), followed by direct and intangible losses to families (ranging from AU$1.54 to AU$0.16) and to justice system (ranging from AU$0.14 to AU$0.01) across all three settings.

Although many youth suicide prevention programs have been developed and implemented, only a few have been formally and rigorously evaluated [10]. Evaluation efforts in the field have been very limited. Literature further points to a dearth of evaluations showing effectiveness of safeTALK and other gatekeeper training in decreasing suicide ideation, suicide attempts or deaths by suicide [42].

A strength of this study was the design and use of a Markov model to capture the dynamic changes to health status on a population basis. The model considered the high variability of suicide rates known to exist regionally. Further, this study utilises a comprehensive approach to evaluation, which included a fusion of the two meta-methodologies: ex-post and ex-ante following the Kirkpatrick Model for evaluating the outcome of training programmes. Integration of the four levels of outcomes enabled thorough evaluation of the safeTALK training. Information was synthesised regarding the extent to which students perceived the training useful, learned the relevant skills, transferred this learning to improve help-seeking and provision behaviours. The current study is one of a few evaluations, which attempted not only to ascertain changes in knowledge and attitudes of participants but also attempted to ascertain economic returns.

4.1 Limitations

When interpreting the findings from this evaluation, several limitations are worth noting. First, a pre-post, follow-up design was applied to assess suicide-related outcomes. We acknowledge that assessing the impact of gatekeeper programs on rates of self-harm and suicide in the community using a randomised control trial is extremely difficult. The low base rate of suicide makes it difficult to implement randomised trials that have sufficient statistical power when suicide is used as the major outcome. Instead, we implemented a pragmatic study in real-life context and assessed intermediate outcomes that are easier to assess, including changes in people’s knowledge of suicide warning signs, their attitudes toward seeking or providing help, and the referral of high-risk individuals to treatment. This design is unable to control for confounding variables and, hence, establishing causation. However, the methodology of the current study is feasible and acceptable to sustainable implementation in schools as part of continues quality improvement. Small sample size and short follow-up periods are among other limitations of the current study. The instruments used to measure knowledge, confidence, and willingness had been used in the past [15], although their psychometric properties had not been validated.

The following limitations related to the modelling part of the evaluation. First, the relative-risk reduction was polled across three randomised control trials of the Signs of Suicide (SOS) program [37,38,39], due to the lack of relevant data on safeTALK. Both trainings are classified as being universal prevention approaches to assist in identification of at-risk individuals, both address suicide and self-injury through showing of a video and a guided discussion. However, the SOS program was specifically designed for young people and delivered in the US settings; therefore, while being the next-best alternative with available and relevant evidence, it might not necessarily reflect the true effectiveness of safeTALK. Second, the length of one cycle in the model was three months and was guided by the availability of most parameters. It is possible that an individual makes an attempt and a re-attempt immediately after. In the current model, the re-attempt will be postponed, which may bias the results in favour of the training. Given the available evidence, the shortest cycle length of three months was selected to reduce this bias. Third, relative incidence rates for suicide in suicidal persons and for re-attempts were not available for Australia, so information from a meta-analysis of international evidence was used instead. Fourth, the coronial inquiry, police, and ambulance costs used in this analysis have been derived from published literature. However, these estimates are also derived using various assumptions that may impact on the ultimate accuracy of Australian-based values. Friction-cost method shall be considered in the future research to quantify productivity losses as an alternative method to human capital approach applied in the current study. The model was gender neutral and it was assumed that safeTALK does not discriminate on the basis of sex or ethnicity. However, evidence suggests that rates of suicide and self-harm vary by sex and ethnicity. In this study, small and unbalanced sample size did not allow the analysis of the impact of safeTALK by sex and can be subject to future research. It can be expected that the students increase their use of within school counselling services and community support due to increased awareness and connectivity. At this stage these costs were excluded due to speculative nature of this assumption that warrants further investigation. Lastly, given the lack of robust data on most parameters and exploratory nature of the current study, probabilistic sensitivity analysis was not preformed, but it ought to be considered in the future studies.

4.2 Implementation Lessons and Future Considerations

SafeTALK delivery is the most efficient when run in small groups, 15 or fewer students. When delivered as an independent 3-h training to a large cohort of students, safeTALK can significantly disrupt the timetable and impact on consistent delivery of classes. To ensure smooth delivery without prolonged windows between groups, at least 2–3 safeTALK facilitators are advisable to reduce disruptions.

Delivery of the training within school days can increase participation and response rates from both students and staff. Young people would less likely attend a 3-h training after school, given their sporting, work and family commitments. It also poses a further challenge for students catching buses to and from schools. In addition, staff are heavily committed after school due to meeting schedule and extra-curricular activities. From the beginning, full support of the leadership team, teachers and a whole-school commitment is vital. To overcome these barriers and increase participation and response rates among students and staff, from 2019 Mercy College have scheduled in the strategic plan for safeTALK to be delivered as a proactive mental health initiative.

Future evaluation of the naturalistic implementation of safeTALK within a larger population with a control site is required to estimate the effectiveness of the training. Likewise, empirical estimates of final outcomes rather than intermediate outcomes are warranted. Future evaluation studies would benefit by adapting comprehensive evaluation techniques combining both ex-post and ex-ante approaches and assessing the degree to which these programs: have sensitised gatekeepers to their role in identifying and appropriately helping those who might be at risk of suicidal behaviour; result in appropriate identification and referral of suicidal individuals; and impact suicide and suicide attempts.