Background

Patient safety and human factors

Patient safety needs to be a central tenant of health care education. The definition from the World Health Organisation (WHO) gives clarity to the purpose of patient safety, ‘the reduction of risk of unnecessary harm associated with health care to an acceptable minimum’ (WHO 2009). This definition is derived from evidence that suggests that patients experience harm in their interactions with health care (Leape 1994; Kohn et al. 2000; Vincent et al. 2001). The definition encompasses every aspect of medicine and healthcare delivery and manifests itself in every patient encounter. It focuses on the outcomes of healthcare but has importance to educators whose intention is to develop safe future practitioners. Whilst the definition is clear and the universal expectation of risk reduction in terms of outcomes undisputed, the mechanisms that result in harm or how harm can be reduced, are not fully understood, especially in the setting of undergraduate medical education.

Aligned to the patient safety movement there is a drive to understand human factors as a component of patient safety (Jackson and Flin 2009). Human factors focus on mechanisms that influence behaviour. The widely accepted definition from the Health and Safety Executive (UK industrial safety regulator) is: ‘Human factors refer to environmental, organisational and job factors, and human and individual characteristics which influence behaviour at work in a way which can affect health and safety.’ (Health and Safety Executive 1999). Human factors research examines how healthcare workers interact with systems, as well as the physiological and psychological characteristics, which influence behaviour at work. In terms of education, human factors are a key component of professional development. They include communication, decision-making, teamworking and situation awareness (Jackson and Flin 2009). Understanding these factors as mechanisms that influence learning and behaviours in the context of patient safety is a component of the process of developing safe future practitioners.

Evidence for mechanisms of learning in patient safety

The current evidence for patient safety education in medical students in the published literature offers evaluations of patient safety educational interventions that have been mainly initiated in single institutions and measure changes in knowledge, attitudes and skills (Halbach and Sullivan 2005; Kerfoot et al. 2006; Madigosky et al. 2006; Patey et al. 2007; Hall et al. 2010). In a review of the undergraduate patient safety literature, Wong (Wong et al. 2010) identifies some of these papers as offering outcomes that impact on behaviours and patient outcomes. This analysis is based on the presumption that self-reporting equates to evidence of behaviour change. In Wong’s analysis of impact on patients an article is identified that reports students engaging in quality improvement via audit, the patient outcomes were effected by health professionals and the students’ audit work was a part of a team wide improvement process (Gould et al. 2002). This seems more appropriate to consider as skill acquisition rather than change in patient outcomes.

Most studies originate from developed health care systems in the USA, Europe or Australia. As reported in the literature, these interventions offer descriptions of the interventions and the outcomes used for evaluation. They are positivistic in nature, testing interventions by identifying changes in levels of knowledge or skills. In order for other educators to use these interventions to their maximum benefit in their own institutions, an understanding of the mechanisms that influence the educational intervention is required and this is not present in the majority of the articles. A minority of studies have linked theory to practice using a constructivist epistemology. One example uses activity theory (de Feijter et al. 2011) where the results identify components of patient safety.

This evidence base at pre-registration level is mirrored across medical education where there is a lack of research that combines understanding the mechanisms and theoretical basis of learning alongside patient outcomes. This may relate to the different ontological perspectives that have traditionally influenced research in this area and the resources available for educational research. In the patient safety context this results in a tension between the patient outcomes focused evidence from the patient safety and quality improvement disciplines and the type of evidence that is currently available from medical education.

This articles focuses on how an approach using the principles of realist research can offer a methodological approach to this dilemma. Realist evaluation and research explore the mechanisms that result in the outcomes of healthcare. Realist research uses the following equation: Context + Mechanism = Outcome (Pawson and Tilley 1997). The key focus is on mechanism. Mechanisms are often hidden and are sensitive to variations in context. In traditional empirical research an intervention is tested via its effect on a specific outcome. The mechanism of action or theoretical foundation of the intervention is not always investigated. Realist research seeks to identify patterns that give an indication of the mechanisms involved, that can be developed into plausible theories. It requires cycles of investigation to explore and refine the developing overarching theory (Pawson and Tilley 1997).

The aim of the study was to identify and explore specific mechanisms involved in patient safety learning in a single cohort over a 5-year curriculum.

The research question is; which mechanisms influence student learning in the context of patient safety? The objectives included identifying learning mechanisms and then completing tests of association between measures of the mechanisms and different learning outcomes.

Methods

Study design

This study followed a single year group through a 5 year curriculum collecting data at three points to answer the research question. These occurred during the 5-year MBChB course at year 1, year 3 and year 5. The purpose was to complete the research in cycles where the results from one data source informed the choice of method for the subsequent data source. The realist approach involved identifying patterns in study one from qualitative data (Pawson and Tilley 1997). From this an overarching theory was developed that was explored using different outcomes from Kirkpatrick’s levels (Kirkpatrick 1967; Barr et al. 2000) in the subsequent data collection cycles. The results of each data source could then inform the continued development of the theory. The different patterns identified could be identified and explored within each cycle and assimilated into the overarching theory of which mechanisms are involved in patient safety learning for medical students.

Setting and participants

The cohort involved in the study included the medical students who commenced the medical MBChB course in 2005 and completed it in 2010. Three data sources were used to answer the research question, when the students were in year 1 in 2006, in year 3 in 2008 and in year 5 in 2010.

Ethical approval

For each of the studies all the students in the cohort were invited to participate. Ethical approval was sought and granted from the University Research Ethics Committee. This involved three different applications over 5 years.

Data sources and analysis

Cycle 1

Patient safety context in cycle 1

First year medical students participated in ward based patient safety activities aligned to the year 1 clinical skills programme. The intention was for students to transfer skills from the classroom to clinical settings. The students had developed skills in simulation in the first semester of year 1 and the ward based activities occurred in the second semester. The patient safety activities were based on students using individual patient safety tools including the SBAR communication tool (Institute for Healthcare Improvement 2012), Hand hygiene audits, Ask Me 3™ (National Patient Safety Foundation 2012) and medicines reconciliation (Vira et al. 2006).

Data source for cycle 1

At the end of the ward sessions there was a wrap-up week, which included evaluation sessions.

Students were invited to hand in completed sheets from the patient safety activities and were invited to participate in focus groups to discuss their experiences of learning about patient safety.

The Framework methodology was used to analyse the focus group data (Ritchie and Spencer 1994). It has been used in the context of applied qualitative research and enabled the interpretation of the data to identify patterns, which could then be explored as students progressed through the curriculum.

Cycle 2

Patient safety context in cycle 2

In the second and third years of the curriculum, this cohort of students used patient safety tools as part of their ward based clerking activities. The tools were included within the course’s structured, anonymised patient record and reflection documents.

Data source cycle 2

The mechanism of reflective thinking was identified in cycle 1 as a pattern in the data impacting on students learning about patient safety. Another mechanism that impacted on student learning about patient safety in cycle 1 was their knowledge and attitude towards patient safety. In cycle 2 the patterns identified in cycle 1 were explored at the outcome level of knowledge and attitudes with the same cohort who had participated in cycle 1.

During year 3 of the MBChB course students were asked to complete validated questionnaires that measured the two types of reflective thinking; critical reflection and reflection (Kember et al. 2000). This questionnaire is based on Mezirow’s model which identifies a vertical framework of reflective and non-reflective thinking (Mezirow 2000). It identifies four types of thinking in the vertical hierarchy:

  1. 1.

    Habitual action

  2. 2.

    Understanding/thoughtful action

  3. 3.

    Reflection

  4. 4.

    Critical Reflection

The students also completed a questionnaire that evaluated knowledge and attitudes to patient safety (Patey et al. 2007). It includes two aspects general knowledge and feeling and a series of questions that are informed by the theory of planned behaviour (Azjen 1991).

General knowledge and feelings

  • Level of Knowledge

  • Knowledge of actions to take

  • Feelings about error (two sections: 1. feelings about making an error 2. difficulty of communicating about error)

Theory of planned behaviour

  • Personal attitudes

  • Safety in the workplace

  • Personal influence

  • Future intentions

The questionnaires were completed and given directly to the principal researcher who identified if there were any missing data. The reliability of the self-administered questionnaires was tested via the use of Cronbach alpha. If the coefficient was acceptable, the data was analysed by an initial examination of the distribution of the data from the questionnaires and then tests of association were carried out using Pearson’s product moment correlation coefficient.

Cycle 3

Patient safety context

All final year students participate in an exercise where they act as a junior doctor in a simulated ward. The exercise involves the students managing a three-bedded bay for 20 min with standardised patients and trained ward staff. The purpose of this exercise is to assess and give feedback on technical and non-technical skills relating to safe practice.

Data source cycle 3

In cycle 3 the patterns identified in cycles 1 and 2 were explored at the level of skills and behaviours with students from the same cohort who had participated in cycles 1 and 2. In cycle 2 the analysis identified two levels of reflection as mechanisms of learning in the context of skills and error behaviours.

The ward simulation exercise enabled assessment of skills and behaviours in a simulated setting. It is assessed via video link by two remote clinical assessors using a validated tool, (Ker et al. 2006; McIlwaine et al. 2007). The assessors are trained prior to assessing students and are trained in the use of the assessment tool. In addition, the assessors noted error behaviours using a descriptive method from the direct observations of the assessors (Morey et al. 2002; McCulloch et al. 2009). These errors were then categorised using a published error framework (Reason 2008). Errors were included separately following the identification of these as an associated mechanism in cycles 1 and 2. The exercise was a compulsory part of formative assessment within the course, so all the assessment documents were checked for missing data and the assessors asked to complete missing data at the time of the exercise. After completing the exercise students completed the reflective thinking questionnaire (Kember et al. 2000). Students were also asked about how they perceived feedback on the exercise would influence their future behaviours in relation to the exercise. The reliability of the self-administered reflection questionnaire and the error data was tested via the use of Cronbach alpha (Morey et al. 2002). The reliability of the ward simulation exercise assessment tool was tested via the calculation of a Generalisability coefficient. If these were acceptable, tests of association were carried out using Pearson’s product moment correlation coefficient.

The structure of the data sources and the study overall is shown in Fig. 1.

Fig. 1
figure 1

Sequence of the data collection cycles within the curriculum

Full size image

Representativeness of the samples

In the second and third data sources the representativeness of the sample was calculated by obtaining mean scores from summative assessments for the participants and comparing the means scores with that of the cohort overall using 95 % confidence intervals. Histograms were also plotted (Fig. 2).

Results

Data source 1

Sixty students participated. Seven focus groups were convened with on average eight or nine students in each.

The data were analysed using the Framework technique, involving the development of a thematic framework, indexing, charting, mapping and interpretation. Reflection was apparent throughout the data but frequently centred on early years objectives for learning about clinical environments and medical students roles within them. In the context of patient safety it was error, in multiple forms, including knowledge about the risks of error that appeared to trigger reflection.

There are different forms of reflection and ways of categorising reflection. An analysis of the reflection literature proposes that reflection can be described in two distinct forms; either as an iterative process or as a vertical hierarchy (Mann et al. 2009). The analysis of the data identified examples of reflection, which aligned to the vertical frameworks in different contexts of error.

For some students using the tools enabled them to consider beyond the initial ward experience to consider what they had learnt. Reflection on the process and method of the patient safety activities appeared to influence learning. The learning fitted with Mezirow’s model of transformative learning (Mezirow 2000). The data showed evidence of reflective and non-reflective thinking and more specifically examples of the two levels of reflective things described; reflection and critical reflection.

The quote below about medicine reconciliation demonstrates how error triggered reflection. This could be identified in the two distinct vertical levels described by Mezirow, reflection promoting an understanding of skills required for safe individual practice and critical reflection demonstrating learning about the nature of medication errors in general such as the role of documentation and communication:

Going through the drugs. Going through the Kardex (drug chart)… the kardex is quite a flexible document, it’s amended and changed, scored out and… how accurate was it to begin with? Is everything on there? There’s one or two drugs that the patients said to us they were on but when we looked in the Kardex they weren’t and… erm… there were other times the patient didn’t mention they were on it… so oh obviously they were and they didn’t understand why they were on it. Medication starts and stops and sometimes it doesn’t always explain why it starts and stops. But for one patient the dose had changed and it looked like it had been discontinued but it was at the end.. there was potential for error there”(7;11)

Data source 2

Sixty-one students participated and the questionnaires showed acceptable levels of internal consistency; Cronbach alpha for the reflection questionnaire was 0.71 and the patient safety questionnaire 0.90.

The associations are shown in Table 1.

Table 1 Correlation coefficients testing association between knowledge and attitudes towards patient safety and reflection and critical reflection
Full size table

Data source 3

Forty-eight students participated and the self-administered questionnaires showed acceptable levels of internal consistency, Cronbach alpha for reflection was 0.70 and for critical reflection was 0.78. The Cronbach alpha for the error data was 0.70. The Generalisability coefficient for the judgments about safe behaviour was 0.84. The associations are shown in Table 2.

Table 2 Correlation coefficients testing associations between reflection, critical reflection, error behaviour, safe behaviour and planned changes
Full size table

Representativeness of the samples from data sources 2 and 3

The confidence intervals for the population mean of the year group and the sample are shown in Table 3. This data is not available for data source 1. There were 174 students in the year group.. The histogram for study 2 is shown in Fig. 2 and for study 3 is shown in Fig. 3.

Table 3 Comparison of sample means with the cohort overall
Full size table
Fig. 2
figure 2

Histogram of representativeness of sample in study 2

Full size image
Fig. 3
figure 3

Histogram of representativeness of sample in study 3

Full size image

Discussion

Key findings

The concept of a realist investigation is based on the exploration of mechanisms within the following equation: context + mechanism = outcome (Pawson and Tilley 1997). In this setting the context is patient safety, the mechanism is reflective thinking and the outcomes are different levels in Kirkpatrick’s framework (Kirkpatrick 1967; Barr et al. 2000). This equation was explored in three cycles and the overarching theory developed.

Reflection was identified as a learning mechanism repeatedly throughout the three cycles in this single cohort. In cycle 1 error within patient safety emerged as an event that triggered reflection as a mechanism within patient safety. This is at the outcome level of reactions. In cycle 2 a pattern emerged that showed that reflection was associated with two specific areas of knowledge and attitudes. The pattern was present in the initial tests of association, but the partial correlation coefficients helped to explore the data in greater depth. This revealed a difference in the patient safety contexts of the patterns for the two levels of reflective thinking. Critical reflection was associated with future intentions and reflection was associated with knowledge of actions to take with regard to patient safety. In cycle 3 this pattern was seen again. The context was patient safety and the outcome was error in skills and behaviours. The mechanism of reflection demonstrated reflection as being associated with fewer high level cognitive errors and again the pattern of critical reflection and future intentions was demonstrated.

When these results are reviewed with the existing literature, this work addresses a gap. Previous work attempted to identify associations between measures of critical reflection and quality improvement and patient safety knowledge and behaviours in previous studies without success (Whittich 2011a, b). The negative result might be because mechanism under investigation was not appropriate to the context.

Recent research has started to explore the concept of reflexivity in the context of healthcare teams in the workplace and how reflection affects patient safety (Iedema 2011). Ethnographic work has revealed processes, which appear to relate to reflective thinking. Staff engaged in reasoning about risk within the workplace and adjusted their behaviour dependent on their individual evaluations (Dixon-Woods et al. 2009). The methodology of realist research used in this work to reveal patterns, has given insights into the mechanisms that influence patient safety learning through a series of cycles that single cycle work is unable to demonstrate.

Critical reflection appears to be associated with future intentions but not with behaviours. Reflection was associated positively with knowledge of actions to take for patient safety and had a negative association with knowledge-based error behaviours.

Reflection and critical reflection are not opposite behaviours in this setting, but adjacent stages within a vertical hierarchy (Mann et al. 2009). In Mezirow’s (2000) model this is the difference between thoughtful action with reflection to ex post facto reflection.

These results suggest that educators should focus on different types of reflection when considering different outcomes. When considering outcomes related to behaviours, a focus on reflection may be applicable, and when considering motivating learners concerning their current and future attitudes, critical reflection may be applicable. Both are required but serve different purposes from the data sources in this study. Therefore the mechanism of different levels of reflection seems to impact on patient safety learning. Figure 4 shows the progression of key results in relation to reflection within the studies.

Fig. 4
figure 4

Progression of key patterns in relation to reflection within the studies

Full size image

Limitations

This study was unfunded and has a number of limitations that relate to financial limitations. The principal researcher was a member of staff integral to the medical school’s skills programme. Therefore this individual might not be considered impartial. In the data source 1 the researcher was not involved in the patient safety aspect of the course and the analysis was completed with another member of staff whose role was also distant to the patient safety programme. The relationship between the principal researcher and the students had the potential to introduce bias. This was addressed in data source 2 through the use of validated questionnaires from the published literature and measures of reliability were included in the analysis to reduce the risks of bias. In data source 3 the assessments were completed within the medical schools’ formative assessment programme and a wide range of clinical assessors made the judgements about safe and error behaviours. Again reliability measures were included.

The study used appropriate data sources for the questions posed at each stage of the study to uncover the mechanisms that influenced the progress of student learning about patient safety. Data source 1 used focus groups to explore factors. Data source 2 used validated questionnaires to assess and measure levels of reflection and knowledge and attitudes to safety appropriate to the students experiences. Data source 3 then used direct observation of performance in a simulated workplace environment using a published error framework.

The cycles of work used self-reporting via two validated questionnaires and observed behaviours in a simulated setting. The reliance on self reporting of reflection was due to the design of the study overall where a method was required that would give a concurrent assessment of reflective thinking with the learning outcomes of knowledge and attitudes in cycle 2 and behaviours in simulation in cycle 3.

In sequential cycles of work in a longitudinal study with a single cohort there is a risk of one set of data collection influencing the next. In this study the data collection episodes occurred in alternate years. Mezirow’s (2000) levels are the link between the different cycles alongside demonstration of progression of learning at different outcome levels. Therefore, when the studies were designed it was decided that repeated use of Kember’s et al. (2000 ) instrument would be required to reveal mechanisms using a realist approach, and that the time interval might mitigate the risk of influence between data collection points.

The use of a simulated ward experience for data source 3 was chosen so that mechanisms could be identified in a setting that was similar to the workplace. The ward simulation exercise involved standardised scenarios that would help reveal the mechanisms. Previous work with postgraduates has demonstrated the realism of this setting (Ker et al. 2006) and so meant that this setting could identify mechanisms within complex processes. The assessment tool for this has been tested extensively and is now used within summative assessments (Ker 2013).

The sizes of the samples were small and the samples did not include the majority of the cohort, to address this, representativeness was assessed. The sample means for data sources two and three were compared with that of the cohort overall, although for both data sources the sample means were higher than the cohort overall. This may indicate that the group who participated were overall a higher achieving group but a comparison of means from the sample versus cohort overall suggest that the results might be applicable to the cohort overall.

Due to nature of separate consent being required for all three data sources, the data from the first study was anonymised and so it was not possible to ascertain if any of the students participated in all three studies. This is a significant limitation.

There is a risk in the multiple comparisons completed here of type I errors (Altman 1991). A Bonferroni correction (Bland and Altman 1995) was applied to address the risk of type 1 errors. In data source 2 when a Bonferroni correction of 0.004 is applied to address the multiple tests of association, two areas are significant. These associate reflection with two areas of knowledge and attitudes towards patient safety. These were reflection and knowledge of actions to take and critical reflection and future intentions regarding patient safety.

When a Bonferroni correction of 0.005 (Bland and Altman 1995) is applied to the results from data source 3, the significant finding is that of critical reflection with planned changes following feedback.

These limitations might seem to reduce the importance of the significant results. The limitations were recognised and processes used to try to minimise bias and to ensure that the results could be viewed as valid.

In the mainstream medical literature the focus is on outcomes that demonstrate changes for patients. In education these are difficult to measure for students one step removed from the registered professional-patient interface. A realist methodology may offer solution to this dilemma through the identification of learning mechanisms. This study raises questions about how reflection in its different guises is a mechanism in the context of patient safety, not just in learning but in behaviours. Professional development in medicine at both undergraduate and postgraduate levels focuses on the concept of the reflective practitioner. Critical reflection is the main focus of many learning activities but these results raise a question; should educators be focusing on developing Mezirow’s lower level of reflection in future health professionals? (Mezirow 2000) To conclude, these results suggest that critical reflection may only be associated with good future intentions but not with behaviour and as such should not be used as a lone educational strategy and more research is required in the area of reflection.