Abstract
Background
Measuring and monitoring safety (MMS) is critical to the success of safety improvement efforts in healthcare. However, a major challenge to improving safety is the lack of high quality information to support performance evaluation.
Aims
The aim of this study was to use Vincent et al.’s MMS framework to evaluate the methods used to MMS in Irish hospitals and make recommendations for improvement.
Methods
The first phase of this qualitative study used document analysis to review national guidance on MMS in Ireland. The second phase consisted of semi-structured interviews with key stakeholders on their understanding of MMS. The MMS framework was used to classify the methods identified.
Results
Six documents were included for analysis, and 24 semi-structured interviews were conducted with key stakeholders working in the Irish healthcare system. A total of 162 methods of MMS were identified, with one method of MMS addressing two dimensions. Of these MMS methods, 30 (18.4%) were concerned with past harm, 40 (24.5%) were concerned with the reliability of safety critical processes, 16 (9.8%) were concerned with sensitivity to operations, 28 (17.2%) were concerned with anticipation and preparedness, and 49 (30%) were concerned with integration and learning.
Conclusions
There are a wide range of methods of MMS in Irish hospitals. It is suggested that there is a need to identify those methods of MMS that are particularly useful in reducing harm and supporting action and improvement and do not place a large burden on healthcare staff to either use or interpret.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
A commitment to safe healthcare is a policy goal of governments across the world. However, progress on delivering on this aspiration has been modest, with patients still suffering avoidable harm [1] and rates of harm remaining unchanged over time [2]. The impact that valid and reliable safety data can have on improvement is clear from many domains of healthcare [3]. Although safety data is complex and multi-faceted, it is vitally important to reducing patient harm [4]. This data is required to support the meaningful comparisons between the safety performance of different healthcare organisations and the assessment of the impact of safety interventions [1].
A recent review found that there has been relatively little research carried out on measuring and monitoring safety (MMS) in the healthcare system of the Republic of Ireland [5]. However, there is a recognition that MMS is central to patient safety improvement. This is evident in the Irish Health Service Executive (HSE) patient safety strategy which includes a commitment to ‘using information to improve safety’ (p.8) [6]. Recognising the complexity and multifaceted nature of the MMS in healthcare, Vincent et al. developed the MMS framework [4, 7]. The dimensions of the MMS framework are as follows:
-
1.
Harm: has patient care been safe in the past? (e.g. national audits).
-
2.
Reliability of safety critical processes: are our clinical systems and processes reliable? (e.g. monitoring of vital signs).
-
3.
Sensitivity to operations: is care safe today? (e.g. safety walk-arounds).
-
4.
Anticipation and preparedness: will care be safe in the future? (e.g. safety culture assessment).
-
5.
Integration and learning: are we responding and improving? (e.g. aggregated analysis of incidents) [4, 7].
The framework has been used previously to categorise studies in systematic or scoping reviews of MMS in primary care [8], prehospital care [9], and secondary care in Ireland [5] and Saudi Arabia [10]. The study reported in this paper uses Vincent et al.’s [4, 7] MMS framework to classify the methods of MMS used within secondary care in Ireland. The aims of the study reported in this paper are to (1) examine how patient safety is measured and monitored in Irish hospitals; (2) map the methods of MMS in these hospitals onto the five dimensions of Vincent et al.’s [4, 7] MMS framework; and (3) reflect on the approaches used to MMS in Irish hospitals.
Methods
Research design
A qualitative descriptive approach was adopted for this study to: (1) support a document analysis of the guidance on MMS used in Irish hospitals; and (2) use semi-structured interviews to explore stakeholders’ perceptions about how patient safety is measured and monitored in Irish hospitals. This approach was based upon a study of methods of MMS use in Saudi Arabian hospitals [11].
Phase one: document analysis
Methods of MMS in Irish hospitals that are described in national healthcare governance documents were identified and classified. Document analysis is a systematic method to review or evaluate documents [12]. The ‘ready materials, extract data, analyse data and distil (READ)’ approach [13] was utilised.
Inclusion criteria
The inclusion criteria required that documents were: national-level documents; explicitly discussed or described how patient safety is measured and monitored in Irish hospitals; produced by a national government agency/or an organisation affiliated with a national government agency; and written in English. No publication date or period was specified. Finally, in cases where a document had multiple versions, only the latest version of the document was included.
Exclusion criteria
Documents were excluded if: they did not describe how patient safety is measured and monitored in Irish hospitals; were not produced by an Irish government agency or an organisation affiliated with a national government agency; were focused on tracking performance progress (e.g. annual reports); were focused on the safety of one process only (e.g. medication safety); were focused on a particular method of measurement that is relevant to specific clinical practises (e.g. clinical audit); or were not written in English.
Search process
The search for relevant documents was completed in January 2022 and consisted of four steps intended to support the retrieval of government reports and policy documents:
-
1.
An advanced google search was completed.
-
2.
A search of the following electronic databases was conducted: Medline, CINAHL, OAIster, WHO IRIS, Lenus, and Google Scholar using various combinations of terms ‘measuring safety’, ‘monitoring safety’, and ‘measurement of safety’ (additional File 1 presents the search strategy used in complete detail).
-
3.
Searches were conducted across the Irish Health Service Executive (HSE), Irish Department of Health, and Irish Health Information and Quality Authority (HIQA) websites using both their relevant search boxes and manual search.
-
4.
Potential further related documents were identified through hand searching the reference lists of documents that met the inclusion criteria.
Document selection
The initial screening was completed by YK using the inclusion criteria to assess the potential for inclusion from the titles, abstracts, and/or executive summaries. Documents that appeared relevant were then downloaded for full-text review. A full-text review was completed by YK and POC to ensure the document met all items in the inclusion criteria. Decisions regarding the inclusion or exclusion of documents were agreed by consensus. All decisions were recorded in an Excel file.
Document analysis
YK and POC independently searched through each document and extracted all described methods of MMS. Only minor differences were found between the information extracted by the two reviewers. These differences were concerned with whether a particular method was one measure or could be split into two measures. Once the final list of MMS had been identified, YK, POC, and SL reviewed each measure and reached a decision by consensus as to which dimension of Vincent et al.’s [4, 7] MMS framework it addressed.
Phase two: semi-structured interviews with key stakeholders
The aim of the second phase of the study was to explore what key stakeholders know about how safety is measured and monitored in the Irish healthcare system.
Ethical review
The study was approved by the Clinical Research Ethics Committee, Galway University Hospitals (Ref: C.A.2604). All participants provided signed written informed consent.
Sampling and recruitment of participants
Participants for the interviews were drawn from three different stakeholder groups: (1) policy makers; (2) medical doctors; and (3) nurses. Recruitment of participants was through a combination of purposive and snowball sampling.
Development of interview guide
The semi-structured interview guide that was used is shown in Table 1. The design of the interview guide was derived from the five dimensions of Vincent et al.’s [4, 7] MMS framework. The interview questions were prepared in accordance with best practises for the formulation of interview questions [14, 15]. The interviews were conducted using an interview schedule developed to obtain information about perceptions of MMS in Saudi Arabian hospitals [11].
Procedure
All interviews were carried out from January 2022 to June 2022. After receiving written informed consent, YK and DM conducted the interviews via video conference call. The audios of the calls were recorded.
Interview analysis
The purpose of the interview analysis was to identify the methods of MMS which the participants knew were being used in Irish hospitals. These methods of MMS were categorised using Vincent et al.’s [4, 7] MMS framework. The methods of MMS described by the interviewees were extracted from the transcripts by YK and DM and then reviewed by POC. Decisions on categorisation were made by consensus.
Results
Phase one: document analysis
A total of six documents were found to meet the inclusion criteria. All six documents had been published since 2008. The search process for identifying documents that met the inclusion criteria is shown in Fig. 1. A summary overview of these documents is provided in Table 2.
A total of 162 methods of MMS were identified across the six documents (see Table 2 and Additional File 2 for a list of these methods and how they were classified). Of these MMS methods, 30 (18.4%) were concerned with past harm, 40 (24.5%) were concerned with the reliability of safety critical processes, 16 (9.8%) were concerned with sensitivity to operations, 28 (17.2%) were concerned with anticipation and preparedness, and 49 (30%) were concerned with integration and learning. One method of MMS addressed two of the safety dimensions (past harm and integration and learning); therefore, the percentages are calculated out of 163.
Phase two: semi-structured interviews with key stakeholders
The mean duration of the interviews was 25 min (SD = 11 min 58 s). The 24 participants included 18 frontline healthcare workers (nine doctors and nine nurses) and six healthcare policy makers. Of the 24 participants, 14 were women and 10 were men. The participants reported a mean of 13 years of professional experience (range = 3–31 years). Twenty-one (87.5%) of the participants worked in teaching hospitals, and three (12.5%) in national health regulation organisations.
The MMS methods reported by interviewees are shown in Table 3. Illustrative quotes pertaining to the most commonly reported measures in each safety domain are provided in Table 4. The interviewees described a total of 76 methods of MMS. Of these methods of MMS, 14 (18.4%) were concerned with past harm, 25 (33%) were concerned with the reliability of safety critical processes, 8 (10.5%) were concerned with sensitivity to operations, 15 (19.7%) were concerned with anticipation and preparedness, and 14 (18.4%) were concerned with integration and learning.
The most frequently reported MMS method for past harm was incident reporting (mentioned by 19; 79.2% of the interviewees). In addition, incident reports were by far the most commonly reported method of MMS across all dimensions. Clinical audit was the most frequently mentioned MMS method for the reliability of safety critical process dimension (mentioned by 18; 75% of the interviewees). Observation and conversations with clinical teams were the most frequently described MMS method for the sensitivity to operation dimension (mentioned by 8; 33.3% of the interviewees), followed by safety walk-arounds (mentioned by 6; 25% of the interviewees) and safety huddles (which were also mentioned by 6; 25% of the interviewees). Analysis of incidents and feedback leading to the implementation of safety lessons (mentioned by 12; 50% of interviewees) was the most often reported MMS method in the dimension of integration and learning.
Discussion
A fundamental challenge to improving patient safety in healthcare is a dearth of high-quality information that allows organisations and individual practitioners to analyse their performance, define priorities, and identify areas of deficiency and risk. In this study, we examined how patient safety is measured and monitored in Irish hospitals, mapped these methods onto Vincent et al.’s [4, 7] MMS framework, and reflected on the meaning of these findings for MMS in Irish hospitals.
Considering both the findings from the document analysis and interviews, it can be seen that a wide variety of methods are used to MMS in Irish hospitals. However, although there were measures from across all five of the MMS framework dimensions, there was some variability in the number of methods within the dimensions. Measures of the reliability of safety critical processes were the most commonly identified methods of MMS in the document analysis and interviews. This may reflect the amount of routine safety data that are collected in Irish hospitals. The dimension with the smallest number of measures was sensitivity to operations. A possible explanation for this finding is that these measures tend to be qualitative (e.g. talking with patients and staff, observing staff) [16]. Such ‘soft intelligence’ is generally more difficult to collect, and analyse, than is the case for quantitative data [17]. However, this qualitative data can provide valuable insights into issues that may not be possible to gain from quantitative methods of MMS. Automated language analysis methods are beginning to be used to analyse qualitative patient safety data [18]. Therefore, there is the potential for the analysis of qualitative data to become much easier, and faster, than in the past.
Of all the measures identified by the interviewees across the five MMS safety dimensions, incident reports were the most common method of MMS—identified by almost four-fifths of the interviewees. However, the international literature suggests that healthcare organisations may overly rely on the analysis of past events and past harms as a source of safety performance information [19,20,21]. Thus, whilst it is positive that frontline healthcare workers are aware of the importance of collecting information on adverse events, it is important that the limitations of this particular measure are recognised. It is well known that reporting systems underestimate the prevalence of patient safety incidents [22] and overestimate the severity of harm [23]. Therefore, it is important to avoid an over-reliance on incident reports and past harm more generally, as the primary source of safety data.
The second most common method of MMS identified by the interviewees was clinical audit—identified by three quarters of the interviewees. Clinical audits are widely used in many healthcare systems to assess clinical performance against pre-set standards and use the data to improve practise [24]. Nevertheless, despite their widespread use, audits’ effectiveness as a practise-improvement strategy is often presumed, rather than supported by robust evidence [24]. A review of the impact of clinical audit on healthcare workers practise and patient outcomes concluded that audits resulted in a little change, only 4% of the studies which used clinical audit resulted in an increase in the desired practise [25]. Research found that audits were more likely to be successful when there was low baseline performance, feedback was offered several times by a colleague or supervisor in both verbal and written formats, and defined objectives and an action plan were included [24]. Moreover, clinical audit places a considerable burden on staff to complete [26]. It is certainly not suggested that the health service abandons the practise of clinical audit. However, there is a need to consider how to reduce the resources, and burden, of MMS [3].
Recommendations
Our study has shown that there are methods of MMS from across all five dimensions of the framework. However, despite collecting large volume of safety data about hospital care, it still remains challenging to determine the safety of the delivery of care [4, 7]. It has been suggested that healthcare stakeholders could get the information they need with a quarter of what is currently being spent on MMS [27]. The WHO has also identified the burden of collecting and analysing data as a barrier to MMS [3]. Therefore, to improve patient safety in the Irish healthcare system, we recommend a number of issues that should be addressed.
-
Reliability of safety data. The reliability of most safety data is unknown, and in some cases the reliability may actually be known to be problematic (e.g. hand hygiene compliance). If measures are poorly designed, this can lead to ‘gaming’, where targets are achieved but the intended changes in practise are not [28]. Therefore, consideration needs to be given to identifying which methods of MMS result in reliable data.
-
Fragmentation of data. There are a huge range of methods of MMS focused at different levels of a healthcare organisation (e.g. units, hospitals), by different organisations (e.g. HIQA, Department of Health). This fragmentation of data creates challenges for healthcare professionals and managers in identifying where improvement efforts should be made, and whether these efforts are effective [1]. It is recommended that there is a consolidation of efforts across these agencies to avoid repetition and overlap of efforts.
-
Quantity of safety data. A total of 162 methods of MMS were identified from the document analysis, and 76 methods of MMS identified from the interviews. This quantity of data can be overwhelming for healthcare workers and managers. There is a need for safety data to be readily interpreted so that safety issues can be identified at unit, hospital, and national levels. Measures that are too burdensome or lack credibility may alienate clinicians and lead to confusion about the impact of interventions [29]. It is suggested that the perspectives of all stakeholders in healthcare should be taken to identify key measures, from across the five MMS domains, that are particularly useful in supporting action and improvement, and do not place a large burden on healthcare staff to use.
-
Lack of ownership of the data. Much of the data is focused on measures generated externally to a clinical team, so the teams may not perceive the data as being related to their performance [30]. Consideration should be given to how to engage front-line clinical staff in MMS so that they feel some ownership and are empowered to act upon the data, and involving them, and other stakeholders, in identifying meaningful methods of MMS.
Limitations
There are a number of limitations of this research that should be acknowledged. The main limitation is that this paper only focused on MMS in the Irish healthcare system. However, the findings are similar to those derived from a study that utilised the same methodology to consider the MMS in the Saudi Arabian healthcare system [11]. Therefore, there would appear to be generalisability of the findings to other healthcare systems. As is the case with other qualitative approaches, our study could be critiqued due to the subjectivity of this type of research. However, these issues were mitigated through the rigorous approach we took to the data collection and analysis. The focus of the interviews was on identifying the methods of MMS, and did not include an analysis of the quality of the data collected using these approaches. Finally, we only considered national-level publications in the document analysis, which may have led to the exclusion of useful hospital-level documents. However, the difficulty in systematically accessing hospital-level documents was a barrier to their inclusion.
Conclusion
There are a wide range of methods of MMS in Irish hospitals. However, having larger numbers of methods of MMS does not necessarily correspond to a robust safety surveillance system. It is suggested that the input of all stakeholders in healthcare are gathered to identify particularly key measures, across the five MMS domains, in order to identify those methods of MMS that are particularly useful in reducing harm, supporting action and improvement, and do not place a large burden on healthcare staff to use or interpret.
Data availability
All data arising from the document analysis is presented either in the article or within the included supplemental material. Interview data is summarised within the article. The ethical approval for this study does not include provision for sharing interview transcripts.
References
Dixon-Woods M, Baker R, Charles K et al (2014) Culture and behaviour in the English National Health Service: overview of lessons from a large multimethod study. BMJ Qual Saf 23:106–115. https://doi.org/10.1136/bmjqs-2013-001947
Shojania KG, Thomas EJ (2013) Trends in adverse events over time: why are we not improving? BMJ Qual Saf 22:273–277. https://doi.org/10.1136/bmjqs-2013-001935
World Health Organization (2021) Towards eliminating avoidable harm in health care. Global patient safety action plan 2021:2030
Vincent C, Burnett S, Carthey J (2013) The measurement and monitoring of safety: drawing together academic evidence and practical experience to produce a framework for safety measurement and monitoring. The Health Foundation, London
O’Connor P, O’Malley R, Kaud Y et al (2022) A scoping review of patient safety research carried out in the Republic of Ireland. Ir J Med Sci 5:1–9. https://doi.org/10.1007/s11845-022-02930-1
Health Services Executive (2019) Patient safety strategy 2019– 2024. Dublin
Vincent C, Burnett S, Carthey J (2014) Safety measurement and monitoring in healthcare: a framework to guide clinical teams and healthcare organisations in maintaining safety. BMJ Qual Saf 23:670–677. https://doi.org/10.1136/bmjqs-2013-002757
O’Connor P, Madden C, O’Dowd E et al (2021) A meta-review of methods of measuring and monitoring safety in primary care. Int J Qual Health Care 33:1–8. https://doi.org/10.1093/intqhc/mzab117
O’Connor P, O’Malley R, Oglesby A-M et al (2021) Measurement and monitoring patient safety in prehospital care: a systematic review. Int J Qual Health Care 33:1–8. https://doi.org/10.1093/intqhc/mzab013
Kaud Y, O’Connor P, O’Malley R et al (2022) A scoping review of patient safety research carried out in Saudi Arabian hospitals. IJQHC Communications 2:1–9. https://doi.org/10.1093/ijcoms/lyac014
Kaud Y, Lydon S, O’Connor P (2021) Measuring and monitoring patient safety in hospitals in Saudi Arabia. BMC Health Serv Res 21:1–12. https://doi.org/10.1186/s12913-021-07228-z
Bowen GA (2009) Document analysis as a qualitative research method. Qual Res J. https://doi.org/10.3316/QRJ0902027
Dalglish SL, Khalid H, McMahon SA (2020) Document analysis in health policy research: the READ approach. Health Policy Plan 35:1424–1431. https://doi.org/10.1093/heapol/czaa064
Bowling A (2014) Research methods in health: investigating health and health services. McGraw-Hill Education, United Kingdom
McGrath C, Palmgren PJ, Liljedahl M (2019) Twelve tips for conducting qualitative research interviews. Med Teach 41:1002–1006. https://doi.org/10.1080/0142159X.2018.1497149
Martin GP, McKee L, Dixon-Woods M (2015) Beyond metrics? Utilizing ‘soft intelligence’for healthcare quality and safety. Soc Sci Med 142:19–26. https://doi.org/10.1016/j.socscimed.2015.07.027
Martin GP, Aveling E-L, Campbell A et al (2018) Making soft intelligence hard: a multi-site qualitative study of challenges relating to voice about safety concerns. BMJ Qual Saf 27:710–717. https://doi.org/10.1136/bmjqs-2017-007579
Gillespie A, Reader TW (2022) Online patient feedback as a safety valve: an automated language analysis of unnoticed and unresolved safety incidents. Risk Anal 1–15. https://doi.org/10.1111/risa.14002
Lilford R, Pronovost P (2010) Using hospital mortality rates to judge hospital performance: a bad idea that just won’t go away. BMJ 20:340. https://doi.org/10.1136/bmj.c2016
Mohammed MA, Manktelow BN, Hofer TP (2016) Comparison of four methods for deriving hospital standardised mortality ratios from a single hierarchical logistic regression model. Stat Methods Med Res 25:706–715. https://doi.org/10.1177/0962280212465165
Olsen S, Neale G, Schwab K et al (2007) Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place. BMJ Qual Saf 16:40–44. https://doi.org/10.1136/qshc.2005.017616
Vincent C (2007) Incident reporting and patient safety. BMJ 334:51. https://doi.org/10.1136/bmj.39071.441609.80
Panesar SS, Desilva D, Carson-Stevens A et al (2016) How safe is primary care? A systematic review. BMJ Qual Saf 25:544–553. https://doi.org/10.1136/bmjqs-2015-004178
Paton JY, Ranmal R, Dudley J et al (2015) Clinical audit: still an important tool for improving healthcare. Arch Dis Child Educ Pract Ed 100:83–88. https://doi.org/10.1136/archdischild-2013-305194
Ivers N, Jamtvedt G, Flottorp S et al (2012) Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.CD000259.pub3
Johnston G, Crombie I, Alder E et al (2000) Reviewing audit: barriers and facilitating factors for effective clinical audit. BMJ Qual Saf 9:23–36. https://doi.org/10.1136/qhc.9.1.23
Berwick DM (2016) Era 3 for medicine and health care. JAMA 315:1329–1330
Bevan G, Hood C (2006) What’s measured is what matters: targets and gaming in the English public health care system. Public Adm 84:517–538. https://doi.org/10.1111/j.1467-9299.2006.00600.x
Dixon-Woods M, McNicol S, Martin G (2012) Ten challenges in improving quality in healthcare: lessons from the Health Foundation’s programme evaluations and relevant literature. BMJ Qual Saf 21:876–884. https://doi.org/10.1136/bmjqs-2011-000760
Woodcock T, Liberati EG, Dixon-Woods M (2021) A mixed-methods study of challenges experienced by clinical teams in measuring improvement. BMJ Qual Saf 30:106–115. https://doi.org/10.1136/bmjqs-2018-009048
Funding
Open Access funding provided by the IReL Consortium. YK is completing a PhD at the University of Galway funded by the Saudi Electronic University.
Author information
Authors and Affiliations
Contributions
All authors were involved in the design and planning of the study. YK and SL developed the search strategy. YK conducted the initial searches. YK, POC, and SL were responsible for completing the data extraction. YK and DM completed the interviews and transcribing. YK, POC, and DM conducted the analysis of the interview data. YK drafted the initial manuscript with POC, SL, and DM reviewing and assisting with redrafting. All authors read and approved the final manuscript prior to submission.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The study was approved by the Clinical Research Ethics Committee, Galway University Hospitals (Ref: C.A.2604), and written informed consent was obtained from all participants.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kaud, Y., McKeon, D., Lydon, S. et al. Measuring and monitoring patient safety in hospitals in the Republic of Ireland. Ir J Med Sci 192, 2581–2593 (2023). https://doi.org/10.1007/s11845-023-03336-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11845-023-03336-3