Abstract
Quality indicators are vital for monitoring the transformation of institution-based mental health services towards the provision of person-centered mental healthcare. While several mental healthcare quality indicators have been identified as relevant and valid, their actual usability and utility for routine monitoring healthcare quality over time is significantly determined by the availability and trustworthiness of the underlying data. In this feasibility study, quality indicators that have been systematically identified for use in the Danube region countries of Bulgaria, the Czech Republic, Hungary, and Serbia were measured on the basis of existing mental healthcare data in the four countries. Data were collected retrospectively by means of the best available, most standardized, trustworthy, and up-to-date data in each country. Out of 21 proposed quality indicators, 18 could be measured in Hungary, 17 could be measured in Bulgaria and in the Czech Republic, and 8 could be measured in Serbia. The results demonstrate that a majority of quality indicators can be measured in most of the countries by means of already existing data, thereby demonstrating the feasibility of quality measurement and regular quality monitoring. However, data availability and usability are scattered across countries and care sectors, which leads to variations in the quality of the quality indicators themselves. Making the planning and outputs of national mental healthcare reforms more transparent and evidence-based requires (trans-)national standardization of healthcare quality data, their routine availability and standardized assessment, and the regular reporting of quality indicators.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Countries of the Danube region like Bulgaria, the Czech Republic, Hungary, and Serbia are aiming to transform their institution-based mental healthcare systems to more community-based systems, thereby improving and assuring quality of care [3, 4, 9, 14, 29]. In this context, quality indicators need to be applied to assess the status-quo and to monitor the development and success of the transformation and implementation processes. Healthcare system quality indicators are generally defined as “quantitative measures that can be used to monitor and evaluate the quality of important governance, management, clinical, and support functions that affect patient outcomes” (JCAHO 1989, quoted in [11], p. 524). They reflect aspects of the structures, processes, and outcomes on different healthcare hierarchy levels, including the care systems (macro-) level, the healthcare institutions (meso-) level, and the individual healthcare professionals and patients (micro-) level. Indicators can be used to benchmark quality of care, to assess the current state (as-is situation) and to establish the targeted future state of mental healthcare, and to monitor change processes [15, 17]. Topics such as deinstitutionalization, healthcare coverage and accessibility, workforce development, or anti-stigma activities are important quality domains in the transformation of mental healthcare systems in the four countries [3, 4, 9, 13, 14, 29, 30]. Accordingly, a global quality target as defined by the World Health Organization (WHO) Mental Health Action Plan is that by the year 2020, 80% of all countries should be routinely collecting and reporting a set of core mental health indicators [25]. This requires a universal and agreed set of quality indicators to be developed, based on methodologically sound and thematically appropriate data sources [8, 23]. WHO’s Mental Health Atlas is the most global attempt to capture a standard set of mental health care indicators across countries [27]. In Europe, quality indicators have already been developed and at least been partially implemented in some countries, such as Germany [6, 7, 10], Scotland [20], Switzerland [1], and the United Kingdom [16].
Against this background, a 2-year project lasting from April 2017 to March 2019, funded by the German Federal Ministry of Education and Research (BMBF), was conducted under the title “Development and Implementation of Quality Indicators for Mental Healthcare in the Danube Region” (DAQUMECA). The aim was to develop quality indicators for the mental healthcare systems of Bulgaria, the Czech Republic, Hungary, and Serbia, and to test these quality indicators regarding their data availability. It was focused on developing quality indicators for measuring the macro- and meso-level of healthcare. In a first project phase, international quality indicators were identified via a systematic literature search, subsequently narrowed down with reference to defined inclusion and exclusion criteria, and then rated for their validity, reliability, and feasibility in a two-stage Delphi study by a multi-disciplinary expert panel from Bulgaria, the Czech Republic, Hungary, and Serbia [15].
The ratings of the Delphi panelists (n = 18) in the first project phase provided a first overall impression of data availability for the quality indicators through an expert-based estimate [15]. However, actual data availability needs to be more objectively assessed in a feasibility study, where data sources are first to be identified for their availability, and then, the data-collection effort can be evaluated additionally. According to Lehmann et al. [15], the Delphi experts mostly agreed on the relevance and validity of the quality indicators. For only six indicators (QI 14, 17, 18, 19, 20, 22, Table 1), there was some uncertainty about the relevance or validity of the indicators with a tendency towards agreement. Overall, there was no disagreement on the relevance or validity of any of the quality indicators [15]. Accordingly, all identified quality indicators shown in Table 1 were tested in a feasibility study, with the exception of QI 22 relating to assisted housing, since the rating of data availability was extremely low for this QI (less than 10% of Delphi panelists expected electronic data to be available). The project consortium members, therefore, decided to not include QI 22 in the feasibility study, since they also agreed on the non-availability of data sources for this QI. Accordingly, 21 QI were evaluated for their feasibility by means of the existing databases in Bulgaria, the Czech Republic, Hungary, and Serbia.
An overview of the developed quality indicators from the first project phase is given in Table 1 [15].
Methods
The feasibility study of the 21 quality indicators focused on current data availability and data-collection effort in each country. Thus, the goal was to target data availability by sources within each country; it was not an aim to compare quality indicator results among countries, because this would have required trans-national standardized data collection.
In mental healthcare, different data sources with different levels of data standardization are available. Administrative data, which are typically used for insurance or reimbursement purposes, such as data on diagnostic and therapeutic procedures, are, in all countries, generally the most standardized and are available electronically. Routinely collected clinical data, such as documented data on adverse effects of therapeutic procedures, can be but are not necessarily standardized. They can be available electronically or in paper files. Finally, additional data, such as patient-reported data, may be collected electronically and can be used to measure quality indicators [22]. In this study, the term “routine data” is used to describe any officially and regularly documented data in the mental healthcare systems, including administrative and clinical ‘routine’ data.
To collect data for the present feasibility study the following potential data sources were defined:
-
Overarching data sets from different care settings and sectors (= cross-sectoral routine data), e.g., data from (national) statistical offices, ministries of health, provider or patient registries, legislative texts, and policy/action plan documents
-
Separate data sets from different care sectors (e.g., inpatient vs. outpatient vs. community care) (= sector-specific routine data), e.g., health insurance data
-
Data generated from retrospective or prospective surveys (= data from additional data collections, either from continuous data collections or intermittent data collections), e.g., data from epidemiologic studies
Between March and August 2018, data were collected by all country partners from Bulgaria, the Czech Republic, Hungary, and Serbia in a retrospective manner. The aim was to collect the data from the best available/most trustworthy databases in each country with the latest available data. Data were summarized into data tables including the following information per quality indicator:
-
Data availability (Yes/No)
-
Data sources (please specify)
-
Calculation of numerator/denominator (if applicable to QI)
-
Result of QI
-
Calculation period
-
Comments.
An interim discussion about the status of the data collection and possible obstacles took place in June 2018. After each country collected their data, the results were summarized in one table to identify similarities and differences with regard to the trustworthiness of the data sources and collected data. These topics were discussed in a structured group meeting with all project partners in March 2019. The aims of this meeting were to identify obstacles in the data-collection processes and to discuss the trustworthiness of the collected data.
Results
Table 2 shows all quality indicators and whether data were available in each country. 81–86% of the quality indicators could be measured based on the existing data sources in Bulgaria, the Czech Republic, and Hungary. Only 38% of all QI could be measured in Serbia.
Eight quality indicators (QI 1–6, QI 16, QI 19) could be measured in all four countries. Seven quality indicators (QI 8–11, QI 17, QI 18, QI 20) could be measured in three countries. One quality indicator (QI 14) could only be measured in the Czech Republic and Hungary. One indicator (QI 15) could only be measured in Hungary. For the two indicators on the utilization of mental health services in children and adolescents (QI 12, QI 13), no data were available in any country.
Overall, the main data sources available to measure each quality indicator are summarized in Table 3. In some QI, multiple data sources were used to measure the numerators and denominators of the QI. For example, QI 7–QI 11 measure the number of persons with a certain diagnosed mental disorder who receive mental health treatment (numerator) in relation to all persons with the same but untreated either diagnosed or still undiagnosed (but diagnosable) mental disorder within a defined time period (denominator). The denominator was measured on the basis of (epidemiological) survey data, while the numerator was measured based on health care utilization or insurance data.
Five quality indicators could be measured with one data source (QI 1–4, QI 15). Fourteen quality indicators were measured with data from more than one data source (QI 5–11, QI 14, QI 16–21).
Most quality indicators (n = 13) were measured with routinely available data that are collected in central offices including national statistical offices, ministries of health and provider, and patient registries of persons with mental disorders or with information that can be extracted from legislative texts and policy/action plan documents (cf. Table 3). Twelve quality indicators were measured by means of health insurance data, including the quality indicators on the utilization of mental healthcare (QI 7–12). For the quality indicators on the utilization of mental healthcare also survey data that are collected in some of the participating countries either recurring or non-recurring were used (QI 7–12).
The data-collection effort was evaluated in the project meetings. The following observations were made in the participating countries:
-
Routine data to measure quality indicators are not available for all quality indicators, leading to increased data-collection efforts.
-
Collection from different data sources overall increased the data-collection effort.
-
Time spent on data collection varied between countries and the type of data needed. Time ranged between immediate data availability (e.g., from online published national health statistics) to a waiting time of up to 6 months before data were available (e.g., data queries for health insurance data).
-
Possible prospective collection of additional data may be necessary to measure the complete quality indicator set. However, this was not feasible within this study.
-
There is a shortage of epidemiologic data (including the incidence and prevalence of mental disorders) due to a lack of nationwide surveys in Hungary and Serbia. In Bulgaria, epidemiologic data are not available in open access.
-
For some quality indicators, data were available, but the quality indicators’ definitions should be aligned with available data to improve the validity of the measurement (i.e., the indicator measures what it intends to measure).
Both, the existence of mental health legislation (QI 1) and the existence of a mental health action plan (QI 2) form a framework, in which the requirements and strategies for high-quality person-centered mental healthcare can be further defined. Therefore, these two quality indicators are regarded as essential, and in the following, the results of each indicator are shown per country (Tables 4, 5).
For both indicators, information was extracted from legislative texts and mental health action plans.
Discussion
This pilot feasibility study provides a first overview of the feasibility of systematically developed mental health quality indicators to be used for regular quality monitoring in countries of the Danube region. Data availability is an important prerequisite in using indicators for measuring healthcare quality. The developed quality indicators mainly focus on structural—as opposed to process or outcome-based—aspects of the mental healthcare system (e.g., availability of mental health service structures, expenditures, and workforce). A survey across 18 member states of the Organization for Economic Cooperation and Development (OECD) showed that availability of structural data is generally very good in mental healthcare [2]. While two countries (Bulgaria and Serbia) in our feasibility study are not OECD countries, all of the participating countries except for Serbia, which is also the only country that is not a member of the European Union, have a high data availability for the quality indicators in this study that focus on structural aspects. These indicators can be used to identify the quality of mental healthcare systems at the macro level, including the availability of legislation, mental health policies, and action plans, the level of mental health budget and expenditures on the health system and per capita, the availability of human resources for mental health, the utilization of services, and also anti-stigma activities. At the meso-level of mental health services, these quality indicators can also be used to measure important domains of service availability such as continuity of care, the number of employed workers, and the number of beds and places.
The different data sources that were used to measure the relevant data for the various quality indicators exhibit mainly the problem of fragmented information systems in parallel to fragmented healthcare systems in all four countries. Fragmented data sources are also internationally a still much more prevalent problem in mental healthcare compared to healthcare for somatic disorders [2, 12]. Data for mental healthcare processes and outcomes are even less readily available than structural data due to lacking data infrastructure [2, 12, 21]. The process indicators “QI 15 Follow-up of visits after mental health-related hospitalization” and “QI 21 Involuntary inpatient admissions”, for example, are less measurable with currently existing data (Table 1).
The best available data sources include national surveys and registries as well as administrative hospital databases [2]. In general, national epidemiological surveys were available only in the Czech Republic and Bulgaria, demonstrating the need for such surveys in Hungary and Serbia. The results of the Bulgarian epidemiologic survey, however, are not accessible for external parties, other than the consortium that conducted the survey, so that they were not used in our study. All four countries were able to obtain data from national statistics offices that collect population data, which can be used for quality indicator measurement (Table 1). Administrative hospital databases were also available in each country. However, these are generally in-house data and not readily available for external parties.
The existence of mental health legislations (QI 1, Table 4) is an important prerequisite in prioritizing mental healthcare on political agendas and in the further formulation of mental health aims [24]. A mental health action plan (QI 2, Table 5) in turn specifies the strategies to reach the aims of legislation, to set priorities and establish resources and timelines. Mental health action plans, therefore, form important tools in the transformation of mental healthcare systems towards high-quality integrated care services. The results on the availability of quality indicators demonstrate that a mental health action plan is only available in two of the four countries (Czech Republic and Serbia, Table 4 and 5). The action plans in the Czech Republic and in Serbia comprise strategies and aims on the promotion of mental health, prevention, treatment, and rehabilitation of mental disorders and activities against stigma and discrimination of persons with mental disorders (Table 5) [5, 19, 14, 18, 28].
However, only the Czech mental health action plan has dedicated funding for its implementation available, and the budget has been used for this intended purpose as shown in the QI results in Table 5. The Serbian mental health action plan lacks funding and, therefore, has so far only an informative character.
Besides policy limitations and limitations due to fragmented mental healthcare provision, technical limitations form an important barrier in the overarching measurement of quality indicators in mental healthcare. Information technologies are lagging behind in mental healthcare in comparison to other fields of medicine, including a lack of electronic patient records and of routine collection and reporting of quality indicators [12].
While it is crucial to monitor mental healthcare quality, data collection needs to be meaningful and reliable, because inefficient and overwhelming demands for data collection can impede motivation and hamper the capacities for providing high-quality mental healthcare [26]. Ideally, quality indicators are linked to electronic health records that collect mental health data and are calculated based on systematic data collections [7, 12].
Therefore, as a follow-up to this quality indicator study, it is planned to develop and implement in the consortium partner countries a trans-national digital mental health platform including the quality indicators based on standardized data and also other functionalities to support the further reform process in these and other European countries. The platform should serve as an important information and navigation tool in increasing transparency of mental healthcare and in monitoring the transformation from a fragmented, hospital-based mental healthcare system towards integrated, needs-based, and people-centered mental healthcare.
The main responsibility for data collection on mental healthcare systems performance lies with each country. The comparison of universal quality indicators on an international level can be used within a benchmarking of mental healthcare performance, can support an improved international networking, and can initiate a learning from each other. For an international benchmarking, a core set of quality indicators should be used measuring important aspects of mental health system governance (legislation and policies), resources (financing and workforce), and availability (number and geographical distribution of facilities), all of which are included in our proposed quality indicator set. The WHO Mental health ATLAS [27] collects similar data that can be used as reference values for the quality indicators included in this analysis. However, the WHO ATLAS results show that data collections for quality assurance purposes still need to be improved, as the four countries incompletely report on the WHO ATLAS indicators.
Overall, there are limitations of this pilot feasibility study that need to be taken into account. First, the country-specific data-collection processes were not standardized, meaning that each country collected its most recent and most reliable data, which led to different, mostly not comparable data bases. In addition, the accessibility of data was different between countries as well as the quality of the data themselves. Therefore, results of quality indicators are mostly not comparable. However, it was possible to evaluate the feasibility of quality indicator measurements in each country, which is an important requirement in setting up a systematic quality monitoring system.
Conclusion
This feasibility study on quality indicators for the mental healthcare systems of four countries in the Danube region clearly demonstrates that the collection of a majority of quality indicators is feasible by means of available data. However, quality data for mental healthcare are scattered across sectors, are reported from various and unconnected sources, and are of variable fidelity.
While quality indicators can serve as important measures in monitoring quality on different levels and in different domains of mental healthcare systems, besides trustful epidemiological data standardized, trans-sectoral routine data collection is needed to measure trans-national mental health quality objectively, timely, and repeatedly to monitor the development and success of mental healthcare reforms.
References
ANQ-Nationaler Verein für Qualitätsentwicklung in Spitälern und Kliniken (2019) Psychiatrie. https://www.anq.ch/de/fachbereiche/psychiatrie/. Accessed 26 June 2019
Armesto SG, Medeiros H, Wie L. Information availability for measuring and comparing quality of mental health care across OECD countries. DELSA/ELSA/WD/HTP(2008)2. https://www.oecd.org/health/health-systems/41243838.pdf. Accessed 6 Aug 2008
Bitter I, Kurimay T (2012) State of psychiatry in Hungary. Int Rev Psychiatry 24(4):307–313
Dimova A, Rohova M, Moutafova E, Atanasova E, Koeva S, Panteli D, van Ginneken E (2012) Bulgaria: Health system review. Health Syst Transit 14(3):1–186. https://www.euro.who.int/__data/assets/pdf_file/0006/169314/E96624.pdf. Accessed 2 July 2018
Dobiášová K, Tušková E, Hanušová P, Angelovská O, Ježková M (2016) The development of mental health policies in the Czech Republic and Slovak Republic since 1989. Cent Eur J Public Policy 10:35–46
Großimlinghaus I, Falkai P, Gaebel W, Janssen B, Reich-Erkelenz D, Wobrock T, Zielasek J (2013) Entwicklungsprozess der DGPPN-Qualitätsindikatoren. Nervenarzt 84:350–365
Großimlinghaus I, Falkai P, Gaebel W, Hasan A, Jänner M, Janssen B, Reich-Erkelenz D, Grüber L, Böttcher V, Wobrock T, Zielasek J (2015) Erhebung von Qualitätsindikatoren anhand von Routinedaten. Darstellung eines Machbarkeitstests in zehn Fachkliniken für Psychiatrie und Psychotherapie. Nervenarzt 86:1393–1399
Hermann R, Mattke S (2004) Selecting indicators for the quality of mental health care at the health systems level in OECD countries. OECD Health Technical Papers, No. 17, OECD Publishing. https://doi.org/10.1787/388745076135
Höschl C, Winkler P, Pec O (2012) The state of psychiatry in the Czech Republic. Int Rev Psychiatry 24(4):278–285
IQTiG (2017) Aktualisierung und Erweiterung des QS-Verfahrens Versorgung von volljährigen Patienten und Patientinnen mit Schizophrenie, schizotypen und wahnhaften Störungen. Abschlussbericht. Stand: 22. Dezember 2017.https://iqtig.org/downloads/berichte/2017/IQTIG_QS-Verfahren-Schizophrenie_Abschlussbericht_2017-12-22.pdf. Accessed 23 June 2019
JCAHO-Joint Commission on Accreditation of Healthcare Organizations (1989) Characteristics of clinical indicators. Qual Rev Bull 163:6401–6406
Kilbourne A, Beck K, Spaeth-Rublee B, Ramanuj P, O’Brien RW, Tomoyasu N, Pincus HA (2018) Measuring and improving the quality of mental health care: a global perspective. World Psychiatry. 17:30–38
Krupchanka D, Winkler P (2016) State of mental healthcare systems in Eastern Europe: do we really understand what is going on? B J Psych Int 13(4):96–99
Lecic Tosevski D, Pejovic Milovancevic M, Popovic Deusic S (2007) Reform of mental health care in Serbia: ten steps plus one. World Psychiatry 6:115–117
Lehmann I, Chisholm D, Hinkov H, Höschl C, Kapócs G, Kurimay T, Lecic-Tosevski D, Nakov V, Winkler P, Réthelyi JM, Zielasek J, Gaebel W (2018) Development of quality indicators for mental healthcare in the Danube region. Psychiatr Danubina 30:197–206
MacDonald AJD, Fugard AJB (2015) Routine mental health outcome measurement in the UK. Int Rev Psychiatry 27(4):306–319
Mainz J (2003) Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care 15(6):523–530
Ministry of Health of the Czech Republic (2013) Psychiatric care reform strategy. MHCR, Prague
Ministry of Health, Republic of Serbia (2007) Strategy for development of mental health care. Republic of Serbia, Ministry of Health
Mitchell J (2018) Mental health quality indicators: background and secondary definitions. Scottish Government. https://www.gov.scot/binaries/content/documents/govscot/publications/factsheet/2018/09/mental-health-quality-indicators-background-secondary-definitions/documents/00540532-pdf/00540532-pdf/govscot%3Adocument/00540532.pdf. Accessed 16 Oct 2019
Pincus HA, Hudson Scholle S, Spaeth-Rublee B, Hepner KA, Brown J (2016) Quality measures for mental health and substance use: gaps, opportunities, and challenges. Health Affairs 35:1000–1008
Reiter A, Fischer B, Kötting J, Geraedts M, Jäckel WH, Barlag H, Döbler K (2018) QUALIFY: instrument for the assessment of quality indicators. Version 1.0 (English). Düsseldorf, BQS gGmbH. Retrieved on 3rd August 2018. https://www.researchgate.net/publication/267256474_QUALIFY_Instrument_for_the_Assessment_of_Quality_Indicators
Spaeth-Rublee B, Pincus HA, Huynh PT, IIMHL Clinical Leaders Group (2010) Measuring quality of mental health care: a review of initiatives and programs in selected countries. Can J Psychiatry 55:539–548
WHO (1997) Quality Assurance in Mental Health Care – Checklists and Glossaries. Division of Mental Health and Prevention of Substance Abuse, 1 and 2. http://apps.who.int/iris/handle/10665/66931. Accessed 7 April 2020
WHO (2013) Mental Health Action Plan 2013—2020. Geneva: World Health Organization https://www.who.int/mental_health/publications/action_plan/en/. Accessed 26 June 2018
WHO Regional Office for Europe (2013) European Mental Health Action Plan 2013–2020. Copenhagen: World Health Organization https://www.euro.who.int/__data/assets/pdf_file/0020/280604/WHO-Europe-Mental-Health-Acion-Plan-2013-2020.pdf. Accessed 3 Jan 2018
WHO (2018) Mental Health Atlas 2017. https://www.who.int/mental_health/evidence/atlas/mental_health_atlas_2017/en/. Accessed 20 June 2018
Winkler P, Koeser L, Kondrátová L, Broulíková HM, Páv M, Kališová L et al (2018) Cost-effectiveness of care for people with psychosis in the community and psychiatric hospitals in the Czech Republic: an economic analysis. Lancet Psychiatry 5(12):1023–1031
Winkler P, Krupchanka D, Roberts T, Kondratova L, Machů V, Höschl C, Sartorius N, Van Voren R, Aizberg O, Bitter I, Cerga-Pashoja A (2017) A blind spot on the global mental health map: a scoping review of 25 years' development of mental health care for people with severe mental illnesses in central and eastern Europe. Lancet Psychiatry 4(8):634–642
Winkler P, Mladá K, Krupchanka D, Agius M, Ray MK, Höschl C (2016) Long-term hospitalizations for schizophrenia in the Czech Republic 1998–2012. Schizophr Res 175(1–3):180–185
Acknowledgements
This study was funded by the German Ministry for Education and Research (BMBF) under the title: “Development and Implementation of Quality Indicators for Mental Healthcare in the Danube Region” (DAQUMECA, project ID 01DS17020).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Additional information
Communicated by Andrea Schmitt.
Rights and permissions
About this article
Cite this article
Gaebel, W., Lehmann, I., Chisholm, D. et al. Quality indicators for mental healthcare in the Danube region: results from a pilot feasibility study. Eur Arch Psychiatry Clin Neurosci 271, 1017–1025 (2021). https://doi.org/10.1007/s00406-020-01124-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00406-020-01124-z