Introduction

Countries of the Danube region like Bulgaria, the Czech Republic, Hungary, and Serbia are aiming to transform their institution-based mental healthcare systems to more community-based systems, thereby improving and assuring quality of care [3, 4, 9, 14, 29]. In this context, quality indicators need to be applied to assess the status-quo and to monitor the development and success of the transformation and implementation processes. Healthcare system quality indicators are generally defined as “quantitative measures that can be used to monitor and evaluate the quality of important governance, management, clinical, and support functions that affect patient outcomes” (JCAHO 1989, quoted in [11], p. 524). They reflect aspects of the structures, processes, and outcomes on different healthcare hierarchy levels, including the care systems (macro-) level, the healthcare institutions (meso-) level, and the individual healthcare professionals and patients (micro-) level. Indicators can be used to benchmark quality of care, to assess the current state (as-is situation) and to establish the targeted future state of mental healthcare, and to monitor change processes [15, 17]. Topics such as deinstitutionalization, healthcare coverage and accessibility, workforce development, or anti-stigma activities are important quality domains in the transformation of mental healthcare systems in the four countries [3, 4, 9, 13, 14, 29, 30]. Accordingly, a global quality target as defined by the World Health Organization (WHO) Mental Health Action Plan is that by the year 2020, 80% of all countries should be routinely collecting and reporting a set of core mental health indicators [25]. This requires a universal and agreed set of quality indicators to be developed, based on methodologically sound and thematically appropriate data sources [8, 23]. WHO’s Mental Health Atlas is the most global attempt to capture a standard set of mental health care indicators across countries [27]. In Europe, quality indicators have already been developed and at least been partially implemented in some countries, such as Germany [6, 7, 10], Scotland [20], Switzerland [1], and the United Kingdom [16].

Against this background, a 2-year project lasting from April 2017 to March 2019, funded by the German Federal Ministry of Education and Research (BMBF), was conducted under the title “Development and Implementation of Quality Indicators for Mental Healthcare in the Danube Region” (DAQUMECA). The aim was to develop quality indicators for the mental healthcare systems of Bulgaria, the Czech Republic, Hungary, and Serbia, and to test these quality indicators regarding their data availability. It was focused on developing quality indicators for measuring the macro- and meso-level of healthcare. In a first project phase, international quality indicators were identified via a systematic literature search, subsequently narrowed down with reference to defined inclusion and exclusion criteria, and then rated for their validity, reliability, and feasibility in a two-stage Delphi study by a multi-disciplinary expert panel from Bulgaria, the Czech Republic, Hungary, and Serbia [15].

The ratings of the Delphi panelists (n = 18) in the first project phase provided a first overall impression of data availability for the quality indicators through an expert-based estimate [15]. However, actual data availability needs to be more objectively assessed in a feasibility study, where data sources are first to be identified for their availability, and then, the data-collection effort can be evaluated additionally. According to Lehmann et al. [15], the Delphi experts mostly agreed on the relevance and validity of the quality indicators. For only six indicators (QI 14, 17, 18, 19, 20, 22, Table 1), there was some uncertainty about the relevance or validity of the indicators with a tendency towards agreement. Overall, there was no disagreement on the relevance or validity of any of the quality indicators [15]. Accordingly, all identified quality indicators shown in Table 1 were tested in a feasibility study, with the exception of QI 22 relating to assisted housing, since the rating of data availability was extremely low for this QI (less than 10% of Delphi panelists expected electronic data to be available). The project consortium members, therefore, decided to not include QI 22 in the feasibility study, since they also agreed on the non-availability of data sources for this QI. Accordingly, 21 QI were evaluated for their feasibility by means of the existing databases in Bulgaria, the Czech Republic, Hungary, and Serbia.

Table 1 Systematically developed quality indicators for Bulgaria, the Czech Republic, Hungary, and Serbia [15]
Full size table

An overview of the developed quality indicators from the first project phase is given in Table 1 [15].

Methods

The feasibility study of the 21 quality indicators focused on current data availability and data-collection effort in each country. Thus, the goal was to target data availability by sources within each country; it was not an aim to compare quality indicator results among countries, because this would have required trans-national standardized data collection.

In mental healthcare, different data sources with different levels of data standardization are available. Administrative data, which are typically used for insurance or reimbursement purposes, such as data on diagnostic and therapeutic procedures, are, in all countries, generally the most standardized and are available electronically. Routinely collected clinical data, such as documented data on adverse effects of therapeutic procedures, can be but are not necessarily standardized. They can be available electronically or in paper files. Finally, additional data, such as patient-reported data, may be collected electronically and can be used to measure quality indicators [22]. In this study, the term “routine data” is used to describe any officially and regularly documented data in the mental healthcare systems, including administrative and clinical ‘routine’ data.

To collect data for the present feasibility study the following potential data sources were defined:

  • Overarching data sets from different care settings and sectors (= cross-sectoral routine data), e.g., data from (national) statistical offices, ministries of health, provider or patient registries, legislative texts, and policy/action plan documents

  • Separate data sets from different care sectors (e.g., inpatient vs. outpatient vs. community care) (= sector-specific routine data), e.g., health insurance data

  • Data generated from retrospective or prospective surveys (= data from additional data collections, either from continuous data collections or intermittent data collections), e.g., data from epidemiologic studies

Between March and August 2018, data were collected by all country partners from Bulgaria, the Czech Republic, Hungary, and Serbia in a retrospective manner. The aim was to collect the data from the best available/most trustworthy databases in each country with the latest available data. Data were summarized into data tables including the following information per quality indicator:

  • Data availability (Yes/No)

  • Data sources (please specify)

  • Calculation of numerator/denominator (if applicable to QI)

  • Result of QI

  • Calculation period

  • Comments.

An interim discussion about the status of the data collection and possible obstacles took place in June 2018. After each country collected their data, the results were summarized in one table to identify similarities and differences with regard to the trustworthiness of the data sources and collected data. These topics were discussed in a structured group meeting with all project partners in March 2019. The aims of this meeting were to identify obstacles in the data-collection processes and to discuss the trustworthiness of the collected data.

Results

Table 2 shows all quality indicators and whether data were available in each country. 81–86% of the quality indicators could be measured based on the existing data sources in Bulgaria, the Czech Republic, and Hungary. Only 38% of all QI could be measured in Serbia.

Table 2 Data availability for QI differentiated by country
Full size table

Eight quality indicators (QI 1–6, QI 16, QI 19) could be measured in all four countries. Seven quality indicators (QI 8–11, QI 17, QI 18, QI 20) could be measured in three countries. One quality indicator (QI 14) could only be measured in the Czech Republic and Hungary. One indicator (QI 15) could only be measured in Hungary. For the two indicators on the utilization of mental health services in children and adolescents (QI 12, QI 13), no data were available in any country.

Overall, the main data sources available to measure each quality indicator are summarized in Table 3. In some QI, multiple data sources were used to measure the numerators and denominators of the QI. For example, QI 7–QI 11 measure the number of persons with a certain diagnosed mental disorder who receive mental health treatment (numerator) in relation to all persons with the same but untreated either diagnosed or still undiagnosed (but diagnosable) mental disorder within a defined time period (denominator). The denominator was measured on the basis of (epidemiological) survey data, while the numerator was measured based on health care utilization or insurance data.

Table 3 Main data sources per QI
Full size table

Five quality indicators could be measured with one data source (QI 1–4, QI 15). Fourteen quality indicators were measured with data from more than one data source (QI 5–11, QI 14, QI 16–21).

Most quality indicators (n = 13) were measured with routinely available data that are collected in central offices including national statistical offices, ministries of health and provider, and patient registries of persons with mental disorders or with information that can be extracted from legislative texts and policy/action plan documents (cf. Table 3). Twelve quality indicators were measured by means of health insurance data, including the quality indicators on the utilization of mental healthcare (QI 7–12). For the quality indicators on the utilization of mental healthcare also survey data that are collected in some of the participating countries either recurring or non-recurring were used (QI 7–12).

The data-collection effort was evaluated in the project meetings. The following observations were made in the participating countries:

  • Routine data to measure quality indicators are not available for all quality indicators, leading to increased data-collection efforts.

  • Collection from different data sources overall increased the data-collection effort.

  • Time spent on data collection varied between countries and the type of data needed. Time ranged between immediate data availability (e.g., from online published national health statistics) to a waiting time of up to 6 months before data were available (e.g., data queries for health insurance data).

  • Possible prospective collection of additional data may be necessary to measure the complete quality indicator set. However, this was not feasible within this study.

  • There is a shortage of epidemiologic data (including the incidence and prevalence of mental disorders) due to a lack of nationwide surveys in Hungary and Serbia. In Bulgaria, epidemiologic data are not available in open access.

  • For some quality indicators, data were available, but the quality indicators’ definitions should be aligned with available data to improve the validity of the measurement (i.e., the indicator measures what it intends to measure).

Both, the existence of mental health legislation (QI 1) and the existence of a mental health action plan (QI 2) form a framework, in which the requirements and strategies for high-quality person-centered mental healthcare can be further defined. Therefore, these two quality indicators are regarded as essential, and in the following, the results of each indicator are shown per country (Tables 4, 5).

Table 4 Results of QI 1 mental health legislation
Full size table
Table 5 Results of QI 2 availability and content of a mental health action plan
Full size table

For both indicators, information was extracted from legislative texts and mental health action plans.

Discussion

This pilot feasibility study provides a first overview of the feasibility of systematically developed mental health quality indicators to be used for regular quality monitoring in countries of the Danube region. Data availability is an important prerequisite in using indicators for measuring healthcare quality. The developed quality indicators mainly focus on structural—as opposed to process or outcome-based—aspects of the mental healthcare system (e.g., availability of mental health service structures, expenditures, and workforce). A survey across 18 member states of the Organization for Economic Cooperation and Development (OECD) showed that availability of structural data is generally very good in mental healthcare [2]. While two countries (Bulgaria and Serbia) in our feasibility study are not OECD countries, all of the participating countries except for Serbia, which is also the only country that is not a member of the European Union, have a high data availability for the quality indicators in this study that focus on structural aspects. These indicators can be used to identify the quality of mental healthcare systems at the macro level, including the availability of legislation, mental health policies, and action plans, the level of mental health budget and expenditures on the health system and per capita, the availability of human resources for mental health, the utilization of services, and also anti-stigma activities. At the meso-level of mental health services, these quality indicators can also be used to measure important domains of service availability such as continuity of care, the number of employed workers, and the number of beds and places.

The different data sources that were used to measure the relevant data for the various quality indicators exhibit mainly the problem of fragmented information systems in parallel to fragmented healthcare systems in all four countries. Fragmented data sources are also internationally a still much more prevalent problem in mental healthcare compared to healthcare for somatic disorders [2, 12]. Data for mental healthcare processes and outcomes are even less readily available than structural data due to lacking data infrastructure [2, 12, 21]. The process indicators “QI 15 Follow-up of visits after mental health-related hospitalization” and “QI 21 Involuntary inpatient admissions”, for example, are less measurable with currently existing data (Table 1).

The best available data sources include national surveys and registries as well as administrative hospital databases [2]. In general, national epidemiological surveys were available only in the Czech Republic and Bulgaria, demonstrating the need for such surveys in Hungary and Serbia. The results of the Bulgarian epidemiologic survey, however, are not accessible for external parties, other than the consortium that conducted the survey, so that they were not used in our study. All four countries were able to obtain data from national statistics offices that collect population data, which can be used for quality indicator measurement (Table 1). Administrative hospital databases were also available in each country. However, these are generally in-house data and not readily available for external parties.

The existence of mental health legislations (QI 1, Table 4) is an important prerequisite in prioritizing mental healthcare on political agendas and in the further formulation of mental health aims [24]. A mental health action plan (QI 2, Table 5) in turn specifies the strategies to reach the aims of legislation, to set priorities and establish resources and timelines. Mental health action plans, therefore, form important tools in the transformation of mental healthcare systems towards high-quality integrated care services. The results on the availability of quality indicators demonstrate that a mental health action plan is only available in two of the four countries (Czech Republic and Serbia, Table 4 and 5). The action plans in the Czech Republic and in Serbia comprise strategies and aims on the promotion of mental health, prevention, treatment, and rehabilitation of mental disorders and activities against stigma and discrimination of persons with mental disorders (Table 5) [5, 19, 14, 18, 28].

However, only the Czech mental health action plan has dedicated funding for its implementation available, and the budget has been used for this intended purpose as shown in the QI results in Table 5. The Serbian mental health action plan lacks funding and, therefore, has so far only an informative character.

Besides policy limitations and limitations due to fragmented mental healthcare provision, technical limitations form an important barrier in the overarching measurement of quality indicators in mental healthcare. Information technologies are lagging behind in mental healthcare in comparison to other fields of medicine, including a lack of electronic patient records and of routine collection and reporting of quality indicators [12].

While it is crucial to monitor mental healthcare quality, data collection needs to be meaningful and reliable, because inefficient and overwhelming demands for data collection can impede motivation and hamper the capacities for providing high-quality mental healthcare [26]. Ideally, quality indicators are linked to electronic health records that collect mental health data and are calculated based on systematic data collections [7, 12].

Therefore, as a follow-up to this quality indicator study, it is planned to develop and implement in the consortium partner countries a trans-national digital mental health platform including the quality indicators based on standardized data and also other functionalities to support the further reform process in these and other European countries. The platform should serve as an important information and navigation tool in increasing transparency of mental healthcare and in monitoring the transformation from a fragmented, hospital-based mental healthcare system towards integrated, needs-based, and people-centered mental healthcare.

The main responsibility for data collection on mental healthcare systems performance lies with each country. The comparison of universal quality indicators on an international level can be used within a benchmarking of mental healthcare performance, can support an improved international networking, and can initiate a learning from each other. For an international benchmarking, a core set of quality indicators should be used measuring important aspects of mental health system governance (legislation and policies), resources (financing and workforce), and availability (number and geographical distribution of facilities), all of which are included in our proposed quality indicator set. The WHO Mental health ATLAS [27] collects similar data that can be used as reference values for the quality indicators included in this analysis. However, the WHO ATLAS results show that data collections for quality assurance purposes still need to be improved, as the four countries incompletely report on the WHO ATLAS indicators.

Overall, there are limitations of this pilot feasibility study that need to be taken into account. First, the country-specific data-collection processes were not standardized, meaning that each country collected its most recent and most reliable data, which led to different, mostly not comparable data bases. In addition, the accessibility of data was different between countries as well as the quality of the data themselves. Therefore, results of quality indicators are mostly not comparable. However, it was possible to evaluate the feasibility of quality indicator measurements in each country, which is an important requirement in setting up a systematic quality monitoring system.

Conclusion

This feasibility study on quality indicators for the mental healthcare systems of four countries in the Danube region clearly demonstrates that the collection of a majority of quality indicators is feasible by means of available data. However, quality data for mental healthcare are scattered across sectors, are reported from various and unconnected sources, and are of variable fidelity.

While quality indicators can serve as important measures in monitoring quality on different levels and in different domains of mental healthcare systems, besides trustful epidemiological data standardized, trans-sectoral routine data collection is needed to measure trans-national mental health quality objectively, timely, and repeatedly to monitor the development and success of mental healthcare reforms.