Introduction

As health care expands to meet the increasing global needs, the quality of health services has become an affair of growing importance to societies (Smith et al. 2006). The World Health Organization defines health care quality by effectiveness, cost and social acceptability and emphasizes its importance in health service delivery (Reinhardt 2000). This is the case in particular for rural and remote areas where health care resources are more constrained, access more limited and poor health outcomes more common when compared to urban areas (Humphreys and Solarsh 2008). A number of studies have investigated the quality of rural health services (Sharma and Narang 2011) and have explored the mechanisms and models for quality evaluation (Smith et al. 2006). Guaranteeing the quality of health services plays a significant role in improving the health status in rural areas (Cao 2011). As is known, competitiveness of health institutions is measured principally by the quality of their health services, and these are directly related to the health status of local residents, being part of their vital interests. Therefore, improving the quality of health services is an important objective, not only crucial for rural health promotion, but also vital to the sustainability of health service institutions (Tham et al. 2010).

Routine tracking and evaluation of health service quality are essential to improve health care quality. For health managers and decision makers, current knowledge on the quality of a specific type of health service might help to identify opportunities and make timely and effective decisions for improving quality by fostering behavioral change in service seeking and service delivery by more efficient resource allocation and utilization and by improving care delivery processes and facilities (Pembe et al. 2010; D’Ambruoso et al. 2009). Theoretical models, frameworks and methods are essential to guide the development, routine tracking applications and evaluation of the quality of rural health services. Given the local content of rural health services, the theoretical framework must not only be valid, but also adaptive and usable. The extent to which the existing literature provides theoretical support and informs application in various social-political contents is less known.

The purpose of this study is to systematically review the published literature to synthesize the existing systems with respect to theory, metrics and their application in tracking and evaluation of the quality of health services in rural and remote areas. Our review looks at the settings, contents and results of service quality assessment in published studies.

Material and methods

Search and strategy

We systematically searched the PubMed, Cochrane Library, Campbell Library, Web of Science, CBM, CNKI and WANFANG databases in March 2016. We also searched websites related to health care such as the World Health Organization. We used the following search terms and basic strategy: (trac* OR assess* OR evaluat* OR supervis* regulat* OR oversight) AND (rural OR remote OR poverty) AND (“health service” OR “health services”) AND (mechanism OR system OR framework OR model OR pattern). In addition, we browsed the references of the included studies as a supplemental search.

Inclusion criteria

Reports of original research studies were included if they met the following criteria: (1) focusing on the health service quality trace or evaluation in rural or remote areas; (2) researching the theoretical model, such as the mechanism, system, method, model and framework; (3) the language of the publication is Chinese or English. Reports on study protocols, abstracts, commentaries or studies that did not report data were excluded.

Data extraction

We first developed a standard protocol for data extraction. Two independent investigators extracted data and assessed the quality of each study included. In case of a disagreement, consensus was achieved through discussion. The extracted information included general characteristics (i.e., authors, countries of studies and corresponding authors, year and language of publication, journal title, journal category and study design) of the studies included and items (i.e., style, setting, subject, composition and outcomes) related to the theoretical models employed.

Quality assessment

We evaluated the quality of each study using the Ekman scale (Ekman 2004). This instrument uses 13 critical questions covering seven different subjects: research/analytical question(s), rationale, methodology, data, goal achievement, findings and results, as well as discussion and conclusions, each of which is considered of virtual importance for scientific quality. For each question, a score of 0, 1, 2 or 3 was given and then the total score was calculated. Three stars (22–25 points) was the highest grade of quality, two stars (17–21 points) the medium grade and one star (0–16 points) the lowest grade. See Appendix 1 for further details of Ekman’s grading system.

Data synthesis and analysis

The present review focused on reported study-observed settings, contents and results of application of theoretical systems for tracking and evaluating rural health service quality. Data synthesis was mostly qualitatively described and analyzed. Descriptive summary statistics (frequency and median) were calculated for each specified item when feasible. Microsoft Excel 2003 (http://office.microsoft.com/zh-cn) was used.

Results

Literature screening

The initial search yielded 4384 potentially relevant references, 630 studies in Chinese and 3754 in English. After removing duplicates and screening by titles and abstracts, 4278 articles were excluded. After reading the full texts of these 106 articles, 88 did not meet the inclusion criteria and were excluded. Further review selected only 18 studies, which were subsequently included for data extraction and analysis (Fig. 1).

Fig. 1
figure 1

Flow chart of the study selection process

Full size image

General characteristics of the included studies

The 18 articles were published between 2001 and 2015 (Fig. 2). Nine studies (50%) were published in Science Citation Indexed journals (SCI). The impact factor of these journals ranged from 0.663 to 5.296. All studies were cross-sectional except one. The corresponding author(s) of eight studies were affiliated with Chinese institutions and three with Australian institutions. Over a quarter (5, 28%) of these studies focused on service quality at township hospitals and the same number (5, 28%) investigated the quality of the primary health care service. Four studies (22%) were about clinical service.

Fig. 2
figure 2

The year distribution of the included studies

Full size image

Of these studies, seven were rated as medium quality (7, 39%) and the rest as high quality (Table 1).

Table 1 General characteristics of the 18 studies on rural health service quality included in the systematic review
Full size table

Theoretical framework, instruments or model

Ten of the 18 studies, (56%) involved a theoretical framework such as the Primary Health Care (PHC) evaluation framework, the Integrated Community Family Health Development Programme, theoretical framework of performance evaluation of township hospitals and other evaluation frameworks. Eight articles investigated or applied seven different instruments, such as the Primary Care Assessment Tool (PCAT), SERVQUAL tool, balanced scoring method, 360-degree performance appraisal method, goal management method and TOPSIS. Four articles investigated theoretical models: the Data Envelope Analysis (DEA) model, three-party comprehensive evaluation model, structure process result model and Elmore Primary Health Service (EPHS) evaluation model (Table 2).

Table 2 Existing systems of tracking and evaluation of rural health service quality
Full size table

Model/pattern

Four studies (Cao 2011; Zhou et al. 2012; Tham et al. 2010; Wang et al. 2007) investigated models. The DEA model (Zhou et al. 2012) integrates concepts of mathematics, economics and management and has been used to assess rural public health service in Southwest China. The comprehensive evaluation model (Wang et al. 2007) involves three parts, “Structure-Process-Result,” as well as the pattern of “Technology-Functional quality.” This model has been applied to township clinics. The Continuous Quality Improvement (CQI) model (Cao 2011) focused on service quality along the process pathway of “Plan-Do-Check-Act (PDCA)” and has been used for rural public health care and service. The EPHS evaluation model (Tham et al. 2010) focused on organizational structure and behavior on service quality in primary health care across varying tiers of health care providers. The quality of the studies was medium for the DEA (Zhou et al. 2012) and comprehensive evaluation models (Wang et al. 2007) and high for the CQI (Cao 2011) and EPHS models (Tham et al. 2010). Additionally, the CQI and EPHS models generated effective results.

Instruments (methods or tools)

Eleven different methods (Cao 2011; Xiong 2008; Liu 2009; Li and Ke 2011; Wang et al. 2015) and three tools (Clapham et al. 2004; Chen et al. 2011; Shaikh et al. 2008) were included in this review. The most frequently reported method was TOPSIS (Li and Ke 2011; Wang et al. 2015), which is designed to identify the disparity between evaluation objects and has been used in Southwestern and Northern China to improve the quality of health services at township clinics. Many of the instruments were used and reviewed for health service assessment and achieved effective results (Cao 2011; Xiong 2008; Liu 2009; Li and Ke 2011; Wang et al. 2015).

Meanwhile, the quality of the studies (Cao 2011; Clapham et al. 2004; Shaikh et al. 2008; Xiong 2008; Liu 2009; Wang et al. 2015) was high for all instruments but PCAT and TOPSIS. The quality of essential obstetric care monitoring tools (Clapham et al. 2004) included ten elements and has been used at minimum for quarterly review and for the development of action plans. The SERVQUAL tool (Shaikh et al. 2008) and PCAT (Chen et al. 2011) were mentioned as useful for monitoring rural health service quality. However, the authors did not provide detailed results of the quality assessment.

Framework

This review refers to five distinct theoretical frameworks. A conceptual framework is reported in six articles (D’Ambruoso et al. 2009; Burgos et al. 2013; Clapham et al. 2004; Puett et al. 2013; Tham et al. 2010; Reeve et al. 2015), and half of them were rated with high quality. It has been broadly applied for public health service quality (Burgos et al. 2013), primary health care service quality (Reeve et al. 2015), general clinical service quality (Clapham et al. 2004; Puett et al. 2013), health organization behavior evaluation (Tham et al. 2010) and access to health services (D’Ambruoso et al. 2009) in rural areas. The conceptual framework is flexible in content with respect to service elements received and outcomes. It can further provide guidance for research in health service evaluation and afford better understanding about the way health services impact rural health. Four articles (Aldana et al. 2001; Xiong 2008; Tham et al. 2011; Lawson et al. 2012) reported other frameworks that might achieve effective results. These four studies were rated with medium and high quality.

Discussion

This review has identified 18 articles investigating the application of 16 kinds of systems that can be used to track and evaluate the quality of health services in rural and remote areas. These systems of quality assessment have been applied to assess service quality in various health service settings: primary health care, public health and organization behavior. In the last years, the number of studies in this subject area has seen steady growth. Almost half of the included studies were published in SCI journals. Most of the studies were cross-sectional with high quality. However, the scientific quality of the included studies might not be directly extrapolated in an appropriate formulation of conclusions.

Strengthening of primary health care has been sought in the last years as a mechanism to improve health equity for many nations (World Health Organization. 2008). From the included studies, one type of model and three types of frameworks reported positive results in primary health care in different aspects. For example, the EPHS evaluation model might offer beneficial outcomes in primary health care (Wang et al. 2015). On the other hand, the Australian evaluation framework seems to help sustain rural primary health care services (Tham et al. 2011). The PHC Evaluation Framework may improve the overall quality of care (Lawson et al. 2012), and the conceptual framework might contribute to a continuous quality improvement (Reeve et al. 2015). These theoretical models differ in contents and methods. For users, they should be selected and adjusted according to their target and setting. For researchers and developers, these models will benefit from more practice to be able to understand the scope of long-term effects. More research is needed on the selection and integration of the existing concepts and frameworks and development of new theoretical models.

In this study, theoretical models referred to clinical services including their conceptual framework (Puett et al. 2013), PCAT (Chen et al. 2011), SERVQUAL tool (Shaikh et al. 2008) and the framework of the Integrated Community Family Health Development Programme (Shaikh et al. 2008), which enable decision-makers to improve the quality of medical care effectively. Besides, the SERVQUAL tool might improve patient satisfaction. However, as a limitation, it is important to mention that these studies are cross sectional, having a low level of evidence in the grading system (Wang 2014). Therefore, more high-quality studies are required.

Public health service is an important part of health care (Shao 2010). Theoretical models on public health including the DEA model (Zhou et al. 2012) and conceptual framework (Burgos et al. 2013) were reviewed in this study. The conceptual framework was regarded as essential to improve interventional strategies and connection of actors impelling environmental actions and policies (Burgos et al. 2013). In China, the rural population accounts for more than 50% of the total population (Zhang 2007). The rural public health system bears the responsibility of providing public health services to about 700 million residents. An effective mechanism for quality evaluation of the public health service in rural areas is lacking, and this has become a great challenge (Guo et al. 2011).

Unfortunately, there are few studies on local health service policies. The general objective of a health policy is to achieve the goal of equilibrium and social welfare through the solution of health problems using specific laws and regulations including the formulation of various health measures and methods (Tang 2011). Countries and governments have, at all levels, always reiterated the great importance of working to improve rural health and have promulgated many health policies trying to benefit the masses of farmers. However, for different reasons, disparities exist among health policies, their implementation and the benefited population (Wang et al. 2014). Policy evaluation is an indispensable part of evaluating the whole system (Yi 2013). Considering its complexity, more attention should be paid by evaluators and researchers.

Our study has some limitations. The studies included in this review might not represent the overall theoretical systems used for studies in this field as our study was limited to the data sources presented. Future higher quality research in this field might modify our conclusions. In addition, we only included studies in English and Chinese. Finally, although most of the studies had moderate quality assessed by the Ekman scale, they were considered low level evidence, which might impact the conclusions of this study.

Conclusion

In summary, most of the existing theoretical systems for tracking and evaluation of health service quality in rural and remote areas obtained effective outcomes. However, in this field, the number and design of research on the systems are limited. Unified standards and a comprehensive evaluation are lacking and facing a challenge. A higher level of evidence for a more comprehensive and realistic system will be necessary.