Significance

What is Already Known on this Subject?

In the absence of a complete and accurate civil registration and vital statistics system, the reliability of maternal and neonatal mortality is an important concern. In such circumstances, an enhanced maternal and perinatal mortality surveillance system that captures data at sub-district level using the existing health information system, and extending coverage to previously uncovered areas, is a better alternative.

What Does this Study Adds?

Application of capture recapture methods provides evidence of a high level of completeness of the enhanced mortality surveillance system in a rural district of Pakistan.

Introduction

A complete and accurate civil registration and vital statistics system provides the most reliable measure of maternal and neonatal mortality. However, in most developing countries, including Pakistan, the data generated by this system are seldom used for monitoring and improving health outcomes due to poor quality and population coverage (Mahapatra et al. 2007). The accuracy of vital statistics data used for mortality rates is mostly determined by coverage (proportion of the population covered by reporting system), and completeness of the reported events within covered areas (Mahapatra et al. 2007). A civil registration system with complete coverage, high completeness, and accurate data on cause of death is the most desirable (Joubert et al. 2012). However, in developing countries other sources of data may also need to be used (e.g. from health services) and corrected for inadequate coverage or completeness as required. Maternal deaths may be missed due to the misclassification of cause of death (Mir et al. 2015), and components of perinatal deaths may be misclassified by the incorrect recording of death in relation to parturition.

Sample-based population surveys, such as the Pakistan Demographic Health Survey 2006/07, provide useful measures of maternal and neonatal mortality at the national and sub-national level (National Institute of Population Studies 2008). Because such surveys are based on a sample of the population rather than a complete count of events, they cannot provide accurate estimates at the district level (Cesare et al. 2015; Godefay et al. 2014). District and sub-district level mortality estimates are crucial in targeting interventions to the areas in most need, yet are unavailable in Pakistan (Barnett et al. 2008). Furthermore, intermittent demographic health surveys do not provide sufficient current trends over time to adequately assess changes; and are based on questionnaire recall data, rather than registration of numerator and denominator events. In many developing countries, the majority of the births take place at home. For instance, 60% of the deliveries in rural areas of Pakistan occur at home in 2012, and of these only 23% of the children under 5 years are registered at birth (National Institute of Population Studies (Pakistan) and ICF International 2013). In such circumstances, with incomplete vital registration and under-reporting of mortality rates, a community-based comprehensive surveillance system is a far better alternative than continuation with the status quo (World Health Organization 2016). Such an enhanced surveillance system was established for a 1 year period as part of a study in the Tehsil of Havelian (population 341,900), district Abbottabad, in Khyber Pakhtunkhwa Province, Pakistan (Anwar et al. 2018). The enhanced surveillance study population comprised 51,690 married women of reproductive age (18–49 years) residing in all 16 Union Councils (each 15–25 villages) of Tehsil Havelian. From these women, there were 7580 pregnancies and 7273 live births over the period 1 year (June 2015 to May 2016). The data were collected from the community by Lady Health Workers augmented by specially recruited Community Health Workers for previously uncovered areas, and data were also collected from public and private hospitals in an attempt to ensure all births and deaths were reported.

A comprehensive surveillance system requires evaluation to ensure that it is operating as intended by capturing all required events. There are various attributes of a surveillance system including data quality, completeness and accuracy (German et al. 2001). Among the various techniques used to evaluate the qualities or attributes of a surveillance system, capture–recapture methods have been employed widely (Azofeifa et al. 2012).

In this study, we assess the accuracy and completeness of the enhanced surveillance data system, and calculate corrected maternal, perinatal and neonatal mortality rates at a sub-district level in Pakistan.

Methods

Study Design

A cross-sectional survey was conducted and the events captured from this were compared with those captured by the enhanced surveillance system using capture–recapture methodology, as described in the dual record system of Sekar and Deming (1949). The total births and deaths in the study population were the addition of those captured by the enhanced surveillance system, and those estimated as missed.

Setting and Study Population

A stratified random cross-sectional survey was conducted from 09 December 2015 to 13 January 2016 in Tehsil Havelian, 6 months after the start of the enhanced surveillance system in the study area. All these households were visited and all married women in the households aged 18–49 years were invited to participate. Women were asked about births after 1 June 2015, which is when the enhanced surveillance system commenced. In case of maternal deaths, husband, sisters or mother-in-law of the deceased women were interviewed.

The sample size of 1228 households was determined by feasibility and resources, and informed by sample size calculations. Using the population neonatal mortality rate of 55 per 1000 live births from the most recent Pakistan Demographic Health Survey 2012/13 (National Institute of Population Studies (Pakistan) and ICF International, 2013), this sample would provide a precision of ± 23% of 55/1000 (i.e. 95% CIs 42–68/1000). The survey sample was drawn from the list of 51,690 married women, prepared by the Lady Health Workers and Community Health Workers at the commencement of the enhanced surveillance system, and distributed in proportion to the population of each of the 16 Union Councils in Tehsil Havelian (Table 3 in Appendix). Each woman in the list was assigned a unique code (number) and entered into a SPSS database with their Union Council code. The population proportion of each of the 16 Union Councils was used to draw a random sample of 1228 women by code number employing SPSS version 24 (Complex Sample Plan). The selected codes were used to identify the women by name. The final list included women’s and their husband’s names and addresses, including the name of the Union Council used to locate the household. Since coverage of the enhanced surveillance system could not be maintained by the assigned community health workers (who were then withdrawn) in 4% of the study population because of remoteness, isolation and the ruggedness of the terrain, completeness was assessed in the 96% of the population that was actually covered by the enhanced surveillance system.

Tools and Instruments

The “Short Household Questionnaire” used in the Pakistan Demographic and Health Survey 2006/07 (National Institute of Population Studies 2008) was adapted for the survey. The instrument captures data on total members of the household, births, infant and child deaths to a woman during last 3 years, along with the date of birth, sex, date and age of deaths, and pregnancy-related information. The sisterhood method was used to ascertain maternal deaths in the household (National Institute of Population Studies 2008). Twelve trained interviewers, independent from the routine data collectors of the enhanced surveillance system, administered the questionnaires.

Matching

Birth and death records collected the interviewers separately employed for the sample survey were matched with the birth and death data collected by the Lady Health Workers and Community Health Workers who collected data for the enhanced surveillance system. The following variables which were common in both data sources were used for matching births and deaths: names of (1) the Union Council-Geographical Unit and (2) Lady Health Worker or Community Health Worker; (3) age of the woman; (4) date of birth and (5) sex of newborn. The names of the women were not available for matching purposes as they had been removed from pregnancy records to comply with ethics requirements to ensure the privacy of the women.

Data Analysis

The Petersen–Lincoln equation was applied (adjusted for bias from small numbers) to estimate the total number of births and deaths in the study population (Chao et al. 2008). The following formula was used.

$${\text{N}}=\left[ {\left( {{\text{n}}+1} \right)\left( {{\text{m}}+1} \right)/\left( {{\text{M}}+1} \right)} \right]-1$$

where N is the total number of births or deaths in the study population, n is the number of births or deaths recorded by the enhanced surveillance system, m is the number of births or deaths captured by the survey, and M is the number of births and deaths that are captured by both the enhanced surveillance system and the cross-sectional survey (Fig. 1). We calculated 95% CI of the estimates of births and deaths by calculating the variance (Var) using the following formula (Chao et al. 2008).

$${\text{Var}}({\text{N}})=({\text{n}}+1)({\text{m}}+1)({\text{n}} - {\text{M}})({\text{m}} - {\text{M}})/[{({\text{M}}+1)^2}({\text{M}}+2)]$$
$$95\% {\text{CI}}= \pm {\text{N}} \times 1.96 \times \sqrt {{\text{Var}}\,(N)}$$
Fig. 1
figure 1

Application of capture–recapture methods in the study areas, District Abbottabad, Pakistan

Full size image

Mortality rates in the study area calculated from the enhanced surveillance system data were compared with the mortality rate in the study area after adjusting for the estimated missed births and deaths; and mortality rate ratios and corresponding 95% CI were calculated. To investigate the statistical significance of the difference between the enhanced surveillance data and the data produced after correction for under-enumeration using capture–recapture, the 95% CIs from the proportions (rates or ratios) and numbers were inflated (widened) for the additional contribution of the variability from the capture–recapture calculations derived from the variances. Completeness is defined as the ability of the surveillance system to capture all births and deaths in the study area, and is measured by dividing the enumerated births or deaths captured by the enhanced surveillance system divided by the total births and deaths expected to have occurred in a defined area. Maternal death is defined as the death of a woman dying during pregnancy or within 42 days of termination of pregnancy. Stillbirth is defined as the baby born without signs of life after 28 weeks of gestation. Neonatal death is defined as the death of a newborn infant in the first 28 days of life (0–27 days). Early neonatal deaths are newborns dying within the first 7 days of life (0–6 days). Perinatal deaths are stillbirths and early neonatal deaths. Statistical analysis was performed using SPSS Version 24 (SPSS Inc., Chicago, IL, USA).

Results

A total of 1228 women (households) were visited, of whom 1156 (94%) participated and completed the questionnaires. Fifty (4%) women refused to participate in the interview. Seventeen (1.4%) women shifted to a new location within the study area after the start of the project, and five women (0.4%) women were not available at home when the interview team visited their households.

Of 1156 women who participated in the survey, 175 women had given birth to a child after the data collection by the enhanced surveillance system (i.e. after 1 June 2015). Of 175 births enumerated in the survey, 173 (99%) were captured by the enhanced surveillance system and only two (1%) births were missed by the enhanced surveillance system.

A total of ten neonatal deaths were recorded by the survey (recaptured). Of these, nine (90%) were matched with the data captured by the enhanced surveillance system (Table 1).

Table 1 Completeness of births and deaths reporting by the enhanced surveillance system (June to December 2015) using capture–recapture methods in Tehsil Havelian, District Abbottabad, Pakistan
Full size table

Five stillbirths were captured by the survey and all of them were matched with the stillbirths recorded by the enhanced surveillance system. A total of seven early neonatal deaths were captured by the survey and of these six (86%) were also captured by the enhanced surveillance system (Table 1).

The neonatal mortality and early neonatal mortality rates calculated by the enhanced surveillance system were 40 per 1000 live births (95% CI 35–44) and 30 per 1000 live births (95% CI 26–33) respectively (Table 2). After adjustment for estimated missed deaths, the neonatal and early neonatal mortality rates were 42 per 1000 live births (95% CI 37–46) and 32 per 1000 live births (95% CI 28–36) respectively (p > .05) (Table 2). A 5% underestimation in neonatal mortality and 7% in early neonatal mortality rates were found in these counts by the enhanced surveillance system.

Table 2 Adjusted annual mortality rates in the study area, Tehsil Havelian, District Abbottabad, Khyber Pakhtunkhwa province, June 2015–May 2016
Full size table

One maternal death was recorded in the cross-sectional survey that occurred since the start of the enhanced surveillance system and was matched with the maternal deaths captured by the enhanced surveillance system. The maternal mortality ratio prior to adjusting for the undercounts of the denominator was 247 per 100,000 live births (95% CI 147–391), whereas after adjusting it was 246 (95% CI 146–389) (Table 2).

Using the Petersen–Lincoln equation, an estimated 49 births, 49 live births, 17 early neonatal deaths, 21 perinatal deaths and 15 neonatal deaths were missed by the enhanced surveillance system from the study population. After calculating the completeness of the data captured by the enhanced surveillance system, a very high completeness of capturing births and deaths by the enhanced surveillance system in the study population was observed and measurements were plausible across various events (Table 1).

For MMR and NMR, the magnitude of the difference was 1%, and for the other mortality rates was 5–7%, as displayed by the mortality ratios (Table 2). The 95% CIs of rates or ratios overlap for all mortality indices, and are thus not statistically significantly different.

The enhanced surveillance system initially aimed to record births and deaths from the entire study population. However, only 96% of the study population could be covered for the entire duration of the study. Although the sampling frame of the cross-sectional survey (recapture) included the entire study population the matching percentage presented here indicates only births and deaths captured from the 96% of the population covered by the enhanced surveillance system.

Discussion

A capture–recapture method was used to ascertain the completeness of the maternal, perinatal and neonatal mortality surveillance system established in a rural district of Pakistan. The birth and death data captured by the enhanced surveillance system has high completeness for births, stillbirths, and maternal deaths. Neonatal death recording requires some improvement, although reporting appears around 90% complete. The enhanced surveillance system captured 99% of births, 100% of maternal deaths, 100% of stillbirths, 87% of early neonatal deaths and 91% of neonatal deaths in the study population. A small difference was observed in the mortality ratios (1–7%) following the capture–recapture correction. The inflated 95% CIs of rates or ratios overlap for all mortality indices.

All maternal deaths were reported by the enhanced surveillance system. This may be because the enhanced surveillance system captured deaths from all sources, including public and private health facilities, the health information system and from the community in the study areas. Further, each identified death was verified from the household followed by the application of verbal autopsy tools and assignment of a probable cause of death by the district death review committee, making it less likely that any maternal deaths were missed. Although stillbirths are more likely to be under-reported, the enhanced surveillance system captured all stillbirths from the study population. Concerning neonatal and early neonatal deaths, the enhanced surveillance system captured 91 and 87% deaths in the study population respectively. One possible reason for missing early neonatal and neonatal deaths by the enhanced surveillance system may be because of belief of Lady Health Workers and Community Health Workers that reporting a neonatal death in the area allocated to them may be considered as their poor performance.

Our study noted that only 1% of births were missed in the study area by the Lady Health Workers, which differs from the findings reported by an evaluation of the Lady Health Workers program conducted by population council in 2011 (Mahmood and Naz 2012). That evaluation was based on a random sample of 154 Lady Health Workers from four districts (Jhelum and Dera Ghazi Khan (Punjab), Mardan (Khyber Pakhtunkhwa) and Sukkur (Sindh) from three provinces of Pakistan). The data were collected from three different sources: Women in the community, Lady Health Workers and from Lady Health Supervisors. Household information was collected to validate the information collected by Lady Health Workers and data collected from Lady Health Supervisors were used to assess the accuracy of data transferred from records to reports, and the completeness of the reports. It was found that 73% of current pregnancies, 82% of births, and 94% of all deaths were correctly recorded by the Lady Health Workers in their assigned areas (Mahmood and Naz 2012) compared to routinely reported data. Concerning reporting rare events, the validation of Lady Health Workers reported data from the households noted 100% of the maternal deaths, 99% of early neonatal deaths and 98% of stillbirths were accurately reported. This means the evaluation of Lady Health Workers performed in 2011, used already reported data of births and deaths to validate the reports of Lady Health Workers yielded better accuracy, but may not reflect the true performance of Lady Health Workers in term of under-reporting of births and deaths at the community level. In our study, interviewers collected birth and death data directly from households, and noted that 90% of the neonatal deaths were reported at the community level. Our finding of under-reporting of neonatal mortality by the Lady Health Workers is further supported by a study of maternal mortality using community informants (religious leaders) and Lady Health Workers in a rural district of Punjab province (Pakistan) in 2013/2014. The study used two source capture–recapture methods to adjust for undercounts by either source and reported a 0.73 probability of capturing a maternal death by the Lady Health Workers. The adjusted maternal mortality in the study was estimated at 309 per 100,000 live births (95% CI 266–358) which is considerably higher than the maternal mortality of 178/100,000 live births estimated by international agencies for Pakistan (Mir et al. 2015).

The use of capture–recapture methods dates back to 1896 when it was first used to assess the population of fish in a pond by Peterson (International Working Group for Disease Monitoring and Forecasting 1995). In demography, the capture–recapture method was first used in India in 1949 to estimate the total number of births and deaths in an area and the extent of completeness of registration by Sekar and Deming (1949). Later, many studies employed capture–recapture methods to evaluate the completeness of population censuses (Himes and Clogg 1992), correcting undercounts, and adjusting disease rates for presumed under-enumeration of cases (International Working Group for Disease Monitoring and Forecasting 1995; Mony et al. 2015), to estimate the number of injecting drug users from two street intercept surveys in Sydney in 1994 (Duque-Portugal et al. 1994). A prospective cohort study conducted in India in 2011 used capture–recapture methods to estimate perinatal mortality at health facility level in 10 districts of Rajasthan. The study assessed the completeness of the routine facility-based health information system in reporting of stillbirths and early neonatal deaths, by collecting information on delivery and its outcomes in the health facilities followed by tracking of delivery and outcomes that occurred at health facilities through midwives. Data collected at the health facilities were matched with data collected over the phone and reported up to 14% under-estimation in perinatal mortality rates by the facility based reporting systems (Mony et al. 2015). Capture–recapture methods were used in Kenya to evaluate the completeness of birth and death registrations by the Demographic Surveillance System. Each household in the study area of Asembo and Gem were mapped in 1996/1997. The study areas were visited biannually to records migrations, births and deaths. Births and deaths occurred in Asembo area between March to September 2000 and in Gem area between January to July 2001. A cross-sectional survey of 5000 households in the study areas Asembo and Gem were conducted in November 2000 and August to September 2001 to collect data on births and deaths. The study reported 88% sensitivity in the recording of births and 61% sensitivity in the recording of neonatal deaths in the study area of Asembo, and 77% for births and 49% for neonatal deaths in the area of Gem (Eisele et al. 2003). In Pakistan, capture–recapture methods have been employed for estimation of deaths due to road traffic accidents in Karachi (Lateef 2010; Razzak and Luby 1998) to assess completeness of neonatal tetanus notifications from routine health information system in a rural district of Dadu (Lambo et al. 2011), for estimation of the disease burden due to tuberculosis in 10 randomly selected districts of all four provinces in 2012 (Fatima et al. 2014), and to estimate maternal mortality in the Punjab province (Mir et al. 2015). We could not find any published reports of studies that used capture–recapture methods to estimate the completeness of birth and death reporting particularly in rural and developing nations, nor to measure perinatal and neonatal mortality rates in Pakistan. This is thus the first study in Pakistan to use capture–recapture methods to assess the completeness of such enhanced surveillance system, and to adjust for estimated undercounts in maternal, perinatal and neonatal mortality rates.

There are four underlying assumptions of two sample capture–recapture methods (Ding and Fienberg 1996). (1) Both sources (of data) are independent of each other. “The same list of eligible women, prepared by the Lady Health Workers and Community Health Workers, was used to select a sample of households for the cross-sectional survey. However, the survey sample was randomly selected by computer from the women that were identified only by a code number, so dependence between the sample and the surveillance system is unlikely. Different interviewers were used for the sample survey and did not include the Lady Health Workers or community health workers who participated in the enhanced surveillance system. (2) Each member of the population has an equal chance of being captured by both systems. Each woman of reproductive age (18–49 years) had an equal chance of being captured by both sources because the enhanced surveillance system listed 100% (51,690) of the eligible women aged 18–49 years residing the study area and they remained under constant surveillance for the entire duration of the study through active household visits by the Lady Health workers and Community Health Workers. Whereas in terms of the chance of being captured by the survey, the same list of eligible women (51,690) residing in the study area was used to draw a random sample for the survey. (3) The population must be constant during the study period, that is, it must be a closed population with no one moving in or out. The survey found only 17 (1.4%) of the eligible women had migrated to a new place within the study area, and the surveillance data were gathered just prior to the survey, thus the population was constant through the study duration. (4) Individuals in both sources can be matched accurately (Ding and Fienberg 1996). The collection of sufficient variables common to both the enhanced surveillance system and the survey allowed the desired events of births and deaths to be matched accurately.

The Lady Health Workers Programme community coverage varies widely across 25 districts of the province of Khyber Pakhtunkhwa, from 79% in Abbottabad (the district containing the study area) to 2% in Kohistan district (UNDP 2011). For districts with low Lady Health Worker’s community coverage, application of capture–recapture methods to existing health information systems may be useful to estimate more accurate maternal and perinatal mortality rates and to adjust for any under-estimation.

Our study was limited by sample size due to inadequate resources. Further, the deaths captured by the survey could not be investigated to ascertain the reasons for being missed from the enhanced surveillance system. In the study area 80% of the population was covered by the Lady Health Workers (Government employees) who maintain a detailed population registers of their assigned population, including the number of married women, pregnancies, births and deaths. The remaining 20% of the population in the study area live in remote areas and were covered by recruiting additional community health workers. This population is most likely to be the poorest and most marginalized. Of this 20% of the population, 16% were included in the surveillance system, but coverage could not be maintained in the remaining 4% because of remoteness, isolation and ruggedness of the terrain, and the community health workers were withdrawn.

Conclusion

Capture–recapture assessment of the completeness of the enhanced surveillance system was ≥ 99% for all births, live births, stillbirths and maternal mortality. Early neonatal, all neonatal and perinatal deaths were 87–92% complete, and under-enumerated data were adjusted as required. The capture–recapture methodology is a useful tool to assess the completeness of existing surveillance systems such as the Lady Health Worker Programme and Mother and Newborn Child Health Program and for correcting any under-counts in births and deaths reporting in the population.