Antenatal care data sources and their policy and planning implications: a Palestinian example using the Lives Saved Tool

Setting effective and appropriate national, sub-national or sector-wide policies is a complex endeavor for health systems everywhere. Investigations of priority setting at national levels have demonstrated a high degree of similarity; critically, a unified understanding of the importance of the health problem is vital [1, 2]. A common complaint among policy makers is the inability to trust the evidence and data, especially when international and local numbers differ [3]. As a result, consistent sources of high quality and trustworthy data, tailored to the local context to inform planning processes, have proven to be a clear gap [4].

High quality data can be used at different points in the policy planning cycle, including for informing discussions as well as projecting the impacts of potential decisions, both of which are commonplace activities. The Lives Saved Tool (LiST) is a policy planning tool which utilizes information on the current health status of a country to project the health (mortality) implications of implementing specific health interventions for women or children [5]. LiST has been used for over ten years for evaluation, advocacy and strategic planning [6], across a wide variety of settings [7, 8]. An unsurprising criticism of LiST is the quality of data available to populate it [9] – in many instances, significant assumptions and estimations are required given the lack of primary data [10]. For any modelling tool, as for any policy setting process, high quality data is required to ensure that the results are accurate enough for usability [9].

LiST requires health status indicators (such as mortality and morbidity), effectiveness data (impact of interventions on health status), and coverage indicators (levels of utilization of health interventions). The coverage indicators required to populate LiST come from a variety of sources, including national statistics, household surveys, facility surveys and research studies, and are less amenable to global evaluation and summarizing due to variability in the implementation of many of these interventions. Few countries have routine high quality data on effective coverage (proportion of those getting an intervention among those in need) for assessing all aspects of their health system within the LiST structure. The frequency and quality of routinely reported data from health systems vary by topic and country, leaving alternative sources of data necessary. Many countries rely on externally funded, population-based surveys such as UNICEF’s Multiple Indicator Cluster Survey (MICS) [11] and the Demographic and Health Surveys (DHS) [12] to collect service related data by asking women to remember the care received during their most recent pregnancy [13], often up to 2–5 years in the past.

The ever-expanding arena of information technology and digital registries has the potential to improve data availability around interventions delivered during antenatal care, childbirth and the postpartum period [13]. ‘eRegistries’ are electronic registries used at the point of care for recording health services delivered [14]. They are specifically designed to facilitate implementation of several digital health interventions such as: decision support tools, and audit and feedback (to aid health care workers in providing quality care); tailored behavior change communication text messages (to encourage women to attend care); and reporting (to provide aggregate data for health system managers and policy makers). An eRegistry for antenatal, postpartum and newborn care has been rolled out in primary health care clinics in the public sector in Palestine as part of a national implementation [15].

The validity of LiST outputs and results is closely linked to the kind of data that is input [5]. However, few studies have assessed the nature and magnitude of consequences to LiST results when using different sources of data. Users of LiST should be aware of such consequences to make informed decisions about intervention effectiveness when considering scale-up. Our objective was to model the scale up of antenatal care interventions in LiST, using all available data sources in Palestine – routine data, survey results, extracted medical records and the eRegistry, to explore how the results might vary, and the implications of using these varied sources to make decisions.

Data for decision-making is a common cry in public arenas. However, not all data are the same, and the implications of using the various alternative data sources available can be significant, especially when multiple choices exist, such as in the case of Palestine. Selection of data source can be even more critical when used as inputs into a formal analytic framework, as many policy makers do not see the raw data but only the results of the processing, assumptions, estimates, and analysis. Global agencies and research teams publish consensus estimates of mortality with uncertainty bounds, but estimates of health intervention coverage show more variability and are less widely available in general. The availability of new sources of local and timely data and indicators is likely to increase as countries shift towards digital data and case-based collection methods. The evaluation of these new data sources is critical to assess their potential for improving the care being delivered and to appropriately inform planning processes.

In this analysis, the four data sources yielded notably different results when utilized in LiST. The maternal deaths averted ranged from 5 to 39, or a reduction of maternal mortality from 1 to 6%. At the same time, the composition of interventions to save these lives varied from 100% for pre-eclampsia management to 75% for hypertensive disorders management. These differences would likely result in different policy and practice decisions being taken. Similar, but less dramatic differences could be seen in newborn, stillbirth, and anemia results using the different data sources. Although the absolute differences were relatively small in this particular context, they would be magnified greatly in countries and settings with higher mortality and morbidity rates, or if interventions beyond antenatal care were included.

The power of the Lives Saved Tool can be maximized when data of better quality and quantity are available to populate it. However, in most country settings, several data points are not directly available in either routine reports or household surveys. Drawing data directly from clinical records allows for a more complete and complex picture of antenatal care and covers almost all data needs. Many surveys, such as the MICS, only include data from live-births [17], thus excluding data on women who experienced stillbirths or miscarriages and their potentially complicated pregnancies. Another aspect of clinical data, not present in most survey or routine data sources, is the longitudinal perspective within a pregnancy. Longitudinal analyses across periods of time and healthcare contacts allow the ability to include only managements based upon true conditions, ensuring that only appropriate and correct referrals are included in the calculation rather than all referrals. An ideal data source for complex indicators would be longitudinally collected at the point-of-care to minimize the need for post data-collection processing. This would ensure that both numerators and denominators were collected simultaneously, and mitigate issues from recall bias of either care providers or mothers. In addition, one of the largest criticisms of the Lives Saved Tool is the quality of estimates around maternal mortality. The current use of indirect estimates greatly increases the likely uncertainty around the LiST estimates of maternal mortality. These results should increase the validity and reliability of future analysis with such data, simply because fewer assumptions will be needed.

The paper-based routine health information systems in Palestine, as in many other places, rely on care providers identifying key characteristics about patients and reporting to district and national health authorities, who aggregate and process the data to generate national indicators. The validity of any individual diagnosis is unknown. This additional reporting burden on care providers limits the ability to demand reporting of a comprehensive set of clinical data, and thus results in a reporting system focused on only the highest priority indicators. Complex health conditions and reporting chains can lead to either over- or under-reporting. For example, knowing the number of women referred for diabetes is useful, but does not indicate the proportion of women correctly diagnosed with diabetes or appropriately referred, leaving the system unable to rectify underlying problems. To create more actionable indicators, providers would need to document every diabetes test, the number of women positive and the number referred according to recommended guidelines. This extensive task is not likely to be a valuable use of time in a paper-based system. The routine system in Palestine also relies on reporting by two different levels of clinics (primary and referral), which makes it difficult to ensure that women are correctly included only once, in either the numerator or denominator, potentially leading to biases. Routine reporting data should be limited and focused on critical indicators that cannot be collected easily in clinical data sources or those needed to triangulate with other sources.

Typically, the primary source of coverage data used in LiST is household survey data, such as the MICS presented here. However, very little data are available from these surveys to directly populate antenatal care (or childbirth care) indicators. Information on antenatal services received or antenatal care attendance can be used to indirectly calculate several other indicators. However, these estimates are dependent on maternal recall, which may be biased towards experiences of women with difficult pregnancies who would tend to remember care more completely relative to uneventful pregnancies and deliveries, while excluding pregnancies ending in stillbirth or miscarriage. Although these indirect indicators (together with non-antenatal care indicators) can be useful for planning, these surveys are typically conducted only every five years making their input less timely for shorter-term planning or course-correction. Additional questions should be asked about the utility of these indirect estimates (which were formulated with sub-Saharan African data) when compared to actual values extracted from antenatal care records. If the Kanyangarara formula is applied to the paper-based antenatal records and the eRegistry, respectively, approximately 62 and 61% of women are estimated to be correctly managed for hypertension while the clinical data indicated that only 7 and 10% were correctly managed. The differences were much smaller for the diabetes management indicator which were predicted to be 29 and 31% respectively, while the actual clinical values were 13 and 35%, respectively.

Data extracted from paper-based antenatal care records and the eRegistry contained the greatest quantity of data for direct analysis. They also allowed for computing indicators that most closely matched the ideals of the Lives Saved Tool (Table 3). Although differences in documentation may account for the different values reported, it should also be noted that indicators from the eRegistry document more carefully the details around management, which are not typically recorded in the paper records, and thus should theoretically be a more precise indicator of correct management. The simplified single checkbox of any iron-folate supplementation in paper records may have over-estimated current performance as the LiST analysis estimated more than two-fold higher numbers of anemia cases being averted in the eRegistry-based analysis compared with paper records. Assuming that care providers are correctly completing their documentation, these results should be more valid and more reliable than survey based data or routine reporting with the multiple additional layers of data processing required. They are certainly more direct estimates that have the potential to be more representative of facility care since they also include all pregnancies, not just all live-births.

Extracting data from paper-based records on a regular basis is neither feasible nor sustainable for routine monitoring due to the expense and tardiness of such a system, and without the quality assurance routines used in this study, also by the likelihood of transcription errors. In addition, paper records can be incomplete and do not have built-in validations at data entry, as seen with the tetanus toxoid vaccination data.

Although the development of an eRegistry is time-consuming and resource-intensive, and up-front implementation costs are relatively high, the benefits can be wide-ranging by integrating multiple digital health interventions in a single system. In Palestine, the point-of-care data entry currently serves as an interactive checklist with clinical decision support, with integrated audit and feedback components and a reminder system for pregnant women. On the back-end, the system routinely generates key indicators at national, sub-national and clinic levels without requiring burdensome reporting.

A limitation of the eRegistry system in Palestine is that it is currently only available in public sector facilities and does not include private or non-governmental organization (NGO) facilities, nor public hospitals. Population coverage cannot be measured with the eRegistry data in this setting. Although the lack of data from private and NGO facilities does not affect the analysis of care delivered at public facilities, LiST analysis might predict larger health improvements than actually could occur, due to missing data on referred patients who seek care in external facilities. At the same time, population based surveys can provide the data needed to understand the flow of patients between public and private or NGO sectors and thus act as a calibration of the clinical data, in conjunction with routine reported data. The lack of data from any of the hospitals also limited the ability to define the interventions in terms of full quality of care at referral centers. However, it is likely that adding this information would only decrease the proportion of women correctly managed.

This study has clearly demonstrated the notable variability of information available for decision making based on the data source chosen in Palestine. Selection of the most complete and appropriate data source for policy and planning is critical. Many frameworks have been developed that attempt to characterize the features of priority setting and networks for informing policy decisions [21–24]. Studies have evaluated the barriers and facilitators to evidence-based decision making at national and local levels, and systematic reviews have described systems for incorporating research evidence into decision-making and less frequently, described the utilization of burden of disease data in decision-making [25, 26]. This paper feeds the discussion around how to support evidence informed decision-making at national levels by outlining the pros and cons of various data sources. We demonstrated the significant data driven benefits for the health system from utilizing data automatically extracted from a digital registry of health contacts – both in terms of quantity and quality. These distal benefits of an eRegistry along with more immediate clinical benefits to care providers and clients can also be used to inform a cost-benefit analysis for implementing complex health system interventions.