Health system barriers to strengthening vaccine-preventable disease surveillance and response in the context of decentralization: evidence from Georgia
- David R Hotchkiss†1Email author,
- Thomas P Eisele†1,
- Mamuka Djibuti2,
- Eva A Silvestre1 and
- Natia Rukhadze2
© Hotchkiss et al; licensee BioMed Central Ltd. 2006
Received: 07 April 2006
Accepted: 05 July 2006
Published: 05 July 2006
A critical challenge in the health sector in developing countries is to ensure the quality and effectiveness of surveillance and public health response in an environment of decentralization. In Georgia, a country where there has been extensive decentralization of public health responsibilities over the last decade, an intervention was recently piloted to strengthen district-level local vaccine-preventable disease surveillance and response activities through improved capacity to analyze and use routinely collected data. The purpose of the study is 1) to assess the effectiveness of the intervention on motivation and perceived capacity to analyze and use information at the district-level, and 2) to assess the role that individual- and system-level factors play in influencing the effectiveness of the intervention.
A pre-post quasi-experimental research design is used for the quantitative evaluation. Data come from a baseline and two follow-up surveys of district-level health staff in 12 intervention and 3 control Center of Public Health (CPH) offices. These data were supplemented by record reviews in CPH offices as well as focus group discussions among CPH and health facility staff.
The results of the study suggest that a number of expected improvements in perceived data availability and analysis occurred following the implementation of the intervention package, and that these improvements in analysis could be attributable to the intervention package. However, the study results also suggest that there exist several health systems barriers that constrained the effectiveness of the intervention in influencing the availability of data, analysis and response.
To strengthen surveillance and response systems in Georgia, as well as in other countries, donor, governments, and other stakeholders should consider how health systems factors influence investments to improve the availability of data, analysis, and response. Linking the intervention to broader health sector reforms in management processes and organizational culture will be critical to ensure that efforts designed to promote evidence-based decision-making are successful, especially as they are scaled up to the national level.
There is broad consensus among health professionals that health information systems (HIS) should play an integral part of any health system [1–3]. Health information is essential in determining whether a health system is effective in detecting health problems, defining priorities, identifying solutions, and allocating resources to improve health outcomes [4, 5]. With the formation of the Health Metrics Network, the Routine Health Information Network (RHINO) and the recently-established Ellison Institute, the issue of strengthening HIS in low- and middle-income countries has received an unprecedented amount of attention in recent years.
One essential source of health-related information is surveillance and response systems focused on epidemic and vaccine-preventable disease (VPD). VPD surveillance systems have the potential to improve the efficiency and effectiveness of VPD prevention and control programs by targeting interventions and documenting their effect on the population. The performance of surveillance and response systems depends not only on the availability and quality of health information, but also on the demand for information by health policy makers and managers. Two key dimensions of demand are the desire to analyze information and the desire to use of information for responses or actions. In a decentralized context, the process of analysis and response should be driven by the local-level surveillance and response program manager's desire to use data to improve decision-making. In order for this to occur, surveillance information must be perceived as useful for decision-making, and expectations for analysis, interpretation, and translation into action must be clearly laid out.
Effectively conducting VPD surveillance and response is particularly challenging in countries that have introduced health sector decentralization reforms [6, 7]. There are a number of reasons why decentralization might have substantial implications for HIS. First, surveillance systems may not perform well in a decentralized context if local epidemiological health staff lack the capacity to carry out devolved surveillance and response functions (i.e. the collection, analysis, and actual use of information for public health responses) previously carried out by national level staff and if there is low demand for health information at the local government level. Second, demand for information may be limited if surveillance systems designed for centralized systems are unaligned with the needs of local users. This mismatch between the availability of information and user needs can be further exacerbated if subsequent efforts to reform the surveillance system are "data led" rather than "action led" . Third, factors that operate at the health system level, such as limited local resources, low health worker motivation due to a lack of incentives, inadequate transportation and communication, and weak accountability, also play greater roles in limiting system performance. Many of these concerns have led some experts to conclude that "a critical challenge that is faced in the health sector in developing countries is to ensure the quality and effectiveness of public health surveillance and response in an environment of decentralization" .
Most previous research that evaluates public health surveillance systems focus on the assessment of one or two major attributes of surveillance systems such as sensitivity and timeliness . However, there is relatively little research that investigates the effectiveness of strategies to strengthen the capacity and motivation of health workers to carry out public health surveillance and response within a decentralized country context. In fact, in an article on public health surveillance that appeared in the World Bank's recently published Disease Control Priorities in Developing Countries (Second Edition), Nsubuga et al state that the most important research question for surveillance that needs to be addressed is how to go about developing and maintaining "a cadre of competent, motivated surveillance and response workers in developing countries" . Given the increased concern regarding the consequences of decentralization on public health surveillance and response activities, attempts to answer this question in countries where surveillance and response functions have been devolved are of particular relevance.
The purpose of this paper is help fill this research gap by 1) assessing how health worker motivation, perceived capacity, and actual use of information for decision making of district-level epidemiological staff in Georgia have been influenced by a pilot VPD surveillance and response intervention, and 2) assessing the remaining barriers to improving the use of data for programmatic and epidemiological responses. Georgia is a middle-income country where there has been extensive decentralization of public health responsibilities over the last decade. Aimed to strengthen local VPD surveillance and response activities through improved analysis and use of routinely collected data, the intervention focused on clarifying roles and responsibilities at the rayon (district) level for analysis and outlining links to actions, by improving capacity, and by ensuring the availability of resources necessary for the non-personnel costs of disease outbreak investigation and selected monitoring functions. All of these efforts were intended to improve the value that health workers place on VPD surveillance information, resulting in improved capacity and motivation to analyze and use information for epidemiological responses, program management, and resource allocation decisions.
The intervention evaluated in this study was implemented in the region of Imereti with the aim of improving the analysis of routinely-collected VPD surveillance data and resultant responses. In terms of population size, Imereti is the largest region in Georgia, making up 16% of the total population of 4.37 million in 2002 .
The effectiveness of the intervention on perceived capacity and motivation was evaluated through the use of a longitudinal pre-post quasi-experimental research design, supplemented where possible with data from surveillance system record reviews and focus group discussions. A cohort of individuals responsible for VPD data analysis and response within CPH offices within all 12 rayons of the Imereti Region served as the intervention group. Individuals within CPH offices within three rayons outside the intervention area were selected to serve as controls to help validate any resultant changes in analysis and response within the intervention group. Because randomization was not feasible, the three control CPH offices (Senaki, Chokhatauri, and Tbilisi) were chosen for their similarity to the intervention rayons in surveillance motivation and performance (based on expert experience), location (mountainous vs. flat) and population composition (by age and sex).
Outcomes pertaining to analysis and response within intervention and control groups were measured among the same individuals at baseline and again one and half years later at follow-up after the implementation of the intervention package (Figure 2).
The Institutional Review Board of Tulane University's Health Sciences Center approved the protocol of the study. All participants gave informed written consent.
Outcome measures were grouped into five topic areas that were hypothesized to improve as a result of the intervention: 1) availability of quality data from subordinate health facilities; 2) capacity to perform routine data analysis; 3) motivation to perform data analysis; 4) value of using analyzed data for decision-making; and 5) motivation to use analyzed data for decision-making. Likert scale questions at five levels, ranging from strongly disagree to strongly agree, were used to ascertain information for constructing outcome indicators. All outcomes are measured and analyzed at the individual level. The decision to focus on perceptions and motivation of rayon and health facility staff was based on the premise that improvements in self-efficacy and the perceived value information are necessary to improve how surveillance data are used to improve the performance of the vaccine preventable disease surveillance and response program.
Composite Indices for measuring underlying constructs of attitudes towards analysis and use of vaccine-preventable disease (VPD) data
Composite Index (Outcome indicator)
Cronbach coefficient alpha (raw)
Subordinate health facilities and labs report their surveillance data in a timely manner.
Reports submitted by subordinate health facilities are fully completed
Perceptions of availability of quality VPD data
I have confidence that the surveillance data reported by subordinate health facilities are accurate
I possess sufficient skills to analyze and interpret surveillance data
I feel fully capable of carrying out analysis of surveillance data
Perceived capability to perform analysis of VPD data
Epidemiological data are essential for providing effective surveillance of vaccine-preventable diseases in my rayon.
Data from subordinate health facilities must be analyzed in order to be useful.
I place great importance on providing feedback to subordinate health facilities based on the data that I routinely analyze.
Perceived value of using analyzed VPD data for decision-making
Analysis of surveillance data is useful because it provides a basis for decision-making.
It is important that decisions regarding prevention and control of infectious diseases be based on solid evidence.
Sample size and data collection
Outcome measures were ascertained using a self-administered survey questionnaire to a cohort of public health professionals responsible for analysis and use of VPD data at CPH offices within intervention and control rayons at baseline and again at follow-up. The questionnaire consisted of Likert scale, yes/no and open-ended questions. The questionnaire was pre-tested and revised prior to baseline data collection. Trained data collectors were not blinded to intervention group status.
A census of all individuals responsible for analysis and response within intervention and control CPH offices was recruited to participate in the study. In total, 31 individuals responsible for analysis and use of VPD data from within the 12 intervention rayons were identified and completed the questionnaire at baseline. All of these individuals, plus an additional 4 who joined or were promoted within the same CPH offices, completed the questionnaire at follow-up post-intervention. Eleven individuals responsible for analysis and use of VPD data from within the three control rayons were identified and completed the questionnaire. Ten of these same individuals completed the questionnaire at follow-up. A total sample size of 87 observations was obtained over both rounds among both intervention groups.
The survey was complemented by a review of CPH records and reports at two points in time after the initiation of the intervention to ascertain whether the intervention package was implemented successfully, and was functioning as intended. The record reviews were conducted within both intervention and control rayons in May 2004 and April 2005 (Figure 2) in order to a) to assess completeness and accuracy of the records in the workbooks distributed to rayon epidemiologists in the program intervention areas; and b) to determine the current level of data analysis and use in other rayons beyond the program pilot area.
Record reviews were using special instruments developed by research team members. Respondents were those professionals who are directly responsible for maintaining all records and conducting the analyses of surveillance data. In addition, all worksheets that had been completed were photocopied for further detailed analysis and review. The worksheet's data aggregation section was considered as accurate if, through a random check, data in the workbook corresponded to those in the primary data sources and there were no mathematical mistakes. The worksheet's logical conclusions section was considered as accurate if analytical conclusions (e.g., which units performed poorly, causes of low coverage, reasons why cases occurred, who belongs to risk groups, barriers to performance) were made based on a complete set of data and logically reflected these data.
In addition, focus group discussions within intervention rayons were conducted post-intervention at two points in May 2004 and April 2005. The following groups of staff members from intervention CPH offices and health care facilities were included in the focus group discussions:
For each group, two focus group discussions were held at each of the two time periods, and the size of the group ranged from five to seven individuals. Guides were developed separately for CPH staff and providers. Participants were mostly the same for the two rounds. The length of the discussion sessions averaged between 2 and 2.5 hours for CPH staff and between 1 and 1.5 hours for Polyclinic staff.
Two researchers conducted each focus group discussion: a moderator who led the discussion and a facilitator who handled all logistics and took notes. The facilitator recorded the personal characteristics of the members making up the discussion, the time, duration, and location. As far as possible, the discussions took place in a setting where the session was not interrupted and people felt that they could voice their opinions freely. Each of the focus group discussions were audio taped and transcribed. The research team analyzed the data using domain analysis, which seeks to create a systematic understanding of a practice by describing and analyzing individual's perceptions, attitudes, and experiences. The research team created a coding scheme using broad categories to organize the data, such as the availability of data, the analysis of data, and the use of analyzed data to carry out public health actions. Using these predefined codes, information was organized and displayed. Notes and selected quotations were translated into English.
All statistical analyses were conducted using SAS 9.1. The intervention group × survey round interaction term in the following model was used to assess the effect of the intervention package on selected outcomes:
Y = β0 + β1(intervention group) + β2(subject) + β3(survey round) + β 4 (intervention group*survey round) + β6(sex) + β7(age) + β8(years of experience) + β9(rayon) + e.
The intervention group × survey round interaction term can be interpreted as the relative change in outcome indicators between comparison groups from the baseline to follow-up round. Age, sex and years of experience are included to control for individual-level confounders. Rayon is included to control for a number of community-level differences that may exist between rayons, hypothesized to include such potential confounders as governmental funding, access to health care and differences in VPD rates. The analysis of individual Likert scale outcomes was modeled using the generalized linear model (GLM) for multinomial outcomes. Composite index outcomes were analyzed using linear regression in GLM with the generalized estimating equation (GEE) used to account for correlations between subjects with repeated measures. Differences between intervention groups for bivariate outcomes were assessed using Chi-square. For all outcomes, a two-sided P-value < 0.05 was considered statistically significant.
Results of regression analyses assessing the impact of the job aid intervention on the five primary outcome indicators for analysis and use of vaccine-preventable disease data
Mean values from Likert scale
n = 87 observations
Interaction coefficient †
1. Perceptions of availability of quality VPD data
2. Perceived capability to perform analysis of VPD data
3. Likert scale question measuring motivation to carry out analysis
4. Perceived value of using analyzed VPD data for decision-making
5. Likert scale question measuring perceived motivation to use surveillance data
Quantitative evaluation results
Following the largely ambivalent responses to these three questions on the availability of quality VPD data, the mean score for this index increased only modestly by 0.53 between the baseline and follow-up among the intervention group, as compared to essentially no change among the control group (Table 2). Accordingly, the effect of the intervention package was not shown to be statistically significant (P-value = 0.1664) on the perception of the availability of quality surveillance data.
However, the effect of the intervention package was found to have significantly improved self-perceived capacity to perform data analysis (P-value = 0.0003), as the mean score for this index increased by 0.50 among the intervention group while decreasing by 0.70 among the control group pre and post-test (Table 2). The intervention package also had a marginally significant impact (P = 0.0579) on the reported motivation to analyze VPD data, as the mean Likert scale responses in the intervention group increased by 0.41 while the mean responses decreased by 0.59 in the control group pre and post-test.
Results from the record review also found improvements in the analysis of data, but suggest that analytical accuracy varied widely across the intervention rayons. For example, the proportion of rayons that produced accurately completed worksheets, as verified through comparison with original source data, ranged from 8.3% on some worksheets to 83.3% on others. While these statistics stand in contrast to survey results reported above on the perceived capacity and motivation to perform analysis, the mean completion rate of workbook sheets increased from 49.2% (Standard deviation (SD) = 13.3) in 2003 to 77.5% (SD = 11.14) in 2004, while the mean completion rate of workbook sheets considered accurate increased from 29.1% (SD = 12.58) in 2003 to 50.0% (SD = 21.52) in 2004 (2003 data not shown, calculated in 2003 since the introduction of the intervention from October to December 2003). No documented evidence of analysis of VPD data was found at the three control rayons during the record reviews. While the actual use of the workbook varied widely across rayons, most rayons (83.3%) did produce evidence that the analysis of surveillance data was used to produce statistical reports, while no such evidence was produced in the three control rayons.
While point estimates and corresponding coefficients of the intervention group × survey round interaction term were in hypothesized directions, the intervention package did not have a significant impact on the perceived value of using analyzed VPD data or the motivation to use such data for decision making.
Qualitative evaluation results
The results of the focus group discussions of CPH and health care facility staff point to a number of improvements as a result of the intervention. These included: improved knowledge about current regulations at both the local and health facility levels; clarification of the roles and responsibilities of staff at regional, rayon, and health facility levels; an increased sense of job responsibility regarding roles in the surveillance system; and increased ability of CPH staff to carry out critical surveillance functions.
Despite these improvements, the qualitative component of the research highlights several potential barriers that prevent the use of the intervention tools to their fullest extent [see additional file 1]. The barriers mentioned include: an insufficient availability of quality surveillance data from subordinate health facilities; the unavailability of phones and electricity in health facilities and CPH offices; low levels of health care utilization; poor reporting of data from some private providers; a common perception that evidenced-based recommendations for public health actions will likely not be adopted by those at higher levels; the lack of CPH authority to impose penalties on low-performing health facilities; and limitations of government resources to carry out surveillance and response.
The results of the study suggest that the intervention package, which included guidelines, the job aid workbook, training, and a financial reimbursement system, were implemented as planned and led to expected improvements in the availability and analysis of surveillance data but not in the use of data for public health response, as assessed through the use of a pre-post quasi-experimental design. As suggested by the results of the focus group discussions with district and health facility staff, external factors, particularly those that operate at the health systems level, played an important role in limiting the effectiveness of the intervention in strengthening the availability and use of data for programmatic and epidemiological responses at the local level.
That the intervention did not result in substantial improvements in the use of data for public health responses during the course of the intervention most likely stems from the problem of weak accountability relationships within Georgia's health system. For example, the process of decentralization that begin in the mid-1990's resulted in fragmentation between health care facilities and CPH offices and a lack of clarity about which level bears responsibility for some key surveillance and response functions, such as outbreak investigations. In order to improve the effectiveness of surveillance and response and other HIS interventions, it will be critical to link the intervention to reforms in management processes and organization culture . In particular, it will be important to assess the various accountability roles that actors in the surveillance and response system play, and to develop accountability-strengthening strategies to help ensure the maximum effectiveness of the intervention in promoting evidence-based decision-making . This is critical given that the government has proceeded to scale up the intervention to the rest of Georgia.
Health systems barriers are also likely to play a critical role in surveillance and response systems and in other types of HIS interventions, both in Georgia and in other countries [1, 2, 6–8, 13–16]. In order to strengthen HIS, governments, international agencies and donors should consider how health systems factors influence the effectiveness of HIS investments not only on the availability and quality of information, but also on the value of the information to those who are expected to use it.
The results of this study should be treated with caution for several reasons. First, despite the census nature of the district-level data collected, limitations in sample size greatly limited the statistical power of the analysis which allowed the detection of only a 20% or greater change in the outcome between treatment groups across study rounds, assuming a two-sided test with alpha set at 5%, the baseline proportion of self-reported motivation to use and analyze data set at 60% (agree-strongly agree), repeated-measure design and 80% power. Secondly, because the intervention package was implemented within the Imereti region as a full coverage program, rayons were not randomly assigned to intervention and control groups, which limit the internal validity of the study design. Third, despite the self-administered questionnaire format, social desirability bias was still possible. This may explain why no statistical impact was detected for the indicator measuring self-reported motivation to use analyzed surveillance data, which was very high at baseline within both the intervention and control groups, and continued to be high in both follow-up survey rounds. In this instance, it is likely that the results were biased towards the null hypothesis of no change. Fourth, this study design prevented us from assessing to what extent each intervention component contributed to improvements in analysis and response. Thus, lack of changes, or detected changes, in various aspects of analysis and response may have been due to limitations in the measurement tools used.
Our findings regarding the health system constraints to strengthening public health surveillance and response functions are consistent with findings of previous researchers who have investigated the effectiveness of strategies to improve the use of data for decision-making in developing countries. For example, in a study of the effectiveness of a multi-faceted strategy supported by USAID and the US Centers for Disease Control and Prevention to improve capacity to use data for decision making at the local-, district-, regional-, and national-levels in four counties with decentralized health system (Bolivia, Cameroon, Mexico, and the Philippines), Pappaioanou et al found that although substantial improvements in data use were achieved, the sustained use of evidence-based public health in the long term requires "the creation of a data-use culture and a behavior change in those involved with the decision-making environment" . A review of public health surveillance and response strategies also finds that health system factors including many identified in our study present substantial challenges in low- and middle-income countries, and that building sustainable programs requires long-term, sustained efforts .
The logical implications of such findings are enormous, as they suggest that the process of building effective surveillance and response systems as well as other types of HIS may best be carried out within the context of a broader and long-term health sector reform process that involves not only clarifying health sector priorities, refining policies, and consensus-building in a decentralized context, but also reforming and restructuring institutions through which surveillance and response activities are implemented.
To strengthen surveillance and response systems in Georgia, as well as in other countries, donor, governments, and other stakeholders should consider how organizational and health systems factors influence investments to improve the availability of data, analysis, and response. Linking the surveillance and response interventions to broader reforms in management process and organizational culture will be critical to ensure that efforts designed to promote evidence-based decision-making are successful, especially as they are scaled up to the national level.
The authors wish to thank Lynne Franco, Anton Luchitsky, and George Gotsadze for providing comments on the research protocol and a previous draft. We also wish to thank the following individuals in Georgia for the valuable support they provided: Levan Jugeli of the Ministry of Labor, Health, and Social Affairs; Levan Baramidze of the Public Health Department; Paata Imnadze of the National Center for Disease Control; and Kote Gvetadze of the Kutaisi Regional Center for Public Health. Finally, we wish to thank the US Agency for International Development for funding the study (Contract/Project No. HRN-C-00-00-00019-00).
- AbouZahr C, Boerma T: Health information systems: the foundations of public health. Bull World Health Organ. 2005, 83 (8): 578-583.PubMedPubMed CentralGoogle Scholar
- Sauerborn R, Lippeveld T: Introduction. Design and Implementation of Health Information Systems. Edited by: Lippeveld T, Sauerborn R, Bodart C. 2000, Geneva , World Health OrganizationGoogle Scholar
- Evans T, Stansfield S: Health information in the new millennium: a gathering storm?. Bull World Health Organ. 2003, 81 (12): 856-PubMedGoogle Scholar
- Stumbling around in the dark. Lancet. 2005, 365 (9476): 1983-10.1016/S0140-6736(05)66671-8.
- Stansfield S: Structuring information and incentives to improve health. Bull World Health Organ. 2005, 83 (8): 562-PubMedPubMed CentralGoogle Scholar
- White ME, McDonnell SM: Public Health Surveillance in Low- and Middle-Income Countries. Principles and Practices of Public Health Surveillance. Edited by: Teutsch SM, Churchill RE. 2000, New York , Oxford University PressGoogle Scholar
- Nsubuga P, White ME, Thacker SB, Anderson MA, Blount SB, Broom CV, Chiller TM, Espitia V, Imtiaz R, Sosin D, Stroup DF, Tauxe RV, Vijayaraghavan M, Trostle M: Public Health Surveillance: A Tool for Targeting and Monitoring Interventions. Disease Control Priorities in Developing Countries. Edited by: Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P. 2006, Washington D.C. , The World Bank, SecondGoogle Scholar
- Sandiford P, Annett H, Cibulskis R: What can information systems do for primary health care? An international perspective. Soc Sci Med. 1992, 34 (10): 1077-1087. 10.1016/0277-9536(92)90281-T.View ArticlePubMedGoogle Scholar
- Romaguera RA, German RR, Klaucke DN: Evaluating Public Health Surveillance. Principles and Practices of Public Health Surveillance. Edited by: Teutsch SM, Churchill RE. 2000, New York , Oxford University PressGoogle Scholar
- State Department for Statistics in Georgia: Population of Georgia: Statistical Abstract. 2002, TbilisiGoogle Scholar
- Stansfield S, Walsh J, Prata N, Evans T: Information to Improve Decision Making for Health. Disease Control Priorities in Developing Countries. Edited by: Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P. 2006, New York , New York: Oxford University PressGoogle Scholar
- Brinkerhoff DW: Accountability and health systems: toward conceptual clarity and policy relevance. Health Policy Plan. 2004, 19 (6): 371-379. 10.1093/heapol/czh052.View ArticlePubMedGoogle Scholar
- Shibuya K, Scheele S, Boerma T: Health statistics: time to get serious. Bull World Health Organ. 2005, 83 (10): 722-PubMedPubMed CentralGoogle Scholar
- WHO: Issues in health information: disease surveillance. 2006, Geneva , Health Metrics NetworkGoogle Scholar
- WHO: Issues in health information: national and subnational health information systems . 2006, Geneva , Health Metrics NetworkGoogle Scholar
- LaFond A, Fields R: The prism: introducing an analytical framework for understanding performance of routine health information in developing countries: South Africa.2003.Google Scholar
- Pappaioanou M, Malison M, Wilkins K, Otto B, Goodman RA, Churchill RE, White M, Thacker SB: Strengthening capacity in developing countries for evidence-based public health: the data for decision-making project. Soc Sci Med. 2003, 57 (10): 1925-1937. 10.1016/S0277-9536(03)00058-3.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/6/175/prepub