- Research article
- Open Access
Using a monitoring and evaluation framework to improve study efficiency and quality during a prospective cohort study in infants receiving rotavirus vaccination in El Alto, Bolivia: the Infant Nutrition, Inflammation, and Diarrheal Illness (NIDI) study
BMC Public Health volume 17, Article number: 911 (2017)
Implementing rigorous epidemiologic studies in low-resource settings involves challenges in participant recruitment and follow-up (e.g., mobile populations, distrust), biological sample collection (e.g., cold-chain, laboratory equipment scarcity) and data collection (e.g., literacy, staff training, and infrastructure). This article describes the use of a monitoring and evaluation (M&E) framework to improve study efficiency and quality during participant engagement, and biological sample and data collection in a longitudinal cohort study of Bolivian infants.
The study occurred between 2013 and 2015 in El Alto, Bolivia, a high-altitude, urban, low-resource community. The study’s M&E framework included indicators for participant engagement (e.g., recruitment, retention, safety), biological sample (e.g., stool and blood), and data (e.g., anthropometry, questionnaires) collection and quality. Monitoring indicators were measured regularly throughout the study and used for course correction, communication, and staff retraining.
Participant engagement indicators suggested that enrollment objectives were met (461 infants), but 15% loss-to-follow-up resulted in only 364 infants completing the study. Over the course of the study, there were four study-related adverse events (minor swelling and bruising related to a blood draw) and five severe adverse events (infant deaths) not related to study participation. Biological sample indicators demonstrated two blood samples collected from 95% (333 of 350 required) infants and stool collected for 61% of reported infant diarrhea episodes. Anthropometry data quality indicators were extremely high (median SDs for weight-for-length, length-for-age and weight-for-age z-scores 1.01, 0.98, and 1.03, respectively), likely due to extensive training, standardization, and monitoring efforts.
Conducting human subjects research studies in low-resource settings often presents unique logistical difficulties, and collecting high-quality data is often a challenge. Investing in comprehensive M&E is important to improve participant recruitment, retention and safety, and sample and data quality. The M&E framework from this study can be applied to other longitudinal studies.
The challenges in implementing epidemiologic studies in low-resource settings, particularly with infants, include effectively engaging with mobile communities , recruiting and training qualified staff , avoiding cold-chain or lab supply disruption , ensuring quality equipment for sample processing, minimizing burden to participants, and building relationships and communicating between multi-national study teams . Though the methods of such studies are sometimes reported [5,6,7], strategies and systems for maintaining data quality and knowledge exchange with local partners are not well documented.
Monitoring and evaluation (M&E) frameworks offer a metric for tracking progress towards project goals through a logical framework documenting intermediate and long-term measurable objectives [8, 9]. Intermediate objectives are ways to check on individual pieces of a study and are measured once or more at pre-set intervals during implementation, allowing study staff to reassess implementation strategy if objectives are not met. Long-term objectives clarify plans for the study . M&E frameworks facilitate adaptation to real-world conditions during study implementation, improving data quality, participant safety, and study efficiency. Participant engagement (e.g., recruitment, retention, safety), biological sample (e.g., stool and blood), and data (e.g., anthropometry, questionnaires) collection and quality during human subjects research with vulnerable populations or in low-resource settings.
Rotavirus (RV) is the most common cause of severe diarrhea in infants worldwide [10, 11], despite widespread implementation of anti-RV oral vaccines Rotarix® and Rotateq®. These vaccines are least efficacious and effective in developing countries where morbidity and mortality from RV are high [12,13,14,15]. Proposed hypotheses for reduced oral vaccine effectiveness include growth impairment, micronutrient deficiency in infants and mothers, and inflammation due to co-infection with other enteric pathogens [16,17,18,19,20,21,22,23,24,25]. Few studies have tested hypothesized causes for reduced RV vaccine effectiveness in low-resource settings . Testing these hypotheses requires human subjects’ studies in low-resource settings, where the vaccines are least effective. The additional logistical challenges of conducting human subjects research in low-resource settings with vulnerable populations (infants) could interfere with data quality and participant safety. A robust M&E framework may be helpful to continuously monitor and improve participant engagement, biological sample, and data quality and collection.
This paper describes the use of a monitoring and evaluation (M&E) framework to improve study efficiency and quality during participant engagement, and biological sample and data collection in human subjects research in a low-resource setting. We use as a case study an observational prospective cohort study to evaluate the effect of infant chronic undernutrition on infant RV-specific immunogenicity of 364 infants in El Alto, Bolivia.
The primary objective of the Infant Nutrition, Immunology, and Diarrheal Illness study (Nutrición, Inmunología y Diarrea Infantil - NIDI study) was to evaluate the effect of infant chronic undernutrition (length-for-age z-score (LAZ) < −2 or weight-for-length z-score (WLZ) < −2 ) on infant RV-specific immunogenicity of infants in El Alto, Bolivia. Secondary objectives were to evaluate the effects of (1) maternal and infant micronutrient deficiency, (2) maternal RV-specific immunity, and (3) early-life enteric infection and inflammation on infant RV-specific immunogenicity. A tertiary objective was to evaluate the effects of early-life enteric infection, subclinical inflammation, nutritional status, and post-vaccination RV-specific immunity on malnutrition at one year of age.
Bolivia was selected because it has a high prevalence of infant and maternal anemia and undernutrition, high incidence of diarrheal illness, high under-five mortality rate , and because it provides, as part of state-sponsored programming, the Rotarix® vaccine free of charge during well-child visits. Following a pilot study in three hospitals, the city of El Alto was selected as a location with proximity to a partner laboratory and home to a marginalized indigenous population with chronic undernutrition.
A sample size of 350 sets of maternal-infant data was calculated to provide 80% power (2-sided α ≤ =0.05), based on 1) an 18% difference between rates of RV-vaccine antibody seroconversion among children with vs. without chronic undernutrition [12,13,14, 28], and 2) an estimated prevalence of 29–38% chronic undernutrition in mothers and infants. Unpublished pilot data collected 2010–2011 at the study hospitals found that 29% of children less than four months of age had moderate to severe chronic undernutrition undernutrition (LAZ < −2 or WLZ < −2 ) and the 2008 Bolivia National Demographic and Health Survey found that 38% of women had anemia (Hemoglobin (Hb) <14.7 g/dl) . Based on an estimated 22% loss to follow-up (LTFU) or failed blood sample collection (unpublished pilot data), the recruitment objective was set at 450 mother-infant (M-I) pairs.
Data were collected June 2013 – March 2015. There were up to ten study visits; seven or eight scheduled hospital visits at well-child visits over 12–18 months, and one or two at home 4 and/or 7 days after the first dose of the RV vaccine (Table 1). Some infants returned to the hospital for an eighth visit for a third blood sample collection if it was not collected at the seventh well-child hospital visit at one year of age. The first two study visits were prior to the initial dose of the RV vaccine and served as baseline data. Please see the first study visit questionnaire provided as an Additional file 1 (NIDI_Study_Questionnaire_Visit_1.pdf) for complete baseline information collected.
Development of the monitoring and evaluation framework
The NIDI study M&E framework was developed using a formal M&E logical framework  based on the overall study goal and specific objectives and included a comprehensive data management plan and objectives in the areas of participant engagement (e.g., recruitment, retention, safety), biological sample (e.g., stool and blood), and data (e.g., anthropometry, questionnaires) collection and quality (Tables 2 and 3).
Participant engagement (e.g., recruitment, retention, safety) M&E indicators were determined based on the statistical power needed to answer the primary and secondary study objectives, the estimated loss-to-follow up based on previous pilot studies at the hospitals, estimated non-compliance with sample collection. Rigorous participant safety M&E indicators focused on adverse events (AE) related to the infant blood draws and on infant mortality (a severe adverse event (SAE) not expected to be related to the study) because of the high infant mortality rate in Bolivia . This AE and SAE monitoring was not required by Emory Institutional Review Board or the Bolivian National Bioethics Committee, as this study was not an interventional trial, but was done to ensure that any unforeseen risks to participants would be identified quickly during data collection, so they could be addressed in real time. Biological sample M&E indicators were determined in collaboration with the laboratories that analyzed the resulting samples, focusing on the factors most likely to affect stool or plasma quality and analysis. Anthropometric M&E indicators were developed by the expert anthropometrist (PSS) based on World Health Organization (WHO) best practice [31, 32]. Anthropometric measurements were monitored closely using multiple indicators (Table 3) because of the difficulty of precise and accurate anthropometric data collection in infants and because these measurements directly affected the primary study outcome. Because the study staff were collecting the infants’ anthropometric data for their well-child visits (rather than the hospital nurses), an additional participant safety M&E indicator included the percentage of infants with chronic or acute malnutrition that were identified as such upon anthropometric measurement, to ensure that study participants were appropriately flagged for evaluation by their pediatricians during the well-child visits that followed the study visits.
NIDI staff underwent a 10-day training on the Bolivian health system, interview protocols, informed consent, anthropometry, clinical sample collection and processing, treatment referrals, and AE. An experienced anthropometrist (PSS) conducted anthropometry training using methods developed by the WHO [33, 34]. During the initial training and bi-annual standardizations, staff measured the recumbent length of 10 children under two years old twice, and continued until they produced measurements within 0.5 cm of repeat measurements and of the expert anthropometrist’s measurement in order to optimize precision and accuracy of measurements . Staff were trained in human subjects research ethics using the FHI 360 Research Ethics Training Curriculum (FHI 360, Durham, NC) and passed the corresponding evaluation . Phlebotomy staff at study hospitals were trained on study protocols for blood collection, processing, and cold chain following Centers for Disease Control and Prevention (CDC) guidelines . During training, indicators of participant’s engagement, biological sample and data collection and quality to be measured during the study were shared with staff.
Participant’s engagement and indicators
Enrollment and retention
Potential study participants were identified in the outpatient clinic waiting rooms of Hospital Municipal Corea or Hospital Los Andes. Mothers of young infants were approached, and if interested, their eligibility was assessed via a screening questionnaire. Mothers were eligible if they were ≥15 years, in general good health, the mother of an eligible infant, and were willing to bring their infant to the study hospital for well-child and study visits through one year of age. Infants were eligible if they were ≥14 days old, in general good health, and had not yet been vaccinated with Rotarix®. Infants were ineligible if they had fever, diarrhea, frequent cough, respiratory infection, or hospitalization in the past week, had been diagnosed with a birth defect, chromosomal disorder affecting growth, or immunodeficiency disorder. Eligible mothers were given information sheets and invited to enroll that day or at their next well-child visit (Fig. 1). The NIDI study logo (Fig. 2b) was on all study documents, and staff wore study clothing and photo identification cards bearing the logo so that mothers could easily identify study staff in the hospitals. Mothers were compensated for their time at each visit with food staples (e.g., 1 pound fortified rice). Enrollment and retention indicators included weekly and monthly enrollment numbers and percent lost to follow up by visit.
RV vaccination was conducted as part of the Bolivian national vaccination schedule following the blood draw at the second study visit which coincided with the infants’ 2-month well-child visits. The blood draw was considered an intervention in this study. RV vaccination was not a study intervention as it was administered as part of standard well-child care by hospital staff following the Bolivian national vaccination schedule. Thus, vaccination-related AE would have been recorded but not considered study-related AE. Expected related AE were blood draw complications including bruising, infection, or excessive bleeding within 48 h of blood collection. No study-related SAE were expected; expected unrelated SAE included hospitalization, disability, or death. All AE and SAE were reviewed by a pediatrician to determine severity and relation to the study within 48 h. AE and SAE indicators included review and follow-up of all AE and SAE reported by participants, number of study related AE and number and follow-up of study and non-study related SAE.
Mothers were referred to treating physicians for a body-mass index <18.5 kg/m2 (or mid-upper arm circumference < 21.0 cm if pregnant) or Hb <14.7 g/dl (Hb < 13.7 g/dl if pregnant) . Infants were referred to treating physicians for moderate or severe undernutrition (LAZ < −2 or WLZ < −2 ), anemia (Hb < 10.9 g/dl as per Bolivian national guidelines ), for any signs of severe illness, or if mothers reported diarrhea. If mothers reported diarrhea between visits, staff would collect a stool sample, test for RV, enterotoxigenic E. coli (ETEC), enteropathogenic E. coli (EPEC), enteroaggressive E. coli (EAEC), and shared results with the treating physician. Maternal or infant morbidity indictors related to participant safety included referral of each malnutrition case to appropriate physicians for treatment and follow-up of all malnutrition cases at subsequent visits.
Biological sample collection, quality, and indicators
At each visit, mothers provided their infant’s diaper with stool, if available. Study staff provided a clean diaper and transferred the stool to a sample container and a portion to Cary-Blair transport medium. Samples were stored at 2–8 degrees Centigrade (°C) and transported on ice within 24 h to the Universidad Mayor de San Andrés Instituto de Biología Molecular y Biotecnología (UMSA-IBMB) laboratory for processing. Stool samples were analyzed for RV with the ProSpecT RV Enzyme-linked immunosorbent assay (ELISA) kit (Oxoid, Basingstoke, UK). If no formed stool was available in the soiled diaper, the soiled area of the diaper was eluted in 500 ul of PBS. Stool was also tested for ETEC, EPEC, and EAEC as described in Gonzales et al., 2013 . Stool collection and quality indicators included the percent of samples successfully collected following maternal report of infant diarrhea.
Venous blood was collected from mothers approximately one and 6–8 months postpartum, and from infants at two, 6–8, and 12–18 months of age by the phlebotomy staff at study hospital laboratories (Fig. 1). Blood was collected by trained hospital phlebotomists and processed by study staff using sterile, disposable equipment. Blood was drawn using Safety-Lok™ 23-gauge winged needles and trace-metal-free EDTA Vacutainers® (BD, Franklin Lakes, NJ). An aliquot of whole blood from the needle was analyzed for Hb using the HemoCue® Hb 201+ System (HemoCue AB, Ängelholm, Sweden); anemia was defined using cut-offs adjusted for altitude (3500 m) and pregnancy . The remaining sample was processed to plasma as per Vacutainer® instructions: the vial was inverted 8 times, then centrifuged for 15 min at 1300 relative centrifugal force (RCF) (LW Scientific USA, Lawrenceville, GA) to separate the blood. The plasma supernatant was then transferred to microcentrifuge tubes and centrifuged for 3 min at 2200 RCF (Cole Parmer, Vernon Hills, IL) to remove any remaining precipitates. The supernatant was stored in screw-cap microtubes at 2–8 °C (up to 24 h) before being transported on ice to the UMSA-IBMB laboratory, where the tubes were stored at −70 °C until being transported on dry ice to Emory University laboratories following International Air Transport Association and CDC regulations. There plasma was aliquoted and shipped on dry ice to final analysis locations. Blood quality indicators included percent of samples with hemolysis, storage temperature and storage time.
Plasma was analyzed for RV-specific Immunoglobulins A (IgA) and G (IgG) by Enzyme-linked immunosorbent assay (ELISA) by the Gastroenteritis & Respiratory Viruses Laboratory, Division of Viral Diseases, CDC as described in Moon et al., 2010 . Plasma samples were analyzed for retinol binding protein (RBP), ferritin, soluble transferrin receptor (sTfR), alpha(1)-acid-glycoprotein (AGP), and C-reactive protein (CRP) by the VitMin Laboratory in Willstaett, Germany, as described in Erhardt et al., 2004 , and for plasma zinc using inductively-coupled plasma optical emission spectrometry by the Children’s Hospital of Oakland Research Institute Elemental Analysis Facility as described in Engle-Stone et al. 2014 . A subsample (10%) were also tested for retinol; results were used to validate RBP cutoffs for vitamin A deficiency since the molar ratio of retinol-to-RBP is not always 1:1 . All micronutrient biomarkers were adjusted for the effect of inflammation using a novel regression correction approach [45,46,47] .
Data collection, quality, and indicators
Vaccination data for the infants was collected from the infants’ clinical vaccination records, provided by the mothers at each study visit and was recorded by study staff as part of each study visit’s data collection. Sociodemographic data were collected at the first and last visit, morbidity and clinical data at all visits. Mothers were given a wall calendar and stickers of images indicating morbidities (diarrhea, fever, cough, hospitalization) as a memory aid to record infant morbidities. Mothers were asked about infant feeding practices, including breastfeeding. Infant nutritional status was determined by LAZ, WLZ, and head circumference at each visit by trained interviewers using standardized wooden boards to measure supine length (Shorrboards, Olney, MD) and an electronic M-I floor scale to measure weight (SECA scales, Hanover, MD). Measurements were placed in the infant’s clinical chart for their corresponding well-child visit, and mothers were given a handout at the end of the study documenting their child’s growth throughout the study (Fig. 2a). The SAS program code used to create the growth chart (NIDI_GrowthCurveMacro.sas) is provided as an Additional file 2. Maternal height and weight were collected at each visit to calculate body mass index. Data were collected on paper forms and double-data entered using the web-based research electronic data capture (REDCap) system hosted at Emory University (UL1 TR000421) . Discrepancies between entries were reconciled within a week. Any biologically implausible data points were corrected or set to a missing value. Sample processing and cold chain data were double-entered and reconciled in Microsoft Excel (Seattle, WA).
Data collection and quality indicators included digit preference for anthropometry measures (height, weight and length), standard deviations of weight-for-length, length-for-age and weight-for-age z-scores and percent missing anthropometric data for each visit.
Reporting and application of M&E indicators
Emory University staff created monthly reports of monitoring indicators including measures of participant retention, biological sample collection (e.g., hemolysis, storage temperature, and storage time for blood samples), and data quality. LTFU and reasons for withdrawal were presented by visit. For anthropometry data quality indicators (Table 3), digit preferences in measurements were visualized in a graph (Fig. 2c). Reports were shared with study staff during monthly meetings, strategies were developed to improve data quality, goals were set for the next month, and rewards were given for meeting quality targets. All study components were monitored in-person by principal investigators or project managers 2–3 times a year.
While indicator targets for participant enrollment were met, the number of infants lost to follow-up was greater than expected. Of 2331 M-I pairs initially screened, 1336 infants were eligible, and 456 mothers and 461 infants (including 5 twins) were enrolled in the study June 2013–April 2014, exceeding the target of 422 M-I pairs. Six percent (27/461) of infants were excluded as ineligible, primarily due to having received the RV vaccine prior to the first infant blood-draw. Fifteen percent (70/461) of infants were lost to follow-up, which did not meet the target of <10% LTFU (Fig. 1). However, 91% (364/400) of infants that completed the first two visits completed the study. In total, 364 infants completed the study through the final visit.
There were four study-related AE (all minor swelling and bruising related to the blood draw), no study-related SAE, and five infant deaths (SAE) unrelated to the study. SAE included hypoxic-ischemic encephalopathy (age 4 months), pneumonia (two at ages 2 and 3 months), diarrhea (age 2 months), and sudden infant death syndrome (age 3 months). After review of death certificates and conducting verbal autopsies when necessary, a pediatrician determined each death was unrelated to the study due to its medical cause. This mortality rate of infants in the study, equivalent to 10.8 per 1000 enrolled infants, is less than half the infant mortality rate in Bolivia (23 per 1000 post-neonatal infants) .
Biological sample indicator targets were met for stool sample collection but not blood sample collection (Table 2). There were 389 successful infant blood draws at 1–2 months (baseline), but only 333 (95% of the required 350 blood sample size) also had a 6–8 month sample (Fig. 1). Hemolysis occurred frequently during blood draws, despite proper collection and processing technique, but did not affect sample analysis. Stool samples to assess post-vaccination RV viral shedding were collected from 75% of infants, while 61% of reported diarrhea cases had associated stool samples collected.
Anthropometry data quality indicators were monitored throughout the study (Table 3). The medians of monthly standard deviations for weight-for-length, length-for-age, and weight-for-age z-scores were 1.01, 0.98, and 1.03, respectively; close to the expected value of 1.0 for a reference distribution. Standard deviations for z-scores varied month-to-month, but never reached the WHO thresholds for measurement error or incorrect age reporting . However, only 61% of measurements indicating stunting and only 56% of measurements indicating wasting resulted in an immediate referral for treatment. Referral practices improved during the second year of data collection: 87% of measurements indicating stunting and 91% of measurements indicating wasting resulted in an immediate referral. Throughout the study, these indicators were presented to study staff each month to encourage improvement and follow-up. Thus, all stunted and wasted children who had not initially been referred were followed-up to ensure they received treatment.
This paper describes the data and participant safety M&E framework of an observational cohort study of mothers and their infants and challenges encountered in the implementation of this study in El Alto, Bolivia. The M&E framework allowed staff to respond both to predicted and unforeseen challenges in study implementation.
The study met its enrollment objective (450 M-I pairs). However, it collected 95% of the required first two infant blood samples from 350 infants which meant it did not meet its sample size objectives (350 infants), although 364 infants completed the final study visit. Follow-up was hindered by unexpected and rapid policy changes to the Bolivian health system in December 2013, which shifted well-child visits to primary care clinics during data collection [50, 51]. Though participants could continue well-child visits at the study hospital, conflicting national messages and the convenience of primary care clinics reduced follow-up.
Reported AE were minor (local bruising and swelling related to the blood draw). All infant deaths were determined to be unrelated to the study after verbal autopsy and review of the death certificate by the study pediatrician. Though the target for this indicator was no SAE during the course of the study, infant mortality in the study was less than half of the post-neonatal infant mortality rate in Bolivia . This could be due to differences between the national population and the populations attending the two study hospitals.
The study collected samples from only 18% of the predicted diarrhea case count, but collected a sample 61% of the time when diarrhea was reported. As the monthly monitoring indicator was the percent of reported diarrhea that resulted in sample collection, no changes were made in study protocol to collect more samples. Diarrhea may have been under-reported, but it is also possible that the population simply had fewer cases of diarrhea during the first year of life than were predicted based on pilot data. Post-RV vaccination stool samples collection (75%) exceeded the goal of 50% of enrolled infants, likely due to the convenience of the home visit(s) for sample collection.
Anthropometry data quality was extremely good, likely due to extensive training, standardization, and monitoring efforts. However, many children were not immediately referred for treatment when they were measured as stunted or wasted. Each month, this indicator was measured and presented to study staff to encourage improvement and follow-up. Thus, all stunted and wasted children who had not initially been referred were followed-up to ensure they received treatment. Poor referral practices were likely the result of failure to identify stunting and wasted infants using the WHO growth charts, especially in the youngest infants and early in the study, when staff burden was highest. In addition, study staff stopped referring children measured as stunted at repeated visits, as they were already being monitored by their physicians.
The monitoring plan, based on the M&E framework indicators, successfully allowed course-correction and quality improvement throughout the study. Well-trained local staff, a clear protocol for AE, and a comprehensive data management plan facilitated the success of the study. Having well-trained staff who spoke local languages and understood the political and cultural landscape allowed the study to continue despite local political unrest and health policy changes. Preparation for, and management of, AE took considerable time and resources. Although this was an observational study, this effort was warranted given cultural sensitivity to blood collection and the vulnerable population (infants as young as two months). Having a comprehensive data management plan allowed for high quality data control. Monthly monitoring meetings in which the M&E framework indicators were reviewed and discussed with in-country staff maintained a sense of accountability and ownership, and allowed for open discussion of study challenges and potential solutions.
Some aspects of development, implementation and adaptation of the M&E framework could be improved in future studies. In the development of the framework, indicators for staff satisfaction and retention were not considered, although staff data collection burden was monitored. As training and management of local staff took considerable time and resources, monitoring of staff satisfaction may have allowed earlier course correction in this area. In the second year of the study, staff satisfaction surveys were implemented, which allowed in-country and Emory University project managers to improve staff satisfaction and provide additional resources to in-country staff as needed. In the implementation of the framework, specific aspects of how indicators would be analyzed and communicated to all study team members were not considered until data collection was underway. Including a detailed communication structure in the framework would have ensured uniformity and continuity of communication from the beginning. Finally, the study design and M&E framework were not revised following an unexpected and rapid policy change in the Bolivian health system that influenced enrollment and retention [50, 51]. Allowing the study design and M&E framework to adapt to external features such as changes in the health system would have improved implementation.
The data and participant safety M&E framework described here and lessons learned from this study can be applied to other observational studies to improve the quality and comparability of data. The data and current [46, 52] and forthcoming publications will advance the base of knowledge on oral vaccine underperformance and infant undernutrition in developing countries.
Enteroaggressive E. coli
Enzyme-linked immunosorbent assay
Enteropathegenic E. coli
Enterotoxigenic E. coli
Loss to follow-up
Monitoring and evaluation
Nutrition, Immunology, and Diarrheal Illness or Nutrición, Inmunología y Diarrea Infantil
Retinol binding protein
Relative centrifugal force
Research electronic data capture
Severe adverse events
Soluble transferrin receptor
Universidad Mayor de San Andrés Instituto de Biología Molecular y Biotecnología
Tindana PO, Singh JA, Tracy CS, Upshur REG, Daar AS, Singer PA, Frohlich J, Lavery JV. Grand Challenges in Global Health: Community Engagement in Research in Developing Countries. PLoS Med. 2007;4(9):e273.
Sitthi-Amorn C, Somrongthong R. Strengthening health research capacity in developing countries: a critical element for achieving health equity. BMJ. 2000;321(7264):813–7.
Matthias DM, Robertson J, Garrison MM, Newland S, Nelson C. Freezing temperatures in the vaccine cold chain: a systematic literature review. Vaccine. 2007;25(20):3980–6.
White F. Capacity-building for health research in developing countries: a manager's approach. Rev Panam Salud Publica. 2002;12:165–72.
Murray-Kolb LE, Rasmussen ZA, Scharf RJ, Rasheed MA, Svensen E, Seidman JC, Tofail F, Koshy B, Shrestha R, Maphula A, et al. The MAL-ED cohort study: methods and lessons learned when assessing early child development and caregiving mediators in infants and young children in 8 low- and middle-income countries. Clin Infect Dis. 2014;59(Suppl 4):S261–72.
Suchdev PS, Ruth L, Obure A, Were V, Ochieng C, Ogange L, Owuor M, Ngure F, Quick R, Juliao P, et al. Monitoring the marketing, distribution, and use of Sprinkles micronutrient powders in rural western Kenya. Food Nutr Bull. 2010;31(2 Suppl):S168–78.
Kirkpatrick BD, Colgate ER, Mychaleckyj JC, Haque R, Dickson DM, Carmolli MP, Nayak U, Taniuchi M, Naylor C, Qadri F, et al. The "Performance of Rotavirus and Oral Polio Vaccines in Developing Countries" (PROVIDE) study: description of methods of an interventional study designed to explore complex biologic problems. Am J Trop Med Hyg. 2015;92(4):744–51.
UNICEF, World Health Organization: Framework for operations and implementation research in health and disease control programs. 2008.
Crawford P, Bryce P. Project monitoring and evaluation: a method for enhancing the efficiency and effectiveness of aid project implementation. International Journal of Project Management. 2003;21(5):363–73.
Kotloff KL, Nataro JP, Blackwelder WC, Nasrin D, Farag TH, Panchalingam S, Wu Y, Sow SO, Sur D, Breiman RF, et al. Burden and aetiology of diarrhoeal disease in infants and young children in developing countries (the Global Enteric Multicenter Study, GEMS): a prospective, case-control study. Lancet. 2013;382(9888):209–22.
Zaman K, Anh DD, Victor JC, Shin S, Yunus M, Dallas MJ, Podder G, Thiem VD, Luby SP, Coia ML. Efficacy of pentavalent rotavirus vaccine against severe rotavirus gastroenteritis in infants in developing countries in Asia: a randomised, double-blind, placebo-controlled trial. The Lancet. 2010;376(9741):615–23.
Madhi SA, Cunliffe NA, Steele D, Witte D, Kirsten M, Louw C, Ngwira B, Victor JC, Gillard PH, Cheuvart BB, et al. Effect of Human Rotavirus Vaccine on Severe Diarrhea in African Infants. N Engl J Med. 2010;362(4):289–98.
Armah GE, Sow SO, Breiman RF, Dallas MJ, Tapia MD, Feikin DR, Binka FN, Steele AD, Laserson KF, Ansah NA. Efficacy of pentavalent rotavirus vaccine against severe rotavirus gastroenteritis in infants in developing countries in sub-Saharan Africa: a randomised, double-blind, placebo-controlled trial. The Lancet. 2010;376(9741):606–14.
Bar-Zeev N, Jere KC, Bennett A, Pollock L, Tate JE, Nakagomi O, Iturriza-Gomara M, Costello A, Mwansambo C, Parashar UD, et al. Population Impact and Effectiveness of Monovalent Rotavirus Vaccination in Urban Malawian Children 3 Years After Vaccine Introduction: Ecological and Case-Control Analyses. Clin Infect Dis. 2016;62(Suppl 2):S213–9.
Patel M, Shane AL, Parashar UD, Jiang B, Gentsch JR, Glass RI: Oral rotavirus vaccines: how well will they work where they are needed most? J Infect Dis 2009, 200(Supplement 1):S39-S48.
Savy M, Edmond K, Fine PE, Hall A, Hennig BJ, Moore SE, Mulholland K, Schaible U, Prentice AM. Landscape analysis of interactions between nutrition and vaccine responses in children. J Nutr. 2009;139(11):2154S–218S.
Lopman BA, Pitzer VE, Sarkar R, Gladstone B, Patel M, Glasser J, Gambhir M, Atchison C, Grenfell BT, Edmunds WJ. Understanding Reduced Rotavirus Vaccine Efficacy in Low Socio-Economic Settings. PLoS ONE. 2012;7(8):e41720.
Bailey RL, West KP Jr, Black RE. The Epidemiology of Global Micronutrient Deficiencies. Ann Nutr Metab. 2015;66(Suppl. 2):22–33.
Burke RM, Leon JS, Suchdev PS. Identification, prevention and treatment of iron deficiency during the first 1000 days. Nutrients. 2014;6(10):4093–114.
Caulfield LE, Richard SA, Rivera JA, Musgrove P, Black RE. Stunting, Wasting, and Micronutrient Deficiency Disorders. In: Disease Control Priorities in Developing Countries. edn. Edited by Jamison DT, Breman JG, Measham AR, Alleyne G, Claeson M, Evans DB, Jha P, Mills A, Musgrove P. Washington (DC): World BankThe International Bank for Reconstruction and Development/The World Bank Group; 2006.
Fischer Walker CL, Ezzati M, Black RE. Global and regional child mortality and burden of disease attributable to zinc deficiency. Eur J Clin Nutr. 2009;63(5):591–7.
Guerrant RL, Oria RB, Moore SR, Oria MO, Lima AA. Malnutrition as an enteric infectious disease with long-term effects on child development. Nutr Rev. 2008;66(9):487–505.
Ngure F, Reid B, Humphrey J, Mbuya M, Pelto G, Stoltzfus R. Water, sanitation, and hygiene (WASH), environmental enteropathy, nutrition, and early child development: making the links. Ann N Y Acad Sci. 2014;1308:118–28.
Rytter MJ, Kolte L, Briend A, Friis H, Christensen VB. The immune system in children with malnutrition--a systematic review. PLoS ONE. 2014;9(8):e105017.
WHO Growth Charts [http://www.cdc.gov/growthcharts/who_charts.htm#The WHO Growth Charts]
Coa R, Ochoa L, MEASURE DHS, Macro International Inc.: Bolivia: Encuesta nacional de demografia y salud 2008. In: DHS Final Reports. Edited by DHS. Calverton, MD; 2008.
Vesikari T, Itzler R, Matson DO, Santosham M, Christie CD, Coia M, Cook JR, Koch G, Heaton P. Efficacy of a pentavalent rotavirus vaccine in reducing rotavirus-associated health care utilization across three regions (11 countries). Int J Infect Dis. 2007;11(Suppl 2):S29–35.
Jensen G. The logical framework approach. Bond for International Development. 2010;4
Sardan MG, Ochoa LH, Guerra WC. Bolivia - Encuesta Nacional de Demografia y Salud 2003. In: MEASURE DHS. Edited by DHS Final Reports. Instituto Nacional de Estadistica, Ministerio de Salud y Deportes: Calverton, MD; 2004.
Mei Z, Grummer-Strawn LM. Standard deviation of anthropometric Z-scores as a data quality assessment tool using the 2006 WHO growth standards: a cross country analysis. Bull World Health Organ. 2007;85(6):441–8.
Wit JM, Himes JH, van Buuren S, Denno DM. Suchdev PS: Practical Application of Linear Growth Measurements in Clinical Research in Low-and Middle-Income Countries. Hormone Research in Paediatrics. 2017;
Nutrition Survey Toolkit [http://www.micronutrient.org/nutritiontoolkit/training.htm]
Onis M. WHO Child Growth Standards based on length/height, weight and age. Acta Paediatr. 2006;95(S450):76–85.
Research Ethics Training Curriculum [https://www.fhi360.org/sites/all/libraries/webpages/fhi-retc2/]
Guidelines for Collecting and Testing Blood Samples for Micronutrients [http://www.cdc.gov/labstandards/vitaleqa_video.html]
Cohen JH, Haas JD. Hemoglobin correction factors for estimating the prevalence of iron deficiency anemia in pregnant women residing at high altitudes in Bolivia. Rev Panam Salud Publica. 1999;6(6):392–9.
Ministerio de Salud y Deportes de Bolivia: Programa de "Desnutrición Cero" Niñas y niños bien nutridos, un compromiso de todos. In., vol. 14, Primero edición edn. La Paz, Bolivia: Serie Documentos Técnicos Normativos; 2007.
Gonzales L, Joffre E, Rivera R, Sjoling A, Svennerholm AM, Iniguez V. Prevalence, seasonality and severity of disease caused by pathogenic Escherichia coli in children with diarrhoea in Bolivia. J Med Microbiol. 2013;
World Health Organization: Haemoglobin concentrations for the diagnosis of anaemia and assessment of severity. In: Vitamin and Mineral Nutrition Information System. Geneva: World Health Organization (WHO/NMH/NHD/MNM/11.1); 2011.
Moon SS, Wang Y, Shane AL, Nguyen T, Ray P, Dennehy P, Baek LJ, Parashar U, Glass RI, Jiang B. Inhibitory effect of breast milk on infectivity of live oral rotavirus vaccines. Pediatr Infect Dis J. 2010;29(10):919–23.
Erhardt JG, Estes JE, Pfeiffer CM, Biesalski HK, Craft NE. Combined measurement of ferritin, soluble transferrin receptor, retinol binding protein, and C-reactive protein by an inexpensive, sensitive, and simple sandwich enzyme-linked immunosorbent assay technique. J Nutr. 2004;134(11):3127–32.
Engle-Stone R, Ndjebayi AO, Nankap M, Killilea DW, Brown KH. Stunting prevalence, plasma zinc concentrations, and dietary zinc intakes in a nationally representative sample suggest a high risk of zinc deficiency among women and young children in Cameroon. J Nutr. 2014;144(3):382–91.
Engle-Stone R, Haskell MJ, Ndjebayi AO, Nankap M, Erhardt JG, Gimou M-M, Brown KH. Plasma retinol-binding protein predicts plasma retinol concentration in both infected and uninfected Cameroonian women and children. The Journal of nutrition. 2011;141(12):2233–41.
Suchdev PS, Namaste SM, Aaron GJ, Raiten DJ, Brown KH, Flores-Ayala R. BRINDA Working Group: Overview of the Biomarkers Reflecting Inflammation and Nutritional Determinants of Anemia (BRINDA) Project. Advances in Nutrition: An International Review Journal. 2016;7(2):349–56.
Burke RM, Suchdev PS, Rebolledo PA, de Aceituno AM, Revollo R, Iniguez V, Klein M, Drews-Botsch C, Leon JS. Predictors of Inflammation in a Cohort of Bolivian Infants and Toddlers. Am J Trop Med Hyg. 2016;95(4):954–63.
Namaste SM, Aaron GJ, Varadhan R, Peerson JM, Suchdev PS: Methodologic approach for the Biomarkers Reflecting Inflammation and Nutritional Determinants of Anemia (BRINDA) project. Am J Clin Nutr 2017, 106(Suppl 1):333s-347s.
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.
World Health Organization: Physical status: The use of and interpretation of anthropometry, Report of a WHO Expert Committee. In. Geneva, Switzerland: World Health Organization; 1995.
Sistema de salud rechaza Ley 475. In: El Diario. Bolivia; 2014.
Ley de Prestaciones de Servicios de Salud Integral del Estado Plurinacional de Bolivia. In: 475. Edited by Plurinacional lAL. Gaceta Oficial del Estado Plurinacional de Bolivia: Bolivia; 2013.
Burke RM, Rebolledo PA, Fabiszewski de Aceituno AM, Revollo R, Iniguez V, Klein M, Drews-Botsch C, Leon JS, Suchdev PS. Early deterioration of iron status among a cohort of Bolivian infants. Matern Child Nutr. 2016;
World Health Organization: Global Database on Child Growth and Malnutrition. In. Geneva, Switzerland: World Health Organization; 2017.
First, we thank our study participants and their families. We also thank our study personnel, colleagues at the Universidad Mayor de San Andrés and Centro de Atención Integral para Adolecentes, and participating hospitals “Infantil Los Andes” and “Modelo Corea” in La Paz and El Alto, Bolivia. We are also grateful to Drs. Donnie Whitehead and Juergen Erhardt for their assistance with the biological samples, and to Ms. Janet Figueroa for her assistance with M&E activities. Thank you to Dr. Emily R. Smith for the initial consultation of an M&E framework in the study design of a low-resource setting. We are grateful to the Emory University Global Health Institute and Emory University Rollins School of Global Public Health Global Field Experience Fund for travel support of several Emory graduate students who participated in this research.
Study design, data collection, and data analysis were supported by NIH-NIAID grants 1K01AI087724–01 and U19-AI057266 (received by JL), the Emory + Children’s Pediatric Center Seed Grant Program (received by PS), the Public Health Service Grant UL1 TR000454 from the NIH Clinical and Translational Science Award Program (received by JL), and the International Collaborative Award for Research from the International Pediatric Research Foundation (received by PS). RB’s time and travel for data collection and analysis were supported by the Laney Graduate School of Emory University, a NIH T32 Institutional National Research Service Award training grant in Reproductive, Pediatric and Perinatal Epidemiology (HD052460–01), and the Burroughs Wellcome Fund’s Molecules to Mankind Program (M2 M). PR’s time and travel for data collection and analysis were supported by the Thrasher Research Fund 2015 Early Career Award and the NIH Vaccinology Training Program award T32AI074492. KS’s time for analysis and interpretation of data and writing the manuscript was supported by the Laney Graduate School of Emory University. AA’s time for analysis and interpretation of data and writing the manuscript was supported by RTI International.
Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available due to ongoing analysis and publication but will be available from the corresponding author on reasonable request once data analysis and publication is complete.
Ethics approval and consent to participate
The study protocol, consent forms, documents, annual reports, and amendments were reviewed and approved by the Bolivian National Bioethics Committee, Research Ethics Commission (Comisión de Ética de la Investigación - Comité Nacional de Bioética) and by Emory University’s Institutional Review Board (IRB00056127). The study convened a Bolivian scientific advisory board, composed of three Bolivian pediatricians, to provide independent oversight and advocate for study participants. The scientific advisory board met annually 2012–2015 and reviewed all study data, protocol changes, and adverse event and severe adverse event reports. Mothers provided written consent in Spanish or Aymara after explaining the purpose of the study in their own words. The 42 mothers who were minors also needed their parent’s consent.
Consent for publication
The authors declare they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
NIDI_Study_Questionnaire_Visit_1.pdf. NIDI Study Questionnaire - visit one. NIDI study questionnaire for visit one – baseline data collection. Includes informed consent. Note: the questionnaire was developed in Spanish and is presented here in Spanish (SAS 7 kb)
NIDI_GrowthCurveMacro.sas. SAS Program Code for Infant Growth Chart. SAS Macro to create graphs that show individual child growth over time compared to a WHO reference (PDF 882 kb)
About this article
Cite this article
Aceituno, A.M., Stanhope, K.K., Rebolledo, P.A. et al. Using a monitoring and evaluation framework to improve study efficiency and quality during a prospective cohort study in infants receiving rotavirus vaccination in El Alto, Bolivia: the Infant Nutrition, Inflammation, and Diarrheal Illness (NIDI) study. BMC Public Health 17, 911 (2017). https://doi.org/10.1186/s12889-017-4904-5