Using C-LQAS to assess vaccination coverage near the end of the integrated measles and YF campaign in Sierra Leone, we identified five areas (lots) with unacceptably low measles vaccination coverage and three with low YF coverage, based on documented proof of vaccination (card). This allowed the health authorities to put in place control measures to increase vaccination coverage before the end of the campaign in the rejected areas. Relying on verbal confirmation of vaccination status, fewer lots would have failed, as it is expected in these settings [17, 22]. With one exception, verbally confirmed district vaccination coverage estimated through the post-campaign cluster-sampling surveys was above the minimum acceptable threshold (75%) used for the end-campaign C-LQAS survey, suggesting that control measures to raise coverage may have been effective in the areas of weakness identified. The fact that one district (Port Loko), failing at the end of the campaign for both low measles and YF coverage, presented post-campaign measles and YF coverage figures not significantly above 75%, suggests that it may have structural problems in reaching the target population with vaccination activities.
We used C-LQAS for end-campaign monitoring. While this approach allows time to implement mop-up activities, undertaking an assessment before the end of the time allocated to achieve the target (95%) may lead to underestimation of coverage . For this reason we set the UT of the C-LQAS plan to 90% instead of 95%, the actual coverage target of the campaign. We then decided to use 75% as LT based on previous experiences showing that using a “grey area” of 15% is a possible solution to keep statistical errors reasonably low if N = 50 (5x10) . By setting alpha (≤8%) lower than beta (≤19%), we gave priority to the classification based on the LT (75%), rather than the UT (90%). We wanted to be more confident that we were not accepting lots with low coverage, which is a risk for the population (alpha is also known as “consumer risk”); rather than failing lots with high coverage, which is a risk for the vaccination programme (beta is also known as “provider risk”). Results of the C-LQAS surveys need to be interpreted as an indication that, towards the end of the campaign, at least 92% (1-alpha) of the accepted lots were probably reaching the minimum acceptable coverage level of 75%; while at least 81% (1-beta) of the failed lots were probably not reaching the desirable coverage level of 90%. In other words, when we accept a lot we are 92% confident that real coverage is at least 75% (i.e. when we reject we cannot be confident that coverage is 75%).
Intense efforts from WHO and UNICEF (United Nations Children’s Fund) to encourage 90% coverage for first dose measles vaccine by 12 months of age and offering a second opportunity for vaccination have contributed to a substantial reduction of measles cases in the African region [23, 24]. In this respect, the results (i.e. 15 lots out of 20 accepted for measles coverage at least at 75%), albeit before the end of the campaign, are far from optimal. Although achieving 75% vaccination coverage may be enough to stop transmission of YF, it is definitely not enough for measles, for which the threshold believed to confer herd immunity is 95% . If we would have used the other, stricter, decision value proposed in Table 2 (d = 4; UT = 95%; LT = 80%) we would have accepted even fewer lots (i.e. 10 out of 20) as with measles coverage at least at 80%. By allowing individuals to present only the vaccination card of the current campaign as evidence of measles vaccination we may have underestimated the actual measles coverage in the country. Even if there was an underestimation of the true coverage rates, it is important to note that three weeks before the vaccination campaign the largest measles outbreak over the last decade started in the country and lasted until July 2010, confirming that measles vaccination coverage was low in Sierra Leone .
The C-LQAS approach works better if there is indication that the lots are homogeneous in terms of coverage [2, 9, 17]. Where possible, we attempted to divide districts into smaller lots to increase the likelihood of homogeneity. The distribution of vaccination coverage between the clusters in the lots exceeded the maximum assumed 0.1 SD always in rejected lots, apart from one occasion, Lot 8 in Port Loko, which had adequate YF coverage. Such a high ICV of vaccination coverage indicates that some of the clusters in those lots had many unvaccinated individuals, while others had few (i.e. the lot is not homogeneous in terms of vaccination coverage). This kind of distribution of coverage may indicate that some of the EAs may have been missed or inappropriately covered by vaccination activities (a good reason to reject the lot for poor quality of vaccination).
We showed that assuming that vaccination coverage would vary according to a SD of 0.1 in the clusters is a similar approach to assuming a DEFF of approximately three (if coverage is at 75%) or approximately six (if coverage is at 90%). We believe that this is more conservative than assuming DEFF = 2 as it is done in a standard manner while designing cluster surveys when there is no previous information to guide this decision . The fact that the assumed DEFF increases proportionally to the assumed coverage is explained by the fact that assuming a variance of 0.1 SD when coverage is high (i.e. 90%) has a greater magnitude than assuming the same variance when coverage is low (i.e. 75%). Therefore, our approach should be more robust as the levels of real coverage get higher.
This study is subject to a number of limitations. First, because of limited resources we conducted the C-LQAS surveys at sub-district level only in the areas of the country targeted by both vaccines. In the part of the country targeted only for measles vaccination, each district was a lot. These districts have a mean total population of approximately 350,000. It may have been difficult to implement targeted control measures in such large districts. To overcome this limitation we advised the surveying teams to discuss the results of the C-LQAS surveys during the daily campaign meetings chaired by the district medical officers, in order to facilitate the implementation of control measures using all the information available also from other sources (e.g. vaccination teams or campaign supervisors). Second, by using PPS to select clusters, we inevitably placed data collection in the most populous communities. These are areas that generally are easier to cover also by vaccinators, so vaccination coverage may have been overestimated. PPS does not yield a spatially even sample as can be seen in Figure 2. Alternative methods based on geographic sampling could be used to allow that sampling locations are more evenly spread across the lots [27–29]. Third, as previously seen with the C-LQAS approach, the high inter-cluster variability (SE > 0.1) seen in some lots, reaching a maximum level of SE = 0.21, may indicate that the plans may have had errors (alpha and beta) above the levels defined in some lots [9, 17], although this finding should be interpreted with caution, since it is not possible to support it statistically given the small sample sizes per lot used to calculate the SE. One way to reduce the inter-cluster variability and consequently increase the precision would be to increase the number of clusters sampled to more than five . Finally, when we aggregated several lots to obtain a district level LQAS classification, we were not able to exactly assess how the unequal probabilities of selection (i.e. different lot population sizes) may have affected alpha and beta. On the other hand, the increased sample sizes at district level combining more than one lot would probably have errors that are lower than the ones assumed at lot level.