Prospective evaluation of a complex public health intervention: lessons from an initial and follow-up cross-sectional survey of the tuberculosis strain typing service in England
BMC Public Health volume 14, Article number: 1023 (2014)
The national tuberculosis strain typing service (TB-STS) was introduced in England in 2010. The TB-STS involves MIRU-VNTR typing of isolates from all TB patients for the prospective identification, reporting and investigation of TB strain typing clusters. As part of a mixed-method evaluation, we report on a repeated cross-sectional survey to illustrate the challenges surrounding the evaluation of a complex national public health intervention.
An online initial and follow-up questionnaire survey assessed the knowledge, attitudes and practices of public health staff, physicians and nurses working in TB control in November 2010 and March 2012. It included questions on the implementation, experience and uptake of the TB-STS. Participants that responded to both surveys were included in the analysis.
248 participants responded to the initial survey and 137 of these responded to the follow-up survey (56% retention).
Knowledge: A significant increase in knowledge was observed, including a rise in the proportion of respondents who had received training (28.6% to 67.9%, p = 0.003), and the self-rated knowledge of how to use strain typing had improved (‘no knowledge’ decreased from 43.2% to 27.4%).
Attitudes: The majority of respondents found strain typing useful; the proportion that reported strain typing to be useful was similar across the two surveys (95.7% to 94.7%, p = 0.67).
Practices: There were significant increases between the initial and follow-up surveys in the number of respondents who reported using strain typing (57.0% to 80.5%, p < 0.001) and the proportion of time health protection staff spent on investigating TB (2.74% to 7.08%, p = 0.04).
Evaluation of a complex public health intervention is challenging. In this example, the immediate national roll-out of the TB-STS meant that a controlled survey design was not possible. This study informs the future development of the TB-STS by identifying the need for training to reach wider professional groups, and argues for its continuation based on service users’ perception that it is useful. By highlighting the importance of a well-defined sampling frame, collecting baseline information, and including all stakeholders, it provides lessons for the implementation of similar services in other countries and future evaluations of public health interventions.
Complex public health interventions – interventions involving multiple interacting components – when applied at a national level, are often implemented in a way that makes evaluating them with rigorously designed trials difficult . Instead, they require a more pragmatic approach using the available data .
Molecular typing of Mycobacterium tuberculosis is a tool for TB surveillance and control. It has been used in combination with epidemiological information to identify outbreaks , identify new routes of transmission , refute suspected transmission [5, 6], evaluate TB control programmes [7, 8] and detect laboratory cross contamination [9, 10].
The National Tuberculosis Strain Typing Service (TB-STS) is a complex public health intervention involving laboratory, public health and clinical services across England and was introduced in January 2010 . A mixed-method prospective evaluation of the acceptability, implementation, effectiveness and cost-effectiveness of the service was undertaken . Here we report in detail on one component of the evaluation: a cross-sectional initial and follow-up survey of those delivering and using the TB-STS to assess their knowledge, and to understand the impact of the service on changes in attitudes and practices associated with strain typing.
A full description of the TB-STS, with laboratory guidelines for MIRU-VNTR strain typing and reporting  and a handbook for public health actions relating to cluster investigations (TB strain typing and cluster investigation handbook ) can be found on the Health Protection Agency website . Briefly, the TB-STS involves prospectively typing the first M. tuberculosis isolate from every culture-confirmed tuberculosis (TB) patient using 24 locus Mycobacterial Interspersed Repetitive Units-Variable Number Tandem Repeats (MIRU-VNTR), a standardised molecular typing method . Based on the strain type result, patients are grouped into ‘clusters’ [13, 14] which are reported to the Health Protection Units (HPUs). If a cluster meets a certain threshold, as outlined in the TB strain typing and cluster investigation handbook,  then a cluster investigation is launched to try to establish epidemiological links between the clustered patients, thereby identifying the transmission setting and/or an outbreak. As part of a cluster investigation the HPU may decide to carry out enhanced contact tracing or screening around the patients in the cluster or the identified transmission setting. By combining patients’ strain type with epidemiological information the TB-STS aims to inform public health decision-making at the local level.
The various components of the TB-STS were implemented at different times (but always on a national scale): prospective strain typing was introduced across England in January 2010; one cluster investigator was appointed in January 2010 and the remaining two were appointed in January 2011; the training programme for health protection staff working in HPUs was carried out between January 2011 and February 2012, consisting of a seminar at the national Health Protection Conference, an online seminar, a workshop conducted at each HPU, the publication of the handbook  and a Q&A sheet  (in December 2010); and the software linking patients’ electronic TB record and strain typing result with information from clusters investigations was not developed during this study period.
An initial survey was conducted in November 2010 and a follow-up survey in March 2012 using a web-based survey questionnaire (http://www.objectplanet.com/opinio). The target population were all public health staff, chest/respiratory physicians and TB nurses working in TB control in England. Questions were asked about the knowledge (awareness of the service, training, resources and self-reported knowledge), attitudes (perceived usefulness of the service) and practice (if and how strain typing is accessed and used, and its associated workload). All questions and possible responses are available in the appendix (Additional file 1). The survey was piloted with a nurse, a physician and a public health specialist. The initial survey was emailed to all users of the TB notification system  and to staff responsible for TB control in HPUs who were asked to pass it on to their local TB teams; the sampling frame could not be enumerated. The follow-up survey was emailed to respondents to the initial survey.
Participants that responded to both surveys were included in the statistical analysis. Responses from people working at national, regional or PCT-level, including cluster investigators, and people working in Wales were excluded from this analysis. We compared the knowledge, attitudes and practices of public health and clinical staff working on TB control in the initial and follow-up surveys by calculating and comparing medians and inter-quartile ranges (IQR), and means and standard deviations (SD), and using two-sample t-tests, chi2 tests and logistic regression, where appropriate. Calculations exclude item non-responses. Analyses were conducted overall, by professional category and the TB incidence of the HPU area in which respondents worked (low, medium and high incidence defined as an annual notification rate of <10/100,000 population, 10 to 19/100,000 and ≥20/100,000 respectively).
The study was classified as a service evaluation by University College London Hospital Foundation Trust therefore specific ethical approval was not required.
There were 248 responses to the initial survey, 137 responses to the follow-up survey (55% retention), and for 124 we have responses to both the initial and follow-up surveys (Figure 1). Respondents to the initial survey who did not respond to the follow-up survey were not significantly different to those that responded to both: no particular profession, full-time/part-time position or those working in areas with different TB incidences was more (or less) likely to respond to the follow-up survey, and there was no significant difference between the proportion of people who had heard of the TB-STS or had access to strain typing at the time of the initial survey (Table 1). Respondents were from all nine regions of England and covered 24 (of 26) HPUs.
Between the initial and follow-up surveys there were increases in the proportion of respondents who had heard of the TB-STS, had access to strain typing results, had received training, and had access to training resources (Table 2). The self-rated knowledge of how to use strain typing also increased over time (Figure 2). Nurses reported lower average knowledge in both surveys compared to physicians and health protection staff.
69 people (69/124 = 56%) from the initial survey and 95 people (95/124 = 77%) from the follow-up survey reported that they used strain typing. Opinions of the usefulness of TB strain typing was high amongst all respondents and did not change between the surveys (95.7% to 94.7%, p = 0.667; Table 3). A greater proportion of respondents from low TB incidence areas found strain typing useful, compared to those working in high TB incidence areas (97.4% compared to 89.3% in the follow-up survey, respectively), though this result was not statistically significant (OR = 0.13, 95% CI 0.014-1.128, p = 0.075).
Figure 3 shows a significant increase in the number of respondents that reported using strain typing between the initial and follow-up surveys. There was an increase in the number of respondents who reported using strain typing to identify links between cases (65.3% to 78.2%, p = 0.02; the most common use), disprove links between cases (46.8% to 58.9%, p = 0.06) and to justify stopping contact tracing (20.2% to 30.7%, p = 0.06) (Table 4).
Table 5 shows workload reported by nurses and health protection staff. For the nurses, no significant changes in contact tracing workload were reported.
Health protection staff reported a significant increase in the mean number of investigations initiated because of epidemiological links between patients over a three month period (mean 0.5 to 2.8, p = 0.04) and the mean number of these investigations for which strain typing was used to provide more information (0.6 to 1.8, p = 0.03), but there was no change in the number that were influenced by the strain typing (1.2 to 0.4, p = 0.17). There was no reported difference in the number of clusters investigated because of their strain type (in high incidence areas a large, but non-significant, decrease was reported) and the number of strain typing investigations that identified epidemiological links between cases remained low (Table 5). Overall, the proportion of time health protection staff spent on cluster investigations increased significantly (from 2.7% to 7.2%, p = 0.04).
There was no reported change over time in the frequency at which physicians were called to incident meetings (a meeting, often multi-disciplinary, held to discuss a TB patient, group or cluster of cases that are of particular concern) (p = 0.503; most reported once every three months or less (65.5% at in the initial survey and 67.9% at follow-up)) and there was no change in the number of physicians who reported strain typing as being relevant to an incident meeting (57.8% to 55.6%, p = 0.875).
We present results from an initial and follow-up survey assessing the knowledge, attitudes and practices of those implementing and using the TB-STS. There were 124 responses to both the surveys, representing health protection staff and clinic-based physicians and nurses from 24 (of 26) HPUs across England. Strain typing was used by more people, and an increase in knowledge of the TB-STS was reported at the follow-up survey. A change in attitude was not measured as the majority of respondents found strain typing useful to them at both time points. With respect to workload associated with the TB-STS, there was no change over time in the contact tracing activities of nurses or the frequency of incident meetings attended by physicians; however the proportion of time health protection staff spent on investigating TB transmission increased significantly. Despite strain typing being used to provide more information to public health staff at follow-up, there was no increase in epidemiological links identified.
How this relates to previous studies
National TB strain typing services have been introduced in other countries [17–20], but the knowledge, attitudes and practices of users have not been evaluated. However, the impact of strain typing on contact tracing activities in the Netherlands has been assessed . Consistent with this study, we found no change in the workload associated with strain typing for nurses and physicians, even though strain typing was used by more people at the follow-up survey (indicating the successful roll-out of the service). This may be because it is difficult to measure marginal changes in workload associated with a particular service where the workforce is already working at full capacity. Health protection staff, however, spent a greater proportion of time on cluster investigations. Given that the Handbook had not been published and all the cluster investigation coordinators were not in position at the time of the initial survey, this is not surprising and suggests that the TB-STS had been integrated into the TB control activities of the HPUs.
Based on evidence from the USA one would expect more possible transmission links to be identified when strain typing informs contact tracing activities [6, 8]. However, in this study we found the proportion of time health protection staff spent on cluster investigations increased and the number of investigations that used strain typing increased, but there was no increase in the reported number of possible transmission links found between clustered cases. This discordance between the findings on subjective report of utility and the public health outcomes reported could be because the current methods used by public health staff to identify epidemiological links may be inappropriate or ineffective, or there may have been an increase in suspected (but not established) transmission because of the strain typing information. For the TB-STS to have a public health impact and reduce TB transmission, cluster investigations would have to lead to the detection of previously unidentified latently infected and active TB cases.
The way the TB-STS was implemented and the survey design have resulted in a number of limitations that provide important lessons for the TB-STS, the evaluation of future services and other complex interventions.
Firstly, the survey was developed after the initiation of the TB-STS so baseline information could not be collected and we are likely to have underestimated the difference between the surveys. However, the initial survey was conducted before the roll-out of any training for the TB-STS and prior to the employment of all national staff to coordinate cluster investigations. An alternative study design, which would have had a control group, would have been possible if the TB-STS was rolled out in a step-wise process across the country, rather than nationally.
Secondly, the target population for the survey was all public health staff, physicians and nurses working in TB control in England. It was not possible to enumerate the sampling frame because no formal or informal register of clinical and health protection staff working in TB could be identified. As a result, we could not calculate a response rate.
Finally, the 50% retention rate between the surveys is quite low and we may have lost the opinions and experiences of a particular group of people. However, non-responders to the follow-up survey did not differ significantly to those that responded to both surveys based profession or burden of TB in their geographical area. Because the study was conducted as part of a programmatic service implementation, results must be interpreted accordingly.
The findings of this survey inform the development of the TB-STS and the design of future evaluations. Despite a significant increase in the number of health protection staff who had received training, there remained some that had not received any, suggesting the need for an ongoing training programme that also takes into account turnover of staff. Self-reported knowledge of how to use the strain typing information was lower for nurses compared with physicians and health protection staff, possibly representing a gap in the training strategy, which did not include nurses. The finding that physicians had the highest self-reported knowledge across the two surveys, even though they were not included in the training strategy, might be because they have had access to information on strain typing from other sources and, relative to nurses and public health staff, might self-rate their knowledge higher.
The perception of usefulness did not change over time as most people found strain typing to be useful in both surveys. This suggests that any changes in practice are due to increasing knowledge and access to strain typing, rather than attitudes towards strain typing. Therefore, to improve use and impact of the TB-STS, there should be a focus on improving training and making strain typing data easily accessible so that it can become better integrated into the TB service.
The findings of this survey argue for the continuation of the TB-STS. A majority of people reported the TB-STS to be useful and health protection staff reported an increase in the number of investigations for which strain typing was used to provide more information, although there was no increase in the number of investigations that were influenced by strain typing. This discordance between the findings on subjective report of utility and the investigation outcomes reported may signify the high value placed on information.
When implementing a public health intervention and planning an evaluation it is essential to have a well-defined sampling frame and a baseline that can be measured before the start of the service implementation. Where possible, the evaluation of a service should start prior to its implementation in order to capture the baseline and to design the evaluation based on the planned service implementation. This survey is an example of where this was not possible and highlights the importance of acknowledging the context in which the service was implemented, both for assessing its success and understanding the limitations of the evaluation design.
The variation in knowledge, attitudes and practices across the professions illustrates the importance of including all the service stakeholders in the evaluation. For example, in the TB-STS, nurse respondents reported lower knowledge, suggesting that they could benefit from being included in the training strategy.
This survey is the first component of the evaluation of the TB-STS. To better understand the public health utility and evaluate the impact of such a service, a comprehensive mixed-methods evaluation is underway . This includes modelling of the effectiveness and cost-effectiveness and qualitative studies.
Evaluating a complex public health intervention requires a pragmatic approach, taking into account how the service has been implemented. In these initial and follow-up surveys, public health staff, physicians and TB nurses found the TB-STS useful and increased the amount they used it in the first two years of the service, arguing for the continuation of the service. Despite this, the impact of the TB-STS on cluster investigations remained unclear. We recommend continuing the service but with ongoing and more thorough training of service users and focussing on improving knowledge and making data more accessible. Future evaluations of complex interventions should be initiated prior to the implementation of the service, and would benefit from an enumerable sampling frame and a measurable baseline.
Health Protection Unit
Mycobacterial interspersed repetitive units-variable number tandem repeats
National Health Service
Public Health England
Tuberculosis strain typing service.
Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P: Framework for design and evaluation of complex interventions to improve health. BMJ. 2000, 321: 694-696. 10.1136/bmj.321.7262.694.
Petticrew M: When are complex interventions "complex"? When are simple interventions "simple"?. Eur J Public Health. 2011, 21: 397-398. 10.1093/eurpub/ckr084.
Ruddy MC, Davies AP, Yates MD, Yates S, Balasegaram S, Drabu Y, Patel B, Lozewicz S, Sen S, Bahl M, James E, Lipman M, Duckworth G, Watson JM, Piper M, Drobniewski FA, Maguire H: Outbreak of isoniazid resistant tuberculosis in north London. Thorax. 2004, 59: 279-285. 10.1136/thx.2003.010405.
Malakmadze N, González IM, Oemig T, Isiadinso I, Rembert D, McCauley MM, Wand P, Diem L, Cowan L, Palumbo GJ, Fraser M, Ijaz K: Unsuspected Recent Transmission of Tuberculosis among High-Risk Groups: Implications of Universal Tuberculosis Genotyping in Its Detection. Clin Infect Dis. 2005, 40: 366-373. 10.1086/427112.
Lambregts-van Weezenbeek CSB, Sebek MMGG, van Gerven PJHJ, de Vries G, Verver S, Kalisvaart NA, van Soolingen D: Tuberculosis contact investigation and DNA fingerprint surveillance in The Netherlands: 6 years’ experience with nation-wide cluster feedback and cluster monitoring. Int J Tuberc Lung Dis. 2003, 7 (12 Suppl 3): S463-S470.
McNabb SJN, Kammerer JS, Hickey AC, Braden CR, Shang N, Rosenblum LS, Navin TR: Added epidemiologic value to tuberculosis prevention and control of the investigation of clustered genotypes of Mycobacterium tuberculosis isolates. Am J Epidemiol. 2004, 160: 589-597. 10.1093/aje/kwh253.
De Vries G, van Hest RAH, Richardus JH: Impact of mobile radiographic screening on tuberculosis among drug users and homeless persons. Am J Respir Crit Care Med. 2007, 176: 201-207. 10.1164/rccm.200612-1877OC.
Clark CM, Driver CR, Munsiff SS, Driscoll JR, Kreiswirth BN, Zhao B, Ebrahimzadeh A, Salfinger M, Piatek AS, Abdelwahab J: Universal genotyping in tuberculosis control program, New York City, 2001–2003. Emerg Infect Dis. 2006, 12: 719-724. 10.3201/eid1205.050446.
Ruddy M, McHugh TD, Dale JW, Banerjee D, Maguire H, Wilson P, Drobniewski F, Butcher P, Gillespie SH: Estimation of the Rate of Unrecognized Cross-Contamination with Mycobacterium tuberculosis in London Microbiology Laboratories. J Clin Microbiol. 2002, 40: 4100-4104. 10.1128/JCM.40.11.4100-4104.2002.
Jasmer RM, Roemer M, Hamilton J, Bunter J, Braden CR, Shinnick TM, Desmond EP: A Prospective, Multicenter Study of Laboratory Cross-Contamination of Mycobacterium tuberculosis Cultures. Emerg Infect Dis. 2002, 8: 1260-1263. 10.3201/eid0811.020298.
Public Health England: National Tuberculosis Strain Typing Service website. http://webarchive.nationalarchives.gov.uk/20140629102627/http://www.hpa.org.uk/Topics/InfectiousDiseases/InfectionsAZ/Tuberculosis/NationalTuberculosisStrainTypingService/,
Mears J, Vynnycky E, Lord J, Borgdorff M, Cohen T, Abubakar I, Sonnenberg P: Evaluation of the Tuberculosis Strain Typing Service (TB-STS) in England. Lancet. 2013, 382 (Supplement 3): S73-[Public Health Science]
Brown T, Evans JT, Sails AD TBDB, Anderson LF, Stone M, Jebbari H, Nikolayevsky V: HPA Mycobacterium Tuberculosis Strain Typing: a Guide to Data Production and Distribution. 2012
Public Health England: TB Strain Typing and Cluster Investigation Handbook. 2014, 3
Supply P, Allix C, Lesjean S, Cardoso-Oelemann M, Rüsch-Gerdes S, Willery E, Savine E, de Haas P, van Deutekom H, Roring S, Bifani P, Kurepina N, Kreiswirth B, Sola C, Rastogi N, Vatin V, Gutierrez MC, Fauville M, Niemann S, Skuce R, Kremer K, Locht C, van Soolingen D: Proposal for standardization of optimized mycobacterial interspersed repetitive unit-variable-number tandem repeat typing of Mycobacterium tuberculosis. J Clin Microbiol. 2006, 44: 4498-4510. 10.1128/JCM.01392-06.
Van Buynder P: Enhanced surveillance of tuberculosis in England and Wales: circling the wagons?. Commun Dis Public Health. 1998, 1: 219-220.
Bauer J, Kok-Jensen A, Faurschou P, Thuesen J, Taudorf E, Andersen AB: A prospective evaluation of the clinical value of nation-wide DNA fingerprinting of tuberculosis isolates in Denmark. Int J Tuberc Lung Dis. 2000, 4: 295-299.
Bidovec-Stojkovic U, Zolnir-Dovc M, Supply P: One year nationwide evaluation of 24-locus MIRU-VNTR genotyping on Slovenian Mycobacterium tuberculosis isolates. Respir Med. 2011, 105 (Suppl 1): S67-S73.
Van Soolingen D, Borgdorff MW, de Haas PE, Sebek MM, Veen J, Dessens M, Kremer K, van Embden JD: Molecular epidemiology of tuberculosis in the Netherlands: a nationwide study from 1993 through 1997. J Infect Dis. 1999, 180: 726-736. 10.1086/314930.
Centers for Disease Control and Prevention: New CDC Program for Rapid Genotyping of Mycobacterium Tuberculosis Isolates. JAMA. 2005, 293: 2086-
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/14/1023/prepub
The authors would like to acknowledge Public Health England’s TB Strain Typing Project Board for their support for this evaluation and all those within PHE and the NHS who cascaded and responded to the survey. JM is funded by University College London and Public Health England Impact Studentship. IA is funded by an NIHR Senior Research Fellowship.
JM, IA, HM, ML and EV have been employed by Public Health England in the last 5 years. Public Health England played no role in the study design, data interpretation, data analysis, writing of the report or the decision to submit for publication. There are no other potential conflicts of interest.
All authors contributed to the original conception, design and editing of the survey, and were involved in the interpretation of the survey results. JM drafted and conducted the survey, and analysed the data. IA and PS contributed to the survey analysis. The paper was drafted by JM, with further comments from all other authors. All authors have given final approval of the manuscript. The authors were all part of the TB-STS Evaluation Group.
Electronic supplementary material
About this article
Cite this article
Mears, J., Abubakar, I., Crisp, D. et al. Prospective evaluation of a complex public health intervention: lessons from an initial and follow-up cross-sectional survey of the tuberculosis strain typing service in England. BMC Public Health 14, 1023 (2014). https://doi.org/10.1186/1471-2458-14-1023
- Strain typing
- Complex intervention
- Service evaluation