Prospective evaluation of a complex public health intervention: lessons from an initial and follow-up cross-sectional survey of the tuberculosis strain typing service in England

Background The national tuberculosis strain typing service (TB-STS) was introduced in England in 2010. The TB-STS involves MIRU-VNTR typing of isolates from all TB patients for the prospective identification, reporting and investigation of TB strain typing clusters. As part of a mixed-method evaluation, we report on a repeated cross-sectional survey to illustrate the challenges surrounding the evaluation of a complex national public health intervention. Methods An online initial and follow-up questionnaire survey assessed the knowledge, attitudes and practices of public health staff, physicians and nurses working in TB control in November 2010 and March 2012. It included questions on the implementation, experience and uptake of the TB-STS. Participants that responded to both surveys were included in the analysis. Results 248 participants responded to the initial survey and 137 of these responded to the follow-up survey (56% retention). Knowledge: A significant increase in knowledge was observed, including a rise in the proportion of respondents who had received training (28.6% to 67.9%, p = 0.003), and the self-rated knowledge of how to use strain typing had improved (‘no knowledge’ decreased from 43.2% to 27.4%). Attitudes: The majority of respondents found strain typing useful; the proportion that reported strain typing to be useful was similar across the two surveys (95.7% to 94.7%, p = 0.67). Practices: There were significant increases between the initial and follow-up surveys in the number of respondents who reported using strain typing (57.0% to 80.5%, p < 0.001) and the proportion of time health protection staff spent on investigating TB (2.74% to 7.08%, p = 0.04). Conclusions Evaluation of a complex public health intervention is challenging. In this example, the immediate national roll-out of the TB-STS meant that a controlled survey design was not possible. This study informs the future development of the TB-STS by identifying the need for training to reach wider professional groups, and argues for its continuation based on service users’ perception that it is useful. By highlighting the importance of a well-defined sampling frame, collecting baseline information, and including all stakeholders, it provides lessons for the implementation of similar services in other countries and future evaluations of public health interventions. Electronic supplementary material The online version of this article (doi:10.1186/1471-2458-14-1023) contains supplementary material, which is available to authorized users.


Background
Complex public health interventionsinterventions involving multiple interacting componentswhen applied at a national level, are often implemented in a way that makes evaluating them with rigorously designed trials difficult [1]. Instead, they require a more pragmatic approach using the available data [2].
Molecular typing of Mycobacterium tuberculosis is a tool for TB surveillance and control. It has been used in combination with epidemiological information to identify outbreaks [3], identify new routes of transmission [4], refute suspected transmission [5,6], evaluate TB control programmes [7,8] and detect laboratory cross contamination [9,10].
The National Tuberculosis Strain Typing Service (TB-STS) is a complex public health intervention involving laboratory, public health and clinical services across England and was introduced in January 2010 [11]. A mixed-method prospective evaluation of the acceptability, implementation, effectiveness and cost-effectiveness of the service was undertaken [12]. Here we report in detail on one component of the evaluation: a crosssectional initial and follow-up survey of those delivering and using the TB-STS to assess their knowledge, and to understand the impact of the service on changes in attitudes and practices associated with strain typing.

Intervention
A full description of the TB-STS, with laboratory guidelines for MIRU-VNTR strain typing and reporting [13] and a handbook for public health actions relating to cluster investigations (TB strain typing and cluster investigation handbook [14]) can be found on the Health Protection Agency website [11]. Briefly, the TB-STS involves prospectively typing the first M. tuberculosis isolate from every culture-confirmed tuberculosis (TB) patient using 24 locus Mycobacterial Interspersed Repetitive Units-Variable Number Tandem Repeats (MIRU-VNTR), a standardised molecular typing method [15]. Based on the strain type result, patients are grouped into 'clusters' [13,14] which are reported to the Health Protection Units (HPUs). If a cluster meets a certain threshold, as outlined in the TB strain typing and cluster investigation handbook, [14] then a cluster investigation is launched to try to establish epidemiological links between the clustered patients, thereby identifying the transmission setting and/or an outbreak. As part of a cluster investigation the HPU may decide to carry out enhanced contact tracing or screening around the patients in the cluster or the identified transmission setting. By combining patients' strain type with epidemiological information the TB-STS aims to inform public health decision-making at the local level.
The various components of the TB-STS were implemented at different times (but always on a national scale): prospective strain typing was introduced across England in January 2010; one cluster investigator was appointed in January 2010 and the remaining two were appointed in January 2011; the training programme for health protection staff working in HPUs was carried out between January 2011 and February 2012, consisting of a seminar at the national Health Protection Conference, an online seminar, a workshop conducted at each HPU, the publication of the handbook [14] and a Q&A sheet [11] (in December 2010); and the software linking patients' electronic TB record and strain typing result with information from clusters investigations was not developed during this study period.

Study design
An initial survey was conducted in November 2010 and a follow-up survey in March 2012 using a web-based survey questionnaire (www.objectplanet.com/opinio). The target population were all public health staff, chest/ respiratory physicians and TB nurses working in TB control in England. Questions were asked about the knowledge (awareness of the service, training, resources and self-reported knowledge), attitudes (perceived usefulness of the service) and practice (if and how strain typing is accessed and used, and its associated workload). All questions and possible responses are available in the appendix (Additional file 1). The survey was piloted with a nurse, a physician and a public health specialist. The initial survey was emailed to all users of the TB notification system [16] and to staff responsible for TB control in HPUs who were asked to pass it on to their local TB teams; the sampling frame could not be enumerated. The follow-up survey was emailed to respondents to the initial survey.

Analysis
Participants that responded to both surveys were included in the statistical analysis. Responses from people working at national, regional or PCT-level, including cluster investigators, and people working in Wales were excluded from this analysis. We compared the knowledge, attitudes and practices of public health and clinical staff working on TB control in the initial and follow-up surveys by calculating and comparing medians and inter-quartile ranges (IQR), and means and standard deviations (SD), and using two-sample t-tests, chi 2 tests and logistic regression, where appropriate. Calculations exclude item non-responses. Analyses were conducted overall, by professional category and the TB incidence of the HPU area in which respondents worked (low, medium and high incidence defined as an annual notification rate of <10/100,000 population, 10 to 19/100,000 and ≥20/100,000 respectively).

Ethical considerations
The study was classified as a service evaluation by University College London Hospital Foundation Trust therefore specific ethical approval was not required.

Survey responses
There were 248 responses to the initial survey, 137 responses to the follow-up survey (55% retention), and for 124 we have responses to both the initial and follow-up surveys ( Figure 1). Respondents to the initial survey who did not respond to the follow-up survey were not significantly different to those that responded to both: no particular profession, full-time/part-time position or those working in areas with different TB incidences was more (or less) likely to respond to the follow-up survey, and there was no significant difference between the proportion of people who had heard of the TB-STS or had access to strain typing at the time of the initial survey (Table 1). Respondents were from all nine regions of England and covered 24 (of 26) HPUs.

Knowledge
Between the initial and follow-up surveys there were increases in the proportion of respondents who had heard of the TB-STS, had access to strain typing results, had received training, and had access to training resources ( Table 2). The self-rated knowledge of how to use strain typing also increased over time (Figure 2). Nurses reported lower average knowledge in both surveys compared to physicians and health protection staff.
Attitudes 69 people (69/124 = 56%) from the initial survey and 95 people (95/124 = 77%) from the follow-up survey reported that they used strain typing. Opinions of the usefulness of TB strain typing was high amongst all respondents and did not change between the surveys (95.7% to 94.7%, p = 0.667; Table 3). A greater proportion of respondents from low TB incidence areas found strain typing useful, compared to those working in high TB incidence areas (97.4% compared to 89.3% in the follow-up survey, respectively), though this result was not statistically significant (OR = 0.13, 95% CI 0.014-1.128, p = 0.075).  Figure 1 Responses to the initial and follow-up surveys. a The email was sent to all users of the Enhanced TB Surveillance database. This included all administrative staff as well as well as staff working at national, regional and Primary Care Trust level, for whom the survey may not be relevant. b It is not known how many people received the email via through the HPU cascade. c This response rate is an underestimation because of the denominator used. d Respondents working at national, regional or PCT-level (n = 27) and those from Wales (n = 9) were excluded from this analysis. e Email addresses not available from the initial survey (n = 2). f In some cases it was not possible to link the follow-up response to the initial response (n = 10). Respondents working at national, regional or PCT-level and those from Wales (n = 4) were excluded from this analysis. Figure 3 shows a significant increase in the number of respondents that reported using strain typing between the initial and follow-up surveys. There was an increase in the number of respondents who reported using strain typing to identify links between cases (65.3% to 78.2%, p = 0.02; the most common use), disprove links between cases (46.8% to 58.9%, p = 0.06) and to justify stopping contact tracing (20.2% to 30.7%, p = 0.06) ( Table 4). Table 5 shows workload reported by nurses and health protection staff. For the nurses, no significant changes in contact tracing workload were reported.

Practices
Health protection staff reported a significant increase in the mean number of investigations initiated because of Area where respondents worked is defined as low, medium and high TB incidence: <10/100,000, 10-19/100,000, ≥20/100,000 population, respectively. There were no significant differences between characteristics of non-responders and responders, including access to strain typing (81.8 % vs. 72.6 %, chi 2 test p = 0.085).   The following question was asked to respondents who reported that they used strain typing data for TB control ( One response was missing from the follow-up survey. c Area where respondents worked is defined as low, medium and high TB incidence: <10/100,000, 10-19/100,000, ≥20/100,000 population, respectively. d chi 2 test for significance comparing responses from the initial and follow-up surveys, missing items were excluded. epidemiological links between patients over a three month period (mean 0.5 to 2.8, p = 0.04) and the mean number of these investigations for which strain typing was used to provide more information (0.6 to 1.8, p = 0.03), but there was no change in the number that were influenced by the strain typing (1.2 to 0.4, p = 0.17). There was no reported difference in the number of clusters investigated because of their strain type (in high incidence areas a large, but nonsignificant, decrease was reported) and the number of strain typing investigations that identified epidemiological links between cases remained low (Table 5). Overall, the proportion of time health protection staff spent on cluster investigations increased significantly (from 2.7% to 7.2%, p = 0.04).
There was no reported change over time in the frequency at which physicians were called to incident meetings (a meeting, often multi-disciplinary, held to discuss a TB patient, group or cluster of cases that are of particular concern) (p = 0.503; most reported once every three months or less (65.5% at in the initial survey and 67.9% at follow-up)) and there was no change in the number of physicians who reported strain typing as being relevant to an incident meeting (57.8% to 55.6%, p = 0.875).

Main findings
We present results from an initial and follow-up survey assessing the knowledge, attitudes and practices of those implementing and using the TB-STS. There were 124 responses to both the surveys, representing health protection staff and clinic-based physicians and nurses from 24 (of 26) HPUs across England. Strain typing was used by more people, and an increase in knowledge of the TB-STS was reported at the follow-up survey. A change in attitude was not measured as the majority of respondents found strain typing useful to them at both time points. With respect to workload associated with the TB-STS, there was no change over time in the contact tracing activities of nurses or the frequency of incident meetings attended by physicians; however the proportion of time health protection staff spent on investigating TB transmission increased significantly. Despite strain typing being used to provide more information to public health staff at follow-up, there was no increase in epidemiological links identified.

How this relates to previous studies
National TB strain typing services have been introduced in other countries [17][18][19][20], but the knowledge, attitudes and practices of users have not been evaluated. However, the impact of strain typing on contact tracing activities in the Netherlands has been assessed [5]. Consistent with this study, we found no change in the workload associated with strain typing for nurses and physicians, even though strain typing was used by more people at the follow-up survey (indicating the successful roll-out of the service). This may be because it is difficult to measure marginal changes in workload associated with a particular service where the workforce is already working at full capacity. Health protection staff, however, spent a greater proportion of time on cluster investigations. Given that the Handbook had not been published and all the cluster investigation coordinators were not in position at the time of the initial survey, this is not surprising and suggests that the TB-STS had been integrated into the TB control activities of the HPUs. Based on evidence from the USA one would expect more possible transmission links to be identified when What do you use strain typing for? (multiple selections possible) (Don't know / Identify clusters and links between cases / Disprove clusters and links between cases / Justify extended contact tracing / Justify stopping contact tracing / To provide more information / Other (please specify)). b Area where respondents worked is defined as low, medium and high TB incidence: <10/100,000, 10-19/100,000, ≥20/100,000 population, respectively. c chi 2 test for significance comparing responses from the initial and follow-up surveys.    strain typing informs contact tracing activities [6,8]. However, in this study we found the proportion of time health protection staff spent on cluster investigations increased and the number of investigations that used strain typing increased, but there was no increase in the reported number of possible transmission links found between clustered cases. This discordance between the findings on subjective report of utility and the public health outcomes reported could be because the current methods used by public health staff to identify epidemiological links may be inappropriate or ineffective, or there may have been an increase in suspected (but not established) transmission because of the strain typing information. For the TB-STS to have a public health impact and reduce TB transmission, cluster investigations would have to lead to the detection of previously unidentified latently infected and active TB cases.

Limitations
The way the TB-STS was implemented and the survey design have resulted in a number of limitations that provide important lessons for the TB-STS, the evaluation of future services and other complex interventions. Firstly, the survey was developed after the initiation of the TB-STS so baseline information could not be collected and we are likely to have underestimated the difference between the surveys. However, the initial survey was conducted before the roll-out of any training for the TB-STS and prior to the employment of all national staff to coordinate cluster investigations. An alternative study design, which would have had a control group, would have been possible if the TB-STS was rolled out in a step-wise process across the country, rather than nationally.
Secondly, the target population for the survey was all public health staff, physicians and nurses working in TB control in England. It was not possible to enumerate the sampling frame because no formal or informal register of clinical and health protection staff working in TB could be identified. As a result, we could not calculate a response rate.
Finally, the 50% retention rate between the surveys is quite low and we may have lost the opinions and experiences of a particular group of people. However, non-responders to the follow-up survey did not differ significantly to those that responded to both surveys based profession or burden of TB in their geographical area. Because the study was conducted as part of a programmatic service implementation, results must be interpreted accordingly.

Recommendations
The findings of this survey inform the development of the TB-STS and the design of future evaluations. Despite a significant increase in the number of health protection staff who had received training, there remained some that had not received any, suggesting the need for an ongoing training programme that also takes into account turnover of staff. Self-reported knowledge of how to use the strain typing information was lower for nurses compared with physicians and health protection staff, possibly representing a gap in the training strategy, which did not include nurses. The finding that physicians had the highest self-reported knowledge across the two surveys, even though they were not included in the training strategy, might be because they have had access to information on strain typing from other sources and, relative to nurses and public health staff, might self-rate their knowledge higher.
The perception of usefulness did not change over time as most people found strain typing to be useful in both surveys. This suggests that any changes in practice are due to increasing knowledge and access to strain typing, rather than attitudes towards strain typing. Therefore, to improve use and impact of the TB-STS, there should be a focus on improving training and making strain typing data easily accessible so that it can become better integrated into the TB service.
The findings of this survey argue for the continuation of the TB-STS. A majority of people reported the TB-STS to be useful and health protection staff reported an increase in the number of investigations for which strain typing was used to provide more information, although there was no increase in the number of investigations that were influenced by strain typing. This discordance between the findings on subjective report of utility and the investigation outcomes reported may signify the high value placed on information.
When implementing a public health intervention and planning an evaluation it is essential to have a welldefined sampling frame and a baseline that can be measured before the start of the service implementation. Where possible, the evaluation of a service should start prior to its implementation in order to capture the baseline and to design the evaluation based on the planned service implementation. This survey is an example of where this was not possible and highlights the importance of acknowledging the context in which the service was implemented, both for assessing its success and understanding the limitations of the evaluation design.
The variation in knowledge, attitudes and practices across the professions illustrates the importance of including all the service stakeholders in the evaluation. For example, in the TB-STS, nurse respondents reported lower knowledge, suggesting that they could benefit from being included in the training strategy.
This survey is the first component of the evaluation of the TB-STS. To better understand the public health utility and evaluate the impact of such a service, a comprehensive mixed-methods evaluation is underway [12]. This includes modelling of the effectiveness and cost-effectiveness and qualitative studies.

Conclusions
Evaluating a complex public health intervention requires a pragmatic approach, taking into account how the service has been implemented. In these initial and follow-up surveys, public health staff, physicians and TB nurses found the TB-STS useful and increased the amount they used it in the first two years of the service, arguing for the continuation of the service. Despite this, the impact of the TB-STS on cluster investigations remained unclear. We recommend continuing the service but with ongoing and more thorough training of service users and focussing on improving knowledge and making data more accessible. Future evaluations of complex interventions should be initiated prior to the implementation of the service, and would benefit from an enumerable sampling frame and a measurable baseline. Competing interests JM, IA, HM, ML and EV have been employed by Public Health England in the last 5 years. Public Health England played no role in the study design, data interpretation, data analysis, writing of the report or the decision to submit for publication. There are no other potential conflicts of interest.

Authors' contributions
All authors contributed to the original conception, design and editing of the survey, and were involved in the interpretation of the survey results. JM drafted and conducted the survey, and analysed the data. IA and PS contributed to the survey analysis. The paper was drafted by JM, with further comments from all other authors. All authors have given final approval of the manuscript. The authors were all part of the TB-STS Evaluation Group.