Process evaluation results of a cluster randomised controlled childhood obesity prevention trial: the WAVES study

The rapid increase in childhood obesity over a relatively short time period, with its associated adverse health and social consequences, is a serious challenge to public health [1]. Interventions aimed at preventing the upward trend in obesity prevalence have been developed and evaluated, with varying success [2]. The importance of process evaluation in public health intervention research is increasingly recognised [3]. Assessing whether interventions are delivered as intended, and factors affecting implementation, allows researchers to add context to the interpretation of intervention outcomes, and policy makers to optimise future implementation of interventions. The release of the Medical Research Council (MRC) framework - Process evaluation of complex interventions [3] - provides much-needed guidance for a structured approach to undertaking evaluations of health-related interventions, by considering the key elements and how they interrelate. To move beyond simple documentation of intervention delivery and inform implementation and mechanisms of impact, a clear understanding is needed of how the intervention is delivered, the level of implementation and the context within which it is delivered [4].

Whilst reporting of process evaluation in health research is improving, a lack of uniformity in approach remains [5, 6]. Process measures have often focused on a limited number of dimensions, such as reach, dose and fidelity [7], or just dose alone [8]. Similarly, data collection methods limited to questionnaires [9, 10], survey data [11], or qualitative data [5, 12], provide no opportunity for data triangulation. The MRC guidance has increased recognition of the importance of using multiple methods for process evaluation data collection [13–15] (usually a combination of observations, logbooks, questionnaires and qualitative methods [6]), however guidance for amalgamation and analysis of such data is lacking.

The WAVES study process evaluation methods (described in detail elsewhere [18]) were based on frameworks and guidance by Linnen and Steckler (2002) [6], and Dane and Schneider (1998) [19]. Although developed prior to the MRC guidance document [3] on comparison the WAVES study process evaluation corresponds well. Contextual factors influencing both implementation and pathways to impact (through qualitative [20, 21], survey, and researcher experience data) were explored, and intervention implementation measured using a variety of direct and indirect methods.

The aim of this paper is twofold: 1) to demonstrate a replicable method of process evaluation data synthesis for use by other complex health intervention researchers, and 2) to present the results of the WAVES study process evaluation, demonstrating how the intervention was delivered and received.

The results of the WAVES study process evaluation provide detailed information on intervention implementation, and a replicable method for analysing process data from health intervention research. Inter-component differences in fidelity were evident, seemingly driven by required teacher workload and the enthusiasm and support from senior staff. We found inter-school variation in delivery of the WAVES study intervention programme, although overall there was good fidelity of implementation in most schools.

Recently, several extensive process evaluations which have used multiple methods for data collection, similar to the WAVES study, have been undertaken [13, 15, 22]. However, reporting tends to focus on the findings of each method (e.g. reporting questionnaire data or observation data) in isolation followed by an overview of what this meant for overall implementation. The findings of these studies provide useful information in helping future researchers learn from the experiences of intervention delivery; however the confined approach to data collection and synthesis limits interpretation. In this study, the triangulation and integration of data sources increases the validity of the findings. It enables a complete picture of implementation and participant response to be synthesised, and identifies variation between clusters. The generation of an overall implementation score also allows for intervention implementation to be considered in relation to the trial outcomes, in line with recommendations in the recent MRC process evaluation guidance [3].

The implementation findings specific to the WAVES study are also useful to help inform future intervention in the research field. Schools are often considered a key setting for the delivery of health interventions as they provide a teaching and learning environment alongside eating and PA opportunities [2, 23]. However in our study many teachers reported finding it challenging to deliver an intervention in addition to their teaching responsibilities. Individual teachers’ beliefs in the importance of the intervention’s overall objective (prevention of childhood obesity through the encouragement of healthy lifestyle behaviours) was found to have a positive impact on implementation fidelity, particularly when they perceived healthy behaviours as central to children’s development and learning.

The daily school-time delivery of PA was the most challenging intervention component for teachers despite the activity packages offered being easy to implement in the classroom setting, flexible to deliver, and teachers having a choice of packages. However, it was the component which placed the most burden on teachers, as it was a daily activity. Most schools achieved at least some additional physical activity, and it may have been different if the intervention was only focused on this one component rather than also having the additional activities schools were asked to incorporate. The findings support the need for leadership within schools to encourage regular inclusion of additional PA, particularly as there is evidence to suggest that moderate-vigorous physical activity (MVPA) may be positively associated with, or at least does not negatively impact, academic attainment [24–26].

The importance of PA, for health and the development of basic movement skills, warrants continued efforts to try to learn from experiences such as those of the WAVES study to help address difficulties in delivery and identify ways in which PA can be incorporated into the primary school day. In the UK, much media attention has been given to a recent initiative - ‘the daily mile’ [27], an intervention whereby all children attending school run outside for 15 min every day, a simple concept that is reportedly easy to deliver may be a more user friendly approach for the teachers compared to the options provided by the WAVES study intervention. Recommendations from the WAVES study experience would be to: (i) encourage teachers to understand the central importance of PA to child development, aiming to improve enthusiasm for delivery, (ii) enable schools and class teachers individually to identify the best way to ensure PA is routinely timetabled every day, (iii) allow teachers adequate time to consider their competing demands and plan delivery to suit their individual class needs, and (iv) provide training and support for teachers to help them feel confident with delivery.

Current national policy in the UK stipulates that schools must teach physical education but there is no guidance on the minimum amount of time that schools need to dedicate to it. Although headteachers see healthy lifestyles as an important part of development of the whole child [23, 28] it is hard for them to give such aspects of child development as much importance as academic achievement due to the present external pressures placed on schools. This is a similar finding to that reported in the results of the Active for Life Year 5 process evaluation – a key reason for teachers failing to adhere to intervention elements was pressure to focus on literacy, numeracy and academic attainment [15].

The CW and VV components of the intervention were relatively well received and delivered. However, as for the PA component, the overarching limiting factor in optimal delivery was time. Although VV achieved the most promising levels of implementation fidelity, it has significant cost implications. In addition, despite teachers being positive about the CWs and reporting that the materials and session plans made them easy to deliver, they indicated that, due to the logistics of organising the sessions, continued delivery in future years would require particularly motivated staff. Cooking skills have since been included in the National Curriculum for all UK schools [29] which is a positive step towards incorporating interventions such as this one. The signposting sheets were resource intensive to produce. This, together with no evidence of their impact on families, suggests that this element should not be included in future school based interventions.

In stakeholder consultations undertaken as part of the development work for the WAVES study intervention, family involvement through activities aiming to improve practical skills in addition to knowledge was identified as a priority [16]. Systematic review evidence also supports the importance of involving family members [2]. The WAVES study tried to involve families through school-specific signposting sheets, parental invitation to the cooking workshops and the healthy challenges element of VV. Unfortunately, the former had little or no impact, and although there was positive feedback regarding CWs from the parent focus groups, attendance rates were often low (mean parental attendance was 41%). However, pre-existing parent-school relationships heavily influenced the level of parental engagement, again highlighting the important contextual influences on intervention implementation. The level of involvement of parents with the VV healthy challenges was dependent on the teacher’s approach to delivery of the weekly challenges. Further research to determine how schools can better engage parents with health promotion initiatives would be valuable for both schools and intervention developers.

Limitations of the WAVES study process evaluation need to be considered. Process evaluation of a multifaceted intervention programme is inevitably a balance between comprehensive and detailed data collection and the resulting participant burden. The latter was a strong driving force during the development of our data collection tools, and in general completion rates of 60 to 70% were achieved, although lower rates were achieved for some items, especially PA teacher logbooks. However it is promising that: most returned logbooks were well completed; the cross check of data between matched logbook and observation time points revealed good consistency; and despite some short session durations, observation data indicated that most CWs covered key content and activities.

To ensure blinding of researchers to trial arm allocation, randomisation of schools was delayed until after baseline measurement completion resulting in very limited time (the last two weeks of the summer term) to introduce the class teacher to the intervention programme, a step we have previously highlighted as critical. This process was further hindered, as despite best efforts to involve class teachers as early as possible in the recruitment stage, it was clear in some schools that the first time they were aware of the expectation for them to undertake intervention delivery during the subsequent school year was at the introductory visit by the research team. Both of these would have resulted in insufficient planning and preparation time for teachers and are likely to have negatively impacted overall implementation fidelity of the programme. Another factor likely to have negatively influenced both quality of intervention delivery and process evaluation data return rates is that the intervention year in half the schools (2011-12) coincided with two events for which schools took on many additional activities (The Queen’s Diamond Jubilee and the London 2012 Olympics).

The possible impact that direct observation of teachers undertaking intervention activities may have had on quality of delivery must be acknowledged. The intention was to arrive at schools unannounced; however this approach was poorly received by schools and also resulted in wasted researcher time due to last minute rescheduling of planned activities (e.g. researchers arriving to find the children were at swimming lessons or school play practice). This meant that subsequent researcher visits were prearranged and as such prior teacher knowledge of session observation may have influenced implementation. Although it is important to acknowledge this as a potential limitation, in reality both the teachers’ and children’s proficiency with the routines provided a good indication of implementation consistency. The observation checklists were tested until inter-rate reliability was achieved, but by their nature the completion and rating of them is subjective.

In the current study it was appropriate to consider intervention implementation at school level as there was limited variation in implementation scores between classes at the same school. In future studies if there was a greater inter-class variation in implementation fidelity between classes at the same school, it may be important to consider implementation fidelity by class rather than by school to avoid the possible masking of such differences.

The use of qualitative data is time intensive both in collection and analysis. However, due to the nature of the WAVES study intervention it provided a useful insight into a school based obesity prevention programme, providing key recommendations for future delivery. It also supported the other methods of data collection and gave a clearer picture of intervention implementation in the schools in which interviews and focus groups were conducted.

Despite the limitations, the WAVES study process evaluation was comprehensive and provides a unique approach to working with process data. The methods allowed for data triangulation and cross checking of data sources. Drawing on multiple sources of evidence allowed for the generation of a score that can be used in analysis of the main trial outcomes. The approach to data analysis was rigorous and several steps were taken to try to minimise the effect of subjectivity in the scoring process. Researchers scored schools independently, and consensus was sought from the wider research team. The WAVES study model (Fig. 1) is replicable and could be applied to process evaluations from many different aspects of health intervention research. This paper reports on the analysis of process evaluation data, providing a level of detail which is rarely reported in the process evaluation literature [6]. Following the MRC recommendation for analysis of process data [3] we present data which meets recommendation by (i) providing information on fidelity, dose and reach for the intervention, (ii) detailing variation in implementation between schools (iii) using thematic analysis to analyse the qualitative data (iv) integrating both qualitative and quantitative data sources to provide an overall indicator of intervention implementation, and (v) completing all analyses before analysis of the main trials outcomes. The WAVES study was undertaken in the West Midlands, UK, − a region that is socioeconomically, ethnically and culturally diverse. The school selection process ensured an over-representation of schools with a higher proportion of South Asian or Black pupils by using a randomly ordered, weighted random sampling procedure from amongst 970 eligible state maintained schools. Randomisation of schools to the control or intervention arm used a statistical procedure to minimise inter-arm imbalance in relation to school size, free school meal eligibility (as an indicator of deprivation) and proportion of pupils of South Asian, Black and White ethnicities. Additionally, as reported earlier, schools from the intervention arm were purposively sampled for inclusion in the interviews/focus groups to ensure representation from a diverse range of schools. These processes helped to improve the generalisability of the findings across different UK locations and the findings from the intervention delivery should be useful to other researchers working in the field.

We have presented a unique, rigorous and replicable approach to the analysis of process evaluation data, demonstrating how it can provide insight into intervention implementation, allow for analysis of main trial results by implementation, and test assumptions about pathways to impact. The findings identified challenges that need to be addressed both in the design of future interventions and in the future direction of national policy to optimise their implementation.