What factors influence implementation of whole-school interventions aiming to promote student commitment to school to prevent substance use and violence? Systematic review and synthesis of process evaluations

Abstract

Background

Whole-school interventions go beyond classroom health education to modify the school environment to promote health. A sub-set aim to promote student commitment to school to reduce substance use and violence (outcomes associated with low commitment). It is unclear what factors influence implementation of such interventions.

Methods

We conducted a systematic review including synthesis of evidence from process evaluations examining what factors affect implementation. Meta-ethnographic synthesis was informed by May’s General Theory of Implementation.

Results

Sixteen reports, covering 13 studies and 10 interventions were included in our synthesis. In terms of May’s concept of ‘sense-making’, we found that school staff were more likely to understand what was required in implementing an intervention when provided with good-quality materials and support. Staff could sometimes wilfully or unintentionally misinterpret interventions. In terms of May’s concept of ‘cognitive participation’, whereby staff commit to implementation, we found that lack of intervention adaptability could in particular undermine implementation of whole-school elements. Interventions providing local data were reported as helping build staff commitment. School leaders were more likely to commit to an intervention addressing an issue they already intended to tackle. Collaborative planning groups were reported as useful in ensuring staff ‘collective action’ (May’s term for working together) to enact interventions. Collective action was also promoted by the presence of sufficient time, leadership and relationships. Implementation of whole-school interventions took time to build. Considering May’s concept of ‘reflexive monitoring’ (formal or informal review of progress), this was important in assessing and enhancing implementation. ‘Quick wins’ could help maintain collective impetus to implement further intervention activities.

Conclusion

We identified novel factors influencing implementation of whole-school elements such as: local adaptability of interventions; providing local data to build commitment; interventions addressing an issue already on school leaders’ agenda; collaborative planning groups; and ‘reflexive monitoring’ as an explicit intervention component.

Background

Whole-school interventions go beyond merely providing classroom health education to modify the school environment to promote health [1]. A sub-set aim to promote student commitment to school to prevent outcomes such as substance use (i.e. tobacco, alcohol and other drugs) and violence. Whole-school intervention is an appropriate school health promotion strategy given that increased student commitment to school is associated with reduced substance use and violence [2, 3] and other health outcomes [4] so that promoting student commitment might prevent these risk behaviours. There is increasing academic and policy interest in such interventions, reflecting awareness that health-education lessons struggle to find a place in school timetables and have patchy results which tend to dissipate over time [5,6,7,8].

However, the effectiveness of whole-school interventions is unclear, given the lack of recent systematic reviews [1, 9]. Furthermore, although previous reviews have identified factors influencing initial delivery and sustained implementation of health interventions in schools [10, 11], these have not focused specifically on whole-school interventions. Key enablers of school interventions reported in these previous reviews are: strong institutional capacity (e.g. supportive senior management); alignment of the intervention with school ethos and priorities; positive pre-existing student and teacher attitudes; and parental support for interventions. However, whole-school interventions are more complex than the largely curriculum-based interventions examined in these previous reviews and so may be affected by different factors. Hence, a review of what factors affect the implementation of whole-school interventions is warranted in order to inform better implementation.

We conducted a systematic review synthesising evidence on whole-school interventions aiming to prevent substance use and violence. The overall review aimed to examine underlying theories of change, explore what factors affect implementation, and assess effectiveness and cost-effectiveness, and will be published shortly. This paper reports on the review and synthesis of process evaluations examining what factors affect implementation, which is a major influence on effectiveness [12]. The review was guided by the theory of human functioning and school organisation as to which interventions were included since this is the most comprehensive theory of how school environments may be modified to promote health [13, 14]. Rather than requiring included interventions to reference this theory (which might bias the review to certain geographies and disciplines), we instead required interventions to include one or more components aligning with constructs in this theory: modifying teaching to increase student engagement; enhancing student-staff relationships; revising school policies with students; encouraging students to volunteer in the community; or involving parents in school life.

The research question which our review of process evaluations aimed to examine was: what factors relating to setting, population and intervention influence the implementation of whole-school interventions aiming to prevent substance use and violence via promoting student commitment to school? Our analysis of implementation was informed by May’s General Theory of Implementation [15]. This is an appropriate framework for addressing our research question because it provides a detailed consideration of the processes involved in implementation and how these are influenced by the characteristics of the intervention and setting. Other frameworks, such as the RE-AIM framework [16], focus on implementation but not on underlying social processes or their influences. May’s theory describes implementers as engaging in processes of: ‘sense-making’ (understanding the intervention); ‘cognitive participation’ (committing to its delivery); ‘collective action’ (collaborating with others to ensure implementation); and ‘reflexive monitoring’ (assessing implementation and need for further action). The theory proposes various factors influencing the enactment of these processes: intervention ‘capability’ (the workability of and possibilities presented by the intervention); institutional ‘capacity’ (the material and cognitive resources, norms and roles present in a setting to support implementation); and ‘potential’ (individual and collective attitudes which might affect implementation). Our review also enabled an assessment of the state of evidence regarding process evaluations on this topic which might inform future work in this area.

Methods

Design

The overarching systematic review, from which this analysis draws evidence, followed guidelines for review conduct and reporting e.g. Centre for Reviews and Dissemination [17] and Preferred Reporting Items for Systematic Reviews and Meta-Analyses [18] (Additional File 1). The protocol was publically registered (https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=154334). No protocol amendments were made.

Identifying references

Informed by the constructs featuring in the theory of human functioning and school organisation as described above [13], we included studies of whole-school interventions (i.e. not merely classroom health education) aiming to reduce violence or substance use via: modifying teaching to increase student engagement; enhancing student-staff relationships; revising school policies with students; encouraging students to volunteer in the community; or involving parents in school life (intervention). Evaluations focused on children and young people aged 5–18 years (population) and the prevention of violence (defined as interpersonal physical, emotional or social abuse) and substance (i.e. tobacco, alcohol or other drug) use (outcomes). In this paper, we report on included process evaluations (design) which provided empirical findings on processes of intervention implementation.

Searches included terms for intervention, population and evaluation design. We originally searched 21 databases, three trial registries and 32 websites (16–27 January 2020), and updated the search (11–25 May 2021) across 14 databases, two trial registries and 32 websites (Additional File 2). Our searches deviated slightly from those in the protocol on the advice of an information scientist. Several databases that are no longer updated or which could not be accessed were dropped. A wider range of education, medical, nursing and public health databases were added to ensure the multi-disciplinary nature of this topic was adequately reflected in the references retrieved and to compensate for the dropped databases. The narrower scope of update searches was due to reduced accessibility of some sources in the context of the Covid-19 pandemic. We also searched reference lists of included studies and emailed topic experts.

Screening references

Citations identified through searches were de-duplicated and uploaded to EPPI-Reviewer 4.0 software. Two reviewers then screened batches of the same 50 references, resolving disagreements by discussion if necessary. Our protocol was for these reviewers to double-screen the same references in batches of 50 until reaching a 90%+ agreement rate. This was achieved on the first batch where there was only one disagreement settled by discussion. Reviewers then single-screened references on title/abstract. Full reports of references not excluded at this stage were reviewed via an analogous process.

Data extraction and quality-assessment

Two reviewers independently extracted data from included process evaluation reports on: study location, timing and duration; individual and organizational participant characteristics; study design; sampling and sample size; data collection; data analysis; findings; and interpretation. We assessed the methodological quality of process evaluations using the EPPI-Centre tool [19] addressing the (1) quality of sampling (e.g., was sampling appropriate to the questions?; were all stakeholders included?), (2) data collection (e.g., were tools validated or piloted?; was data collection comprehensive, flexible and/or sensitive to provide a rich description of processes?), (3) data analysis (e.g., was analysis systematic?; was diversity in perspective explored?), (4) the extent to which study data informed findings (e.g., were enough data presented to show how authors derived findings?; do the data presented fit the interpretation?), (5) whether the study privileged student perspectives (e.g., were students included?; was there a balance between open-ended and fixed-response options?), and (6) the breadth and depth of findings (e.g., were a range of process issues covered in the evaluation?; were the perspectives of participants fully explored in contrasting two or more perspectives and insight into a single perspective?). Informed by guidance, studies were rated as low, medium or high on the reliability of the findings, and as low, medium or high on the usefulness of the findings for addressing the research questions. Study reliability was judged high when steps were taken to ensure rigour in at least four of the above assessment criteria, medium when addressing only three and low when addressing fewer than three. To be rated ‘high’ on usefulness, studies needed to privilege student perspectives and present findings with breadth and depth. Studies rated as ‘medium’ usefulness only partially met this criterion, and studies rated ‘low’ were judged to have limited relevant findings.

Synthesis

We synthesised process evaluation findings (including quotes from study participants) and author interpretations regarding the factors influencing implementation using meta-ethnographic synthesis methods. As with earlier reviews [20, 21], these were applied to textual reports of qualitative but also quantitative research, it not being possible to synthesise quantitative findings from process evaluations statistically because of methodological heterogeneity. In the case of findings from quantitative research, we coded author interpretations, checking whether these aligned with quantitative data. Meta-ethnographic analysis examined recurrent themes, identifying cases of ‘reciprocal translation’ (similar concepts being expressed differently in different sources) and ‘refutational synthesis’ (concepts from different sources contradicting each another). We then developed a ‘line of argument’ synthesis drawing together concepts from different sources to develop an overall analysis of factors influencing the implementation. The synthesis was not restricted to high-quality studies but poorer-quality reports were given less interpretive weight.

Synthesis involve the following steps. First, two reviewers prepared tables describing the quality, empirical focus and site/population of each study. Then the two reviewers piloted the analysis of two high-quality reports, reading and re-reading these reports and applying line-by-line codes. Next, reviewers drafted memos explaining these codes. Coding began inductively with in-vivo codes closely reflecting the words used in reports’ findings. The reviewers then grouped and organised codes, applying axial codes to identify higher-order themes. This stage of analysis was informed by May’s General Theory of Implementation as a sensitising device [15]. The two reviewers then met to compare these codes for the two studies, finalising an overall set of codes. This finalisation was facilitated by the reviewers having developed similar sets of codes. The reviewers proceeded to code the remaining studies drawing on the agreed set of codes, developing new codes as needed and writing memos to explain these. The two reviewers then met to compare their codes and memos, and agreeing a single set of overarching themes drawing on the strengths from each set of codes.

Results

Included reports and quality

The original searches retrieved 62,742 unique references and 56 eligible reports. The updated search retrieved 9,709 unique references and nine eligible reports (Fig. 1). In total, 65 reports on 27 studies of 22 interventions were included. Sixteen of these were process evaluations, covering 13 studies and 10 interventions [22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. Of these ten interventions, one was delivered to children approximating to English primary school age [5,6,7,8,9,10, 38], six to children of English secondary school age (age 11–18) and three to children whose ages spanned these ranges. Of the thirteen studies, five were from the USA, four from the UK three from Australia and one from Uganda. Nine reports drew on quantitative and qualitative data, five only on quantitative data and two only on qualitative data. Table 1 summarises the characteristics of process evaluations.

Fig. 1

Searches and screening

Table 1 Study characteristics

Initial agreement over study quality was high (> 90%). Three studies were judged to be of both high reliability and high utility in addressing our research questions. [25, 26, 30, 35,36,37] One study was judged to be of high reliability and of medium utility in addressing our research questions. [22] One study was judged as medium reliability but low utility. [23] A further study was judged as being of low reliability but of medium utility. [24] Four studies were judged as of low quality and low utility [27, 28, 33, 34]. Three studies were rated as of high [31, 32] or medium reliability, [29] but low utility (Table 2).

Table 2 Quality of process evaluations

Synthesis of evidence on factors affecting implementation

Various themes and sub-themes were apparent in quotes from study participants and author interpretations, the structure of which is summarised in Table 3. These are presented below structured according to the constructs from the General Theory of Implementation (indicating by these being in inverted commas) with which they aligned. References indicate which studies informed which themes.

Table 3 Coding structure

Sense-making

‘Sense-making’ was a recurrent theme in studies i.e. a process of staff coming to understand the intervention which contributed to the enactment of interventions. Sense-making was reported to accrue over time and pervade all, not just the initial, stages of implementation [22, 24, 26, 30, 35]. Various factors were reported to affect how school staff and students understood intervention resources.

Intervention capability’ to be made sense of

A sub-theme suggested that sense-making could be facilitated by an intervention’s ‘capability’ (i.e. workability). This could be in terms of providing good-quality materials and/or ongoing support in the form of training, external facilitation or coaching [22, 24,25,26, 30, 33,34,35,36]. Materials and resources that included tangible, contextually relevant examples were reported as enabling providers to understand how intervention activities might occur in their setting, for example as reported by one head-teacher in an evaluation of Learning Together rated as of high reliability and usefulness [25]:

“The one thing schools need is a model, of how it’s going to work in the school, in a real-life school, so that they can almost touch it, taste it, feel it, and then start implementing it in their own schools”. (p.39)

Two studies reported that staff were sometimes initially confused by intervention materials or external providers [25, 26]. In the study of the Healthy School Ethos intervention rated as of highly reliability and usefulness [26], an initial presentation by an external facilitator was reported to have caused staff and students to misunderstand the aims of a whole-school intervention.

School ‘capacity’ to make sense of an intervention

Another sub-theme apparent in one UK study was how staff’s making sense of an intervention could be influenced by their existing priorities and the school’s institutional ‘capacity’ in terms of the resources present to support implementation [22, 26, 35, 36]. Those leading implementation in one school were said to have creatively reinterpreted Learning Together, an anti-bullying intervention, as an intervention aiming to maintain the emotional health of pressurised students in an academically selective school [35]. This evaluation further reported that, in another school, the lead reinterpreted the staff-student action group as being a site for students to learn the skills needed to avoid or respond to bullying (rather than, as intended, to coordinate intervention activities). This occurred in the context of the lead’s imprecise grasp of the intervention and inability to involve other staff.

Cognitive participation

The notion of ‘cognitive participation’ also recurred as a theme across studies, presented as a process of staff commiting to implement an intervention. Various factors concerning the intervention and the school were identified as influencing the extent to which school agents felt able to commit to enact intervention activities. Like sense-making, cognitive participation was a process that was built across all stages of implementation [22]. Several factors affected how cognitive participation developed.

Intervention ‘capability’ for local tailoring and adding value

A key sub-theme apparent in several studies was that school staff assessed intervention ‘capability (workability) in terms of ease of integration with existing practices [22, 24,25,26,27, 36]. Interventions that could be locally tailored or build on existing work were more likely to secure staff’s cognitive participation. An evaluation of low reliability and usefulness of the Drug Abuse Resistance Education (DARE) Plus intervention [27] described how the assessment phase of the intervention was essential to tailor the intervention and develop commitment:

”The assessment phase of the organizing process is critical to its long-term success. It is invaluable to take the required time to get to know the community before attempting to launch an action team.” (p.17)

Another report describes how school staff bought-in to use of restorative practice as an approach to discipline because this was viewed as providing a means of building on existing work and developing a consistent approach to discipline [25].

Interventions not viewed as being capable of local tailoring often failed to engender staff commitment, as reported by evaluations of the Responsive Classrooms and Positive Action interventions, respectively of high reliability and medium usefulness and low reliability and usefulness [22, 34].

A sub-theme apparent was that this lack of intervention capability for tailoring or adding value was particularly undermining for whole-school elements [22, 34]. Head-teachers and other school leaders could withhold commitment when they felt that whole-school actions might jeopardise their wider strategies. This could be the case, for example, where interventions required changes to school rewards or discipline policies that school leaders thought might weaken the school’s ability to pass school inspections or attract parents to send their children to the school. As an evaluation of Positive Action [34] reported:

”Reluctance to change whole-school policy may be exacerbated by circumstances such as an upcoming [government] inspection: ’It was hard to make a whole-school change to sanction and reward policy, so whole-school activity was harder to implement. [The government inspectorate] was coming and it would have been too big a change.’” (p.34)

Intervention ‘capability’ for using data to build commitment

Another sub-theme was that the provision of local data as part of the intervention could improve its ‘capability’ (workability) and build staff commitment [24,25,26,27, 30, 35, 36]. The evaluation of the Learning Together intervention suggested that providing such data could make it harder for staff to dismiss the need for intervention [25, 35, 36]. A staff-member on a pastoral team commented [35]:

“I remember when [facilitator] came to present to [senior leadership team] and said how terrible our data was… it was like a tumbleweed moment; it was so funny. I mean... it wasn’t funny in a good way, but... but it was a realistic... realisation for everyone if you know what I mean... Because we all knew it was like that, but we didn’t realise how much the children didn’t actually like us.” (p. 990)

However, in an example of refutational synthesis, several studies identified that the provision of data could sometimes undermine staff commitment when staff interpreted the data as a criticism of their work to date or where data did not indicate positive trends after implementing an intervention [26, 30, 36, 37].

Intervention ‘capability’ in terms of student participation

A sub-theme from several UK evaluations of interventions aiming to encourage student participation in decisions was that students were more likely to commit to an intervention where this offered an opportunity for them to express their views [25, 26, 34,35,36,37].

Staff ‘potential’ for commitment based on perceived need

Staff commitment to interventions was influenced by staff’s ‘potential’: whether staff were attitudinally ready for such an intervention. A key sub-theme was that interventions should offer school leaders something they already knew they needed [22,23,24, 34, 36]. This might be a way of responding to government policies, pressures from parents or inspection requirements. Or it might address internal imperatives, such as school leaders’ existing strategies for school change. This theme was particularly clear in the UK studies of both Healthy School Ethos [26] and Learning Together [25, 30, 35, 36]. The pilot evaluation of Learning Together [30], for example, reported:

“head teachers and their [management teams] consistently reported that it was important to address aggressive behaviours in order to recruit and retain ‘the best’ parents and students. [Managers] also suggested that this project was prioritised as it was seen as likely to impress the national school inspectorate… due to its focus on student voice and behaviour.” (p.328)

Interventions aiming to achieve whole-school change were more likely to get school leaders’ commitment when there was already a recognised need for change, for example because of poor inspection results [25, 30]. Reciprocally translating with this concept, it was apparent that in schools where leaders perceived no such urgent imperative for change, genuine school commitment was less likely.

Staff ‘potential’ for commitment based on existing strategies and values

A related sub-theme was that school staff had more attitudinal ‘potential’ for commitment to a whole-school intervention when their existing strategies and values made this seem attractive [22, 23, 25, 30, 32, 36]. New head-teachers were reported as particularly likely to commit to interventions involving whole-school change because these aligned with their desire to make their mark and change schools [30]. Reciprocally translating with this concept of school leaders’ ‘potential’ was teachers’ ‘potential’ [23, 25]. For example, teachers with a prior commitment to social and character education within their classes were more likely to implement curricula addressing this according to a study of Positive Action of medium reliability and low usefulness [23].

In cases where the values or priorities did not align, staff commitment appeared less likely [22, 23, 32]. For example, where staff or students perceived restorative practice to be a softer option, they were reportedly unlikely to commit to enacting it. As a study of Responsive Classrooms [22] reported:

“In contrast, some middle school staff members’ beliefs about the value of punitive responses to problem behavior were incompatible with the core tenets of the intervention, which emphasized inclusion and opportunities to learn: ’When you steal, there are real consequences; there’s jail or fines…’ These staff members believed that zero-tolerance policies, which use punishment as an extrinsic motivator for behavior change, were more effective than RC approaches.” (p.84)

A sub-theme concerned the possibility of schools committing to implementing only those intervention components aligning with their existing strategies and values, rejecting components that they regarded as deviating from these [26, 36].

Collective action

Evaluations also examined processes by which those in schools engaged in ‘collective action’ (working together) to divide up responsibilities for delivering interventions. A number of factors were identified as influences on such processes.

Intervention ‘capability’ as workable

A key sub-theme was the importance of interventions being locally workable for staff enacting interventions as planned [22, 26, 36]. For example, curriculum materials which did not fit into the school curriculum or which did not provide staff with clear lesson plans tended to be adapted before they were delivered, or were not delivered at all [24, 35, 36].

An important aspect of workability was the extent to which guidance materials spelt out how delivery should proceed. For example, materials underpinning a restorative practice interventions needed to specify which staff-members were responsible and whether the intervention was intended to complement or replace punitive discipline [22].

Some interventions were not collectively enacted as had been planned. For example, an evaluation of Responsive Classrooms [22] found that a new approach to discipline failed to work within the reality of schools:

”[P]articipants reported that a key RC strategy, Logical Consequences, in which a response to student misbehavior is tied to the specific incident and creates an opportunity for learning, was too unwieldy to implement in a way that students could anticipate and incorporate: ’I totally agree with the theory behind logical consequences where you want the consequences that match the behavior and that’s, like, respectful to the child and respectful to the teacher. But it’s hard because it’s different every time… It’s not a system where they know, like, oh, if I do this I know what’s going to happen.’” (p.85)

Planning groups as a key element of intervention ‘capability’

An important sub-theme was that interventions which included planning groups, consisting of staff and sometimes students, parents or other community-members, were more workable in ensuring collective action. This was apparent from reports of the Gatehouse Project (of low reliability and medium usefulness), Learning Together interventions (of high reliability and usefulness) [24,25,26, 35,36,37] and other interventions [23, 24, 27, 32,33,34]. Diverse participation in such groups could support implementation by ensuring that the decisions made by the group were pragmatic and by achieving wider commitment across the school.

Such groups were reported to be particularly facilitative of whole-school approaches [24, 26, 35, 37]. These groups could also help ensure that intervention activities added up to a coordinated process of integrated school transformation, rather than merely a disparate set of initiatives.

Synergy between intervention components as a key element of intervention ‘capability’

A further sub-theme was that some interventions were more workable because they had better synergies between intervention components than others [22, 24,25,26, 30, 35,36,37]. Some intervention activities created the informational and relational resources needed to enable agents to enact other actions. The evaluation of the Gatehouse Project [24] reported for example:

“It is clear from our work that these elements - the adolescent health team, the school social climate profile, and the critical friend - do not work in isolation. The profile provides local data that are essential for identifying risk and protective factors relevant to the particular school community. The adolescent health team ensures that the responses to the profile are owned and implemented by the whole-school community. The critical friend provides expertise, impetus, motivation, and links to external resources.” (p. 380)

As described above, data on student needs being provided as part of an intervention could encourage others to implement intervention activities or lead to school staff producing or sharing other data [25, 30].

One area of synergy was where training components provided staff with the skills they needed to deliver other intervention elements. This could be valuable in ensuring staff accumulated and consolidated their skills [22].

Other reports focused on lack of intervention-component synergy as an inhibitor of collective action. For example, some evaluations reported that there was a noticeable lack of effective interaction between curriculum and whole-school components. In some cases, classroom curriculum activities were enacted but whole-school changes were incompletely delivered [34]. In other case, whole-school elements which aimed to build on existing school achievements were enacted but curriculum elements were not delivered with fidelity because these were judged unworkable [35, 36].

School ‘capacity’ to support collective action

The extent to which agents in schools could come together to collectively enact interventions also depended on school ‘capacity’ (i.e. the resources available to these agents). The lack of space in school timetables, and the lack of non-contact time within which school staff could plan intervention activities was frequently reported by evaluations [22, 25, 27, 28, 33, 34, 36]. For example, the evaluation of low reliability and usefulness of the Cyber Friendly Schools intervention reported [28]:

“Many teachers reported not being able to find sufficient time in their teaching curriculum to complete the eight learning activities.” (p. 104)

In the evaluations of low reliability and usefulness of the DARE Plus intervention and the PPP intervention [27, 33], whole-school elements were described as the most challenging and time-consuming to organise.

Staff struggled to marshal time and other resources when they were expected to deliver a new intervention alongside other initiatives. These situations diffused the resources available for any one intervention and eroded agents’ ability to commit the time needed to support effective decision-making and delivery. The evaluation of Responsive Classrooms for example reported [22]:

”A school leader noted that ‘It’s not one new thing; it’s always five new things that we’re working on. I think the attention span is tested.’” (p.84)

Another resource factor in determining whether interventions were collectively enacted with fidelity was whether those charged with leading the intervention possessed leadership resources, such as a budget, the ability to direct other staff or the ability to modify policies or systems [22, 25,26,27, 30, 36, 37, 39]. Schools that gave intervention leadership roles to powerful staff consistently achieved better implementation according to several evaluations. Power and authority could be formal or informal, the latter reflecting individuals or groups having a long track-record at the school, strong relationships and an informal ability to persuade people to make things happen [35, 36]. An evaluation of Learning Together [35] for example reported:

”In another school, despite there being no senior leaders on the group, the lead had worked for a long time at the school and was well respected and liked by both students and staff. Thus, it was possible to galvanise action without the formal involvement of senior leaders in some cases.” (p.989)

Where leadership commitment to intervention activities was limited or inconsistent, there may thus have been less collective vision and impetus for implementation, as reported in the evaluation of the Responsive Classrooms intervention [22]. Lack of senior level support could also affect the drawing down of material and cognitive resources to support intervention activities [35, 36]. For example, some decisions made by action groups were stalled or rejected by other agents within the school system, such as head-teachers or school-leadership teams [35, 36].

Interventions could also be better implemented in schools characterised by strong connections between staff or with strong cultures of innovation [22,23,24, 32, 36]. In schools with strong connections, those agents leading interventions could draw on existing relational resources such as mutual support, observation and learning to support enactment, rather than attempting to develop this from a low baseline. An evaluation of Positive Action [32] of high reliability and low usefulness reported:

“Stronger affiliation among teachers likely led to more opportunities to share ideas about PA materials and observe other teachers as they carried out PA activities outside of the classroom. This may have influenced teachers’ use of these supplementary program components, with higher levels of use by teachers who had perceptions of high engagement and support among teachers in their schools.” (p.1091)

An evaluation of the Gatehouse Project [24] similarly reported the importance of networks connecting staff in enabling collective action.

A culture of teacher autonomy, as reported in the evaluation of Friendly Schools [22], could undermine collective action, because it was difficult for those leading an intervention to encourage the consistent enactment of new practices which deviated from locally understood norms and expectations of staff roles. Similarly, the evaluation of the Responsive Classrooms [22] intervention reported:

“School staff observed that [Responsive Classrooms], a schoolwide intervention, ran counter to the school’s culture of individuality. For example, one teacher noted: ’One… characteristic of [the school is]… there’s a lot of autonomy in terms of how teachers run their classrooms… it’s a little bit of territorial, like… I know what I’m doing and I have my way of doing it so I don’t need to participate necessarily in a whole-school anything.’” (p.84)

A staff culture of innovation could also support collective implementation. Such cultures could encourage staff to take the time to identify who would implement the intervention and then enact this with fidelity [23, 34].

Reflexive monitoring

Whole-school interventions took time to build. ‘Reflexive monitoring’ (whereby staff assessed the success of implementation through formal or informal processes) was important in determining the extent to which implementation built or dissipated over time.

Intervention ‘capability’ for reflexive monitoring

Reflexive monitoring worked well when interventions included this as an explicit component [24, 26, 30, 36] increasing their ‘capability (workability). Studies indicated that interventions were particularly successful when they included an action group that reviewed data, identified priorities, oversaw delivery and reflected on the results. This enabled members to reflexively monitor what was being enacted and with what consequences. Evaluations suggested that this gave participants the permission and resources to try different things, persist with what was perceived as working and refine or reject what was perceived to go less well. This approach allowed staff to abandon activities viewed as unsuccessful without rejecting the intervention overall. For example, an evaluation of the Gatehouse Project [24] reported:

”This common purpose gave permission for teachers to try new strategies such as substantially restructuring student and teacher teams. For example, in one school, teachers worked together to reorganize classes into small groups of four or five learners and teachers into teaching teams to promote a collaborative and an academic environment.” (p.375)

As part of processes of reflexive monitoring, ‘quick wins’ evidencing positive outcomes could also can help maintain and further build coalitions and commitment, and collective impetus to implement further intervention activities [27].

As well as groups, ongoing support from training, facilitation or coaching could also support reflexive monitoring by providing an opportunity for reflection and/or an outsider perspective. The importance of an external facilitator was, for example, described as follows in an evaluation of the Gatehouse Project [24]:

“The support that [critical friend] provided in the staff room, in staff meetings, has been invaluable. We wouldn’t be where we are now, because I’d never recognized the value of having a person who is not a practicing teacher in the school at the moment… the way that you’ve been able to involve yourself in the discussion and the activities that are going on and come through with some very well-made points at crucial times, but in small groups and large groups.” (p. 377)

‘Collective reflexive monitoring’ to refine implementation

Reflexive monitoring could be a collective action oriented towards refining how an intervention was implemented [24, 33, 35]. For example, in the case of two interventions, over time staff in some schools opted to recruit fewer disengaged or disadvantaged students to participate in intervention activities [33, 36].

When external facilitators were removed in the Learning Together intervention, this resulted in the overall fidelity of implementation declining but some intervention components becoming mainstreamed so that their ‘form’ was modified at the same time as their ‘function’ became integrated within school policies and systems, as one evaluation [35] evaluation reported:

”Most interviewees suggested that external facilitation was not necessary in the final year, but a few suggested this was a significant loss: ’The absence of [facilitator] has been incredibly significant because she… was able to tie it in all the time to the agenda. And was a touchstone I suppose really for that. And then… so that… I think that was a loss’. (Senior leadership team member…)” (p.991)

Reflexive monitoring reinforcing implementation

Reflexive monitoring could reinforce the conditions necessary for further implementation [24, 26, 33, 36]. Staff and students recognised through processes of reflexive monitoring that interventions had diverse consequences for different parts of school systems, many of which were unanticipated. For example, an evaluation of the Gatehouse Project [24] reported:

“not only has the work of the adolescent health team facilitated reviews of organizational structure, but it has also contributed to a substantial shift in the perceptions of what is the core business of schools. [As one staff member reported:] ’But just really reinforcing the ideas of the positiveness and feeling secure at school, and certainly encouraging staff, that irrespective of what subject they teach, they can have an influence. And it’s a bit like planting a seed…’. There was also evidence of changing professional identity - teachers shifted their position from being a teacher of a subject or program to placing the young person and learning at the center of practice.” (p.379)

Similarly, involving students in decision-making or being surveyed about their needs could transform staff and student attitudes by suggesting that the school was becoming a more participative institution [26].

Discussion

Summary of key findings

We examined what factors relating to setting, population and intervention influence the implementation of whole-school interventions aiming to prevent substance use and violence via increasing student commitment. We used the General Theory of Implementation [15] as a framework to inform our analysis. This aligned strongly with our findings and provided us with a basis to summarise the school, population and intervention-related factors that influenced implementation. It enabled us to draw out which points in the process of implementation were affected by particular factors.

In terms of ‘sense-making’, evaluations suggested that school staff were more likely to understand what was required when provided with good-quality materials and ongoing support, a point also reported by previous reviews [10, 11]. A novel finding was that school staff could sometimes wilfully or unintentionally misinterpret intervention aims.

In terms of ‘cognitive participation’, evaluations suggested that various factors influenced whether school staff were prepared to commit to enacting intervention activities. As identified in previous reviews [10, 11], staff assessed interventions in terms of their ease of integration with existing practices. A novel finding from our review was that intervention lack of local adaptability was particularly undermining for whole-school elements, such as proposed changes to school policies or discipline systems. Interventions providing local data was reported as helping build commitment. The factors affecting implement are influenced by the intervention theory of change. For whole-school interventions aiming to build student commitment to school, it is important that interventions are tailorable to school cultures and structures.

As identified in previous reviews of school-based interventions [10, 11], staff commitment to deliver an intervention was also reported to be affected by the school capacity. A novel finding from our review was that school leaders were more likely to commit to a whole-school intervention when this addressed an issue they were already interested in tackling, for example, providing a way to respond to a new government policy or inspection requirements. Schools were also more likely to commit when there was already a recognition of the need for change, for example because of poor exam results or a new head wanting to transform a school. Again, it appears that this factor is a more important influence on the implementation of interventions aiming to transform whole-school cultures and systems than classroom education interventions.

In terms of ‘collective action’, as found in previous reviews [10, 11], interventions needed to be locally workable if staff were to work together to deliver them: fitting with school timetables and providing staff and students with clear guidance. A novel finding which again appears to be specific to interventions aiming to transform whole-school organisation was that planning groups (consisting of staff and possibly also students, parents or other community-members) were reported as particularly successful in ensuring collective action to enact interventions. Diverse participation in such groups could support implementation by ensuring decisions were pragmatic and by maintaining commitment. Studies also reported that these groups could also help ensure that intervention activities added up to a coordinated process of integrated school transformation, rather than merely a disparate set of initiatives.

Another novel finding, which may be particularly important regarding multi-component whole-school interventions, was that better synergies between intervention components appeared to facilitate implementation. Synergy appeared to occur where the consequences of enacting one intervention activity (e.g. training or provision of local data) helped provide the conditions required for the implementation of other components.

The extent to which staff in schools could come together to collectively enact interventions also depended on local ‘capacity’ (available resources). As identified in previous reviews of implementing school health interventions [10, 11], time was in chronic short supply in many schools, which undermined staff’s ability to implement interventions. A novel finding from our review was that another key resource was whether those charged with leading whole-school interventions were empowered to do this. Such leadership resources could include the appropriate budget, the ability to direct other staff or the ability to modify policies or systems. Another key resource was whether school systems possessed strong connections between staff and cultures of innovation. Conversely, a culture of teacher autonomy could undermine collective action with regard to whole-school work.

In terms of ‘reflexive monitoring’ (review of implementation), an important new finding specific to implementation of whole-school interventions was that it took time to achieve whole-school change as school staff and students gradually came to define their roles in intervention processes and develop the commitment and cognitive capacity they needed to effectively facilitate implementation. Reflexive monitoring, whether through formal or informal processes, was therefore critically important in determining the extent to which implementation built or dissipated over time. Interventions were particularly successful when they included an action group that oversaw delivery and reflected on the results. These processes enabled members to reflexively monitor what was being enacted and with what consequences. This could give participants the permission and resources to try different things, persist with what was perceived as working and refine or reject what was perceived to go less well. ‘Quick wins’ evidencing positive outcomes could help maintain collective impetus to implement further intervention activities. Ongoing support from training, facilitation or coaching could also support reflexive monitoring by providing an opportunity for reflection and/or an outsider perspective.

Limitations

The original review searches involved multiple sources and methods, and aimed to maximise sensitivity. However, the updated searches were necessarily narrower because of the limits imposed by the Covid-19 pandemic. However, the sources that yielded all of the included study reports found as a result of the original electronic searches were included in the updated searches so we think it unlikely that any studies were missed because of this reduced scope. We synthesised qualitative research on implementation processes and thus any assessments of implementation are based on the assessments of those interviewed. It did not aim to synthesise quantitative evidence on implementation fidelity. Our process evaluation synthesis was limited by the size and quality of eligible reports. Study quality was mixed with a minority assessed as of high reliability and usefulness. We acknowledge that studies in which some of the authors of this review were involved were assessed as relatively high quality. Rather than this reflecting bias in our assessments, we think this reflects the fact that our design of the evaluations in question was informed by the quality assessment criteria used in this review [19]. Our review was not able to distinguish factors influencing implementation in primary versus secondary schools because studies were conducted in a diversity of school systems with different age bands.

Conclusion

Whole-school interventions to prevent substance use and violence are feasible to implement in schools. Because good implementation is so critical to intervention effectiveness [12], interventions need to be designed so that they can be delivered with strong fidelity and so achieve significant public-health benefits. Our review suggests that interventions should be optimised by designing them to be maximally implementable, for example by providing good guidance, data on local needs and developing collaborative coordinating bodies. However, there is a lack of evidence from low-income countries and we cannot be confident that our findings apply to such settings. Future evaluations of whole-school interventions need to occur across different settings and should include process evaluations to examine intervention acceptability and fidelity as well as factors affecting this. This should contribute to intervention refinements and inform assessments of potential intervention transferability to other settings and populations. The wider review of which this review of implementation is one element will report on the effectiveness of such interventions in preventing substance use and violence. The quality of process evaluations of the sort reviewed here could be improved by evaluators referring to quality-assessment tools such as those used in this review [19].