Table 1 Study characteristics
Study report and location | Intervention name, type and outcome addressed | Process evaluation design | Evaluation examined | Data collection methods | Evaluation participants and sample |
|---|---|---|---|---|---|
Australia/ Perth | Cyber Friendly Schools Student participation in policy decisions (Students sat on a decision-making group with staff) Bullying | Mixed methods | Feasibility Acceptability/ satisfaction Context | Students acting as cyberleaders were surveyed each year of the two-year intervention about their proposed activities and confidence in acting as a cyberleader. Cyberleaders also completed survey evaluations of the training activities at the end of the workshop and a brief telephone interview to measure their progress and any barriers to activity implementation at the end of each year. School project teams completed a baseline survey during the training workshop they attended and evaluations of training activities. Teaching staff completed baseline surveys prior to curriculum training and follow up surveys a year later. Project co-ordinators in each school were interviewed about their whole-school planning and activities at BL and at post tests in 2011 and 2012. | Students and school staff Cyberleaders completing surveys during training n = 87, n = 138 at post-test one, and n = 48 at post-test two. Teachers completing BL surveys n = 51. Teachers completing FU surveys at post-test 1 n = 78. Project co-ordinators completing interviews at BL n = 28. Project co-ordinator interviews at post-test 1: n = 25 and at post-test 2: n = 33. |
Bosma [27] USA/ Minnesota | DARE Plus Student participation in policy decisions (other approaches) Drug use and violence | Quantitative | Feasibility Fidelity/ quality | Data collected included number and types of activities, numbers and names of participants, issues addressed, and leadership roles of team members. Seven forms were used to collect quantitative data on: (a) adult one-on-ones, (b) youth one-on-ones, (c) presentations at community meetings to recruit support, (d) adult action team meetings, (e) youth action team meetings, (f) adult activities, and (g) youth extracurricular activities. | Students and community members Not stated |
Australia/ Perth | Friendly Schools, Friendly Families Student relationships with teachers but not student participation in decisions or delivery (other approaches) Bullying | Quantitative | Coverage /reach/ accessibility Acceptability/ satisfaction | Parent questionnaire covering knowledge about bullying; self-efficacy to talk to children about bullying; parents attitudes to bullying behaviour; frequency of parent-child communication about bullying; parents perception of their ability to help their children respond to bullying; parents perception that bullying is a-priority at the child’s school. Data collected at baseline and ten months and twenty-two month post-points. | Parents 3211 parents completed the questionnaire at baseline. (1,077 Grade 2 parents, 1094 Grade 4 parents and 1,040 Grade 6 parents). 2152 parents at post-test 1 (10 months) and 56% 1784 at post-test 2 (22 months). 45% (n D 1,444) of the original cohort of parents completed surveys at baseline, post-test 1 and post-test 2. |
Bond [24] Australia/ Victoria | Gatehouse Project Student relationships with teachers but not student participation in decisions or delivery (other approaches) Substance use, bullying | Mixed methods | Feasibility Acceptability/ satisfaction Mechanism Context | Field notes and records documenting meetings, the changes that occurred over time and the professional development provided to the school collected by the ‘critical friends’ from the Centre for Adolescent Health. Semi-structured interviews with ‘key informants’ (curriculum coordinators, student welfare, and administration) for each intervention school in the last year of implementation. At the end of each year, school background information was obtained on all schools via a structured interview with senior personnel. Audits related to school structures, policies, programs in place, strategies used to promote emotional well-being of students, and demographic information and could capture whole-school-level changes related to policies and programs. | School staff Not stated |
Knight [31] Uganda/ Luwero district | Good Schools Toolkit Student participation in policy decisions (Students sat on a decision-making group with staff) Violence | Quantitative | Fidelity/quality Coverage/ reach/ accessibility | Cross-sectional survey at endline to measure student and staff exposure to the toolkit. Data on delivery of intervention to schools was collected routinely by intervention programme officers. All interactions with the schools— including technical support visits, group trainings and telephone calls—were systematically documented by each program officer termly. School led implementation was measured using termly ‘action plans’ routinely completed by schools. Adoption of toolkit elements by schools were tracked by an independent ‘Study Process Monitor’. Head teachers were asked a standard set of questions about Toolkit structures in each school termly, a sub-set of which were verified by direct observation. | Students and school staff 1921 students and 286 staff completed endline surveys. |
UK/ south east England | Healthy School Ethos Student participation in policy decisions (Students sat on a decision-making group with staff) Substance use, violence | Mixed methods | Feasibility Coverage/ reach/ accessibility Acceptability/ satisfaction Context | Pre and post intervention survey of year-7 students conducted in private classrooms with support from two fieldworkers to examine reach. Semi-structured interviews with students and staff in intervention schools and intervention providers to examine feasibility, acceptability, awareness and contextual factors. Unstructured observations of various meetings to examine processes of participation and enable triangulation with interview accounts were also undertaken. Field notes written during observations and sometimes augmented later the same day from memory. | School staff, students and intervention providers Across all four schools, 721 (90.4 per cent) took part in the follow-up survey. Semi-structured interviews were carried out with: one head teacher per intervention school (n = 2); the external facilitator; the two trainers; and a sub-set of action team members. In one school three senior staff, one junior staff and one student involved in action groups took part (n = 5). In the other one senior staff, two junior staff and two students involved in action groups participated (n = 5). Interviews were also carried out with two other staff per school (one experienced one less experienced), four in total, who participated in the training as well as with three students in one school and five in the other who participated in other intervention activities and seventeen students in each school not participating in specific actions. |
Bonell [25] Fletcher [30] UK/ south east England | Learning Together (Pilot) Student participation in policy decisions (Students sat on a decision-making group with staff) Bullying, aggression, substance use | Mixed methods | Feasibility | To examine fidelity of implementation of action groups documentary evidence was collected via intervention facilitator checklists, action group meeting minutes and school policies. To assess delivery of student needs assessment, response rates for the baseline survey were examined to ascertain if they were greater or lower than 80% of all year 8 students at the school. To examine reach of staff training and the uptake of restorative practices, evidence was collected from training provider and facilitator checklists. To examine the delivery of the student curriculum, evidence from intervention facilitators checklists was collected. Observations of training and action group meetings were carried out. Focus groups were carried out with students and school staff and semi-structured interviews with school leadership at each participating school. A subsample of action group members and all intervention providers were interviewed to explore their views on the intervention. Action group members at each intervention school were surveyed to examine their views on acceptability. Intervention reach was assessed via student survey. A teacher survey also included questions on implementation. | Students and school staff Students (n = 1114) and teachers (n = 336) took part in the follow follow-up survey. 34 action group members plus 16 other students and 4 staff took part in interviews. 112 students took part in focus groups. |
Bonell [36] Warren [35] Warren [37] UK/ south east England | Learning Together Student participation in policy decisions (Students sat on a decision-making group with staff) Bullying, aggression, substance use | Mixed methods | Feasibility Fidelity/ quality Coverage/ reach/ accessibility Acceptability/ satisfaction Mechanism Context | n all schools, diaries were completed by trainers; researchers observed training; and staff completed a satisfaction survey for training. Semi-structured telephone interviews were also carried out with trainers. Diaries were kept by facilitators of action groups and minutes were reviewed. A survey was carried out with AG members each year of the intervention. Researchers carried out observations of action groups; semi-structured telephone interviews with AG facilitators in years one and two of the intervention; and semi-structured interviews with AG members (two per year) were also completed. A survey was carried out with staff leading curriculum implementation annually and semi-structured interviews were conducted each year with staff delivering the curriculum. In all schools a survey was also carried out termly with staff implementing restorative practice and interviews were conducted with other school staff in years 1 and 3. In six case study schools, focus groups were conducted with staff each year of the intervention and with students who were directly involved in intervention activities as well as those who were less so. Semi-structured interviews were also carried out with students involved in restoratives practice. | Students and staff Not stated. |
Mitchell [33] USA/ Portland | Portland Peers Programme Student participation in policy decisions (Students sat on a decision-making group with staff) Substance use | Mixed methods | Feasibility Acceptability/ satisfaction | Biannual administration of the Portland Peer Helper Scale. Parent questionnaire. Initial assessment of student drug use. Interviews with project and school staff and students. Direct observation of a sample of program activities. Review of the student database of referrals for assessment. | Students, parents project and school staff Sample of research participants not described. The peer helper sample group was approximately 45% male and 55% female. The authors report that diverse ethnic groups were represented in the samples of this study p.13. |
Beets [23] USA/ Nevada and Hawaii | Positive Action Student participation in policy decisions (Students sat on a decision-making group with staff) Substance use, violence | Quantitative (Cross sectional survey) | Fidelity/ quality Context | Year-end repeat cross sectional self-administered staff survey carried out in ten elementary schools at years 2 and 3 of the multiyear evaluation. The survey assessed: teacher perceptions of support from their administrators and connectedness with their school; their belief in their responsibility to teach social and character development concepts; their attitude towards Positive Action; the amount of the Positive Action curriculum delivered; and their usage of program-specific materials in the classroom and school wide context. School climate was assessed via of two series of questions assessing perceived administrative support and school connectedness. | Teachers In total 171 teachers in the ten elementary schools completed year-end process evaluations in year two and 191 in year three. |
Malloy [32] USA/ Chicago | Positive Action Student participation in policy decisions (Students sat on a decision-making group with staff) Substance use, violence | Quantitative | Fidelity/ quality Context | Teacher unit implementation reports (UIRs) from first year of implementation (2004-5) administered as six time-points. Teacher work climate survey administered at baseline and after training. | Teachers Forty-six teachers who completed both the URIs (for one or more of the units) and the Teacher Work Climate Survey were included in the analysis. |
O’Hare, [34] UK /south east England | Positive Action Student participation in policy decisions (Students sat on a decision-making group with staff) Substance use, violence | Mixed methods | Feasibility Mechanism Context | Teachers completed an implementation survey at the end of each unit of the programme for lessons and the end of every week for classroom activities used and whole-school activity to assess dose. Head teachers completed a school climate survey. Students completed a satisfaction questionnaire measuring their engagement and pupil-teacher relationships. Qualitative data was collected through classroom observations to assess fidelity (quality). Pupil focus groups and teacher and head teacher interviews were conducted to explore implementation. | Students and staff Nineteen teachers completed the implementation survey. For survey (engagement and relationships measures), 473 students moving through years 4 and 5 across fifteen schools. Twenty-five students from five schools randomly selected from the 15 took part in focus groups. One teacher and one head teacher was interviewed from each of the five randomly selected schools. |
Anyon [22] USA | Responsive Classrooms Student Participation in Decision-making (other approaches) Violence | Mixed methods | Feasibility Fidelity/ quality Acceptability/ satisfaction Context | Observation by trained research staff of teacher implementation of RC rated on a three-point scale. Classroom practices frequency survey (CPFS) to capture teachers’ self-reported use of intervention strategies. Focus groups carried out with a range of school staff to examine factors that constrain or enable implementation. | School staff Sample included thirty school staff. Twenty-four teachers participated in classroom observations at two time-points, and 19 teachers completed the classroom practices survey. Fifteen participants completed a first round of focus groups in fall 2013, and 19 individuals participated in spring 2014. |