Skip to main content

Table 2 Quality of process evaluations

From: What factors influence implementation of whole-school interventions aiming to promote student commitment to school to prevent substance use and violence? Systematic review and synthesis of process evaluations

Study report

Were steps taken to minimise bias and error/increase rigour in sampling?

Were steps taken to minimise bias and error/increase rigour in data collection?

Were steps taken to minimise bias and error/increase rigour in data analysis?

Were the findings of the study grounded in/supported by data?

Was there good breadth and depth achieved in the findings?

Were the perspectives of young people privileged?

Overall reliability of findings based on the above (low, medium or high)

Overall usefulness of findings to this review low, medium or high

Cross [28, 29]

No

None specified. Recruitment and sampling of participants for process evaluation not discussed.

No

None clearly specified. Reference to use of a standardised protocol for project co-ordinator interviews.

No

None specified.

No

Discussion of findings is limited and does not flow clearly from methods as described. Full range of data is not discussed and there is a lack of clarity concerning where some findings are drawn from. Data tables or quotes from qualitative data are not provided to support findings.

No

Surface level data on training satisfaction for cyeberleaders discussed, but full range of findings from different data sources and participants not presented. Touches on a broad range of potential implementation issues identified in student data, but these are covered in very limited depth. Qualitative data is not well reported or utilised.

No

Presentation of student responses to surveys is limited and young people’s accounts from interviews are reduced to lists.

Low

Limited information on methodological rigour is provided. Unclear how bias and error were controlled for in sampling, data collection and analysis. Limited data presented to support findings.

Low

Some useful insights in relation to factors facilitating and impeding implementation of the intervention, but focus is mainly on cyberleader component and discussion of findings lacks depth.

Bosma [27]

No

None described.

No

None described.

No

Methods of analysis are not described.

No

Limited data provided to support findings. Data sources for many of the findings are not clear and limited data examples are used to support findings.

No

Findings are largely limited to describing levels of participation and frequencies of events/activities implemented. No data on participant views is presented.

No

Data on young people’s perspectives is not included.

Low

Methods are poorly described and it is unclear from what data many of the findings and conclusions are drawn.

Low

Findings on factors affecting implementation are limited.

Cross [28, 29]

No

Self-selected sample.

Yes

Yes

Appropriate statistical tests used.

Yes

Detailed data tables are provided to support findings.

No

Limited data from parent questionnaire examining reach and dose only.

No

Medium

Sampling open to self-selection, but otherwise well conducted study.

Low

Limited information on factors influencing implementation of parent component.

Bond [24]

No

Sampling methods for individual participants unclear.

No

None stated.

No

None stated.

No

Quotes used in places but not all findings are supported with data or linked clearly to data sources.

Yes

Multiple aspects of and perspectives on implementation explored in good depth using qualitative data.

No

Low

Limited information on sampling of schools and participants provided and findings not always supported by data presented. Limited information to assess quality and robustness of methods and analysis provided.

Medium

Provides useful information to understand how features of the intervention (and to a lesser extent) context may facilitate/support implementation. Range of data collection tools and multiple perspectives used to explore implementation, but not data from young people included.

Knight [31]

Yes

All students and staff invited to participate, with high student and staff response rates reported. All head teachers included in assessment of implementation.

Yes

Standardised data collection tools used. All measures pilot tested before use.

Yes

Triangulation (observation and teacher report) used in measurement of adoption. Appropriate statistical models and tests for reliability used.

Yes

Data described in detail and full data tables provided to support conclusions.

No

Range of measures, data collection methods and participants included to explore implementation, exposure/reach, but focus is on a limited set of questions on how level of implementation affected reach and outcomes, with some data provided on how reach varied by participants. In depth perspectives on implementation and factors shaping delivery or receipt are not explored.

No

Student survey data on reach is included and forms part of the main analysis, but wider perspectives on implementation are not included.

High

Standardised data collection tools used and triangulation with direct observation to verify findings on implementation. Survey measures piloted and were subjected to appropriate tests for reliability. Comprehensive data tables are provided to support conclusions.

Low

Some useful data on how receipt (reach) varies by gender, year group and educational needs but main focus is on how level of implementation impacts reach and outcomes. Does not contribute useful information on how features of interventions, context or providers influence implementation.

Bonell [26, 39]

Yes

Large subset of participants used for qualitative data collection, purposively selected to ensure diversity, capturing a range of roles and levels of involvement with the intervention. Student sample selected to broadly reflect student body.

Yes

Questionnaire was piloted with similar age students in another school and conducted, privately in classrooms with fieldworkers. Interviews were conducted by researchers in private rooms using standardised interview guides.

Yes

Interviews were recorded and transcribed in full. Two researchers coded the data (both inductively and deductively) and compared, contrasted their analysis and interpretation before coding a second time to arrive at a final set of themes. Unstructured observations of meetings were triangulated with interview accounts.

Yes

Supported with appropriately detailed data tables and illustrative quotes from a range of participants.

Yes

Range of aspects of implementation explored in detail via multiple participant perspectives and data sources, including in-depth qualitative data.

Yes

Student accounts used to address relevant research questions.

High

Steps taken to avoid bias and improve rigour in sampling, data collection and analysis. Data, including sample of appropriately illustrative quotes presented to support findings.

High

Provides highly useful information drawing largely on in-depth qualitative data from a range of participants to illustrate key factors facilitating and impeding implementation.

Bonell [25]

Fletcher [30]

Yes.

Schools purposively selected for diversity. For qualitative data collection individuals were selected purposively to represent diversity of students and staff.

Yes

Student surveys completed individually in classroom settings monitored by researchers not teachers. Staff surveys completed anonymously and confidentially. Interviews/FGs conducted by researchers in private rooms using standardised and pre-piloted interview schedules. Standardised data collection tools (checklists) also used to monitor implementation.

Yes

Detailed and layered qualitative analysis methods described, drawing on well recognised methodological approaches. Data thoroughly triangulated (verified) using range of data sources. Thorough approach to qualitative analysis including double researcher coding/analysis and regular discussion and refinement of coding and interpretation. Log book of decisions was used.

Yes

Findings flow logically from methods and are supported by detailed data tables, descriptions of participant responses and a number and range of quotes from different participants.

Yes

Perspectives from a range of participants on a number of different areas of implementation explored using both quantitative and qualitative data, providing good breadth and depth of enquiry.

Yes

Student responses considered in detail and lengthy quotes used to support interpretations.

High

Well conducted study with appropriate steps taken to increase rigour in sampling, data collection and analysis. Findings follow from methods and are well supported with detailed descriptions and data.

High

Range of implementation factors related to delivery and receipt considered using robust methodological approach. Provides high quality, in-depth information from a range of participants on factors influencing implementation.

Bonell [36]

Warren [35]

Warren [37]

Yes

Either all relevant sample included in research activities or participants were purposively sampled for diversity.

Yes

Surveys were anonymous, completed independently by students in classrooms with a researcher present and returned in envelopes sent to researchers. Interviews and FGs were conducted in private rooms with only researcher present.

Yes

Multiple data sources used to triangulate data. Analysis of qualitative data carried out by two researchers using standardised coding framework and recognised methods of qualitative analysis drawn from grounded theory, such and constant comparison and examination of deviant cases.

Yes

Findings are described in detail and follow logically from methods. Full data tables and appropriate, lengthy quotes from a number of participants are provided to support conclusions.

Yes

Range of data collection methods and participants included across all schools providing very comprehensive picture of implementation.

Yes

Survey data and qualitative data from young people is drawn upon and discussed in detail as part of the main findings.

High

Well conducted study which includes broad range of measures, methods and diversity of participants, with data collected over a three-year intervention period creating a very comprehensive and reliable picture of implementation.

High

Well conducted study using range of methods to capture diverse perspectives on breadth of implementation issues.

Mitchell [33]

No

Methods of sampling not described.

No

None stated. Data collection methods are poorly described.

No

None stated. Data analysis methods are not described.

No

Unclear from what data sources findings have been derived. Data is not presented to support findings.

No

Limited detail/depth to findings on implementation and qualitative data is poorly reported.

No

Interviews were carried out with students, but these are reduced to case studies written by researchers.

Low

Methods are poorly described so is difficult to assess rigour and quality of study. Discussion of findings is limited and sufficient data are not provided to support conclusions.

Low

Findings on implementation are limited and it is difficult to assess the rigor and quality of the study. Small amount of useful data provided on intervention acceptability and features of intervention that impeded implementation of parent component.

Beets [23]

No

All staff invited to participate, but response rates were low in some schools and sample may be subject to self-selection bias.

Yes

Surveys anonymised to promote more “truthful” answers (teachers only asked to identify year and grade level taught). Validated and pre-piloted scales used to measure key constructs.

Yes

Appropriate statistical analysis and testing used accordingly.

Yes

Data to support interpretations clearly presented in tables.

No

Focus is on teacher survey data relating to few key concepts related to implementation.

No

Study does not include student data.

Medium

Appropriate steps taken to minimise bias data collection and analysis, but not sampling. Data to support findings is presented.

Low

Very useful information on role of teacher beliefs and attitudes and perception of school climate in shaping implementation of curriculum and whole school materials, informed by appropriate theory. Although analysis is limited to quantitative data from teachers and small range of concepts and variables used.

Malloy [32]

Yes

All teachers invited to participate, with 73% response rate. Data collected for the teachers who did not take part showed they were not significantly different from those that did, suggesting the sample was representative.

Yes

Existing predictor variable measures were used and piloted a refined using principle component factors analysis prior to their use. Standardised data collection tools and measures used for teacher reported implementation.

Yes

Although UIRs were self-report and some of the implementation data were missing, weekly implementation data were triangulated with end of term summaries and with student reported levels of engagement with the program, which supported the validity of these data. Appropriate statistical tests used.

Yes

Findings follow logically from methods and full data tables are used to support conclusions.

No

Limited range of concepts related to implementation explored using staff survey data.

No

Focus on teachers.

High

Well conducted study with bias and error in sampling, data collection and analysis accounted for. Sufficient data to support findings is provided.

Low

Provides useful and reliable data on association between teacher perceived organisational climate and implementation, but sample and breadth and depth of analysis is limited, with other implementation issues relating to intervention, context or population not explored.

O’Hare, [34]

Yes

All students and teachers in study invited to participate in survey (although response rates not reported. Unclear if sample was representative). Schools selected at random for qualitative work. Students for FGs selected at random from five selected schools. Unclear how teachers for interviews were selected.

Yes

Survey measures were developed based on existing pre-tested scales. Observation, focus group and interview schedules were piloted in in each of the fifteen schools in an earlier phase and refined prior to implementation.

No

Methods of data analysis not described.

No

Full data tables are provided to support quantitative findings. Description and presentation of qualitative data is limited, however.

No

Range of methods used to capture information on different aspects of implementation from different perspectives. Depth of data on participant perspectives limited though.

No

Use of student data from FGs is limited.

Low

Steps taken to reduce bias and error in sampling and data collection but methods of analysis not described and description and presentation of qualitative data is poor.

Low

Although depth of data on implementation is limited, provides some useful data on student engagement and characteristics of interventions that might affect this (and consequently implementation and outcomes) as well as on feasibility for schools in terms of curriculum dose and challenges to implementing whole-school elements.

Anyon [22]

Yes

All staff members invited to participate. Potential for self-selection bias, but researchers claim sample was representative of total population.

Yes

Different instruments used to triangulate data on implementation. Standardised protocol used for qualitative data collection and previously validated instruments used for quantitative data collection.

Yes

Quantitative and qualitative data triangulated. Qualitative data analysis carried out by three independent coders. Appropriate statistical tests used for quantitative data.

Yes

Data presented to support quantitative findings. A number of appropriate participant quotes used in text to support qualitative findings.

Yes

Mixed methods used to capture both breadth (level of implementation across all classrooms) and depth (factors shaping this). Comprehensive data collection on implementation and factors shaping this. Range of teaching staff included to capture different perspectives, but lacks data on student perspectives.

No

No student data reported in study.

High

Steps taken to minimise bias in all areas.

Medium

Good quality, detailed information on implementation factors provided, but no student data reported.