Sample size and power
The STARS cluster RCT sample size was selected to provide 85% power at the 5% level of significance to detect a difference in the mean teacher Strengths and Difficulties Questionnaire (SDQ) score [29] between trial arms equivalent to an effect size of 0.3 of a standard deviation or a difference of 2 points on the raw SDQ scale [23]. The STARS trial will work with one headteacher, one teacher, their class and parents from each of 80 state funded primary schools. The class must be a single year group class between Reception and Year 4 (children aged between four to nine years at recruitment) and the teacher must be in the class at least four days per week. The target sample size of children at final follow-up is 1600.
This sample necessarily sets the sample frame for the process evaluation. As the control condition is to receive the intervention with a year’s delay, quantitative attendance and satisfaction data will be collected from all 80 teachers, and there will be eight courses, comprising six sessions each, conducted by six group leaders working in pairs. There is little methodological guidance when planning sample sizes for qualitative studies, but empirical study suggests that data saturation is possible within the first 6–12 interviews, although many more are often conducted as in the case in this study [30]. We will systematically review our data for data saturation and stop data collection if it is clear that data saturation has occurred, or conduct additional interviews/focus groups if new themes are still emerging at the end of the planned data collection.
Sampling within the qualitative part of the process evaluation will be purposive [30,31] to facilitate data collection of the views and experiences of the full range of participants who can comment on trial processes and the delivery and the uptake and use of TCM strategies. Focus groups with teachers will be carried out with four of the eight TCM groups from the three cohorts, and will include teachers from both intervention and control arms. Forty five headteachers will be invited to take part in a telephone interviews: 15 headteachers from Cohort 1 and the intervention headteachers from Cohorts 2 and 3. Fidelity and attendance data will be collected on all groups and from all teachers attending TCM.
Data collection
Qualitative data
Data collection will run parallel to the pilot and main phases of the trial.
Focus groups
These are useful for collecting relevant and informative data on complex behaviour, such as what different groups think about a subject and why they hold such views [31]. Each focus group will have a researcher-facilitator and observer, will last for a maximum of one and half hours and will have ground rules (including confidentiality) and a clear structure. Researchers will follow a topic guide to facilitate discussion (available from the authors on request). There will be time for summary and reflection at the end of each group and participants will have an opportunity to discuss the focus group with a researcher via email should they wish. Intervention teachers unable to attend the focus group will be invited to take part in an individual telephone interview to elicit their views using the same topic guide.
Semi-structured interviews
Data will be collected using telephone interviews with headteachers. Telephone interviews can take the same form as face to face interviews, with a semi-structured format, but are usually shorter in length so can help improve the participation of busy professionals [32,33]. The interviews will last a maximum 30 minutes and will be pre-planned to optimise participation and privacy. The researcher will follow a topic guide (available from the authors on request).
Interviews and focus groups will be audio-recorded and transcribed for subsequent analysis.
Pilot Phase (Year 1)
Course experience and research processes
The focus will be on teachers’ and headteachers’ experiences and views of the TCM course content and delivery and trial processes (recruitment, arrangements and data collection), aiming to identify any changes needed prior to main phase. All intervention group teachers (n = 10) will be invited to join a focus group which will run soon after the course finishes.
Main Phase (Years 2–5)
Teacher learning and use of TCM strategies
In Year 2 two of the TCM groups (n =20) will be invited to attend focus groups. Additional sources of support used by staff to manage behaviour in the classroom will also be explored. In Year 3, one of the three TCM groups will be invited to join a focus group (n = 10) to provide additional data on perceptions of TCM and learning and use and transference of strategies. For teachers in their follow-up year (i.e. Years 2, 3 and 4) all teachers who attended a focus group the previous year will be invited to re-attend to explore the sustainability of teachers’ use of the TCM strategies.
Impact of course
In years 3 and 4 the headteachers from intervention schools (n = 30) will be interviewed one year post course to assess whether TCM has had any wider impact on their school, including any change in the use of outside support services, and dissemination within the school.
Quantitative measures
Data will be routinely collected relating to the administration of the TCM course to provide contextual information about the feasibility and acceptability of the course for teachers, and the fidelity with which groups are delivered.
Course attendance
The number of TCM sessions that each teacher attends out of a possible six will be recorded.
Teacher completed
Standardised session evaluations will be completed by teachers after each session that ask them to rate the content of the session, the videoed examples, the group leaders management of the session and the group discussions on a four point scale (“not helpful”, “neutral”, “helpful” and “very helpful”). Teachers will be encouraged to write any additional comments they may wish to on the feedback form. The Teacher Satisfaction Questionnaire will be completed after the final session to record the teachers’ view of and application of the techniques covered in the course. The questionnaire asks them to rate the course as “helpful”, “somewhat helpful” or “unhelpful” in developing their classroom management skills, whether they would recommend the TCM course to other teachers (“would not”, “might” and “would strongly”) and to rate 24 specific techniques covered in the course on a five point scale (“not all useful”, “not very useful” “neutral”, “a little bit useful”, “very useful”). Teachers are provided space to comment on anything that they would change about the course and the TCM strategies that they use regularly. Comments from all teachers’ feedback will be examined prior to each focus group to ensure adequate discussion of particular issues of pertinence to particular groups.
Group leader completed
Group leaders will complete standard checklists after each session that indicate which parts of the expected curriculum were covered in terms of concepts and strategies (coded: yes/no).
Video films
TCM sessions will be filmed for supervision with the Incredible Years Foundation, the developers of the TCM course. The group leaders will select a section of video from each session to be discussed as part of the supervision process and will be supported to apply for formal accreditation.
Analysis
Recent MRC guidance on process evaluation of complex interventions [24] suggests that theory driven approaches, which assess whether an intervention works by focusing on the assumptions about why it is thought to work, can help to illuminate contextual influences on uptake and utility of an intervention and variation in outcomes. As indicated earlier, TCM draws on a number of social learning theories relevant to the process evaluation that will explore teachers’ experiences of individual, social (group) and contextual influences on their own learning within the course, and their uptake and use of TCM strategies. One example is Bandura’s [34]model of self-efficacy, in which he suggests that a range of influences such as role play, coaching, past experience and ‘group’ learning and feedback, as well as individual emotional state, may influence self-efficacy, and in turn behaviour and performance. This and other theories can help guide analytical questions, in order to illuminate contextual and individual influences and to help identify what worked well or not so well for teachers within the context of TCM.
Qualitative data
All audio-taped qualitative data will be transcribed verbatim and anonymised. Data will be stored using Nvivo software [35] and will be password protected. Analysis will draw on understandings of social learning [14] and on subtle realist perspectives to help identify experiences as the lived ‘reality’ of participants [36]. The ways in which participants account for their experiences within the context of the trial and their own schools-based experiences [37] will be explored. Thematic analysis of interview and focus group data will be framed by research questions and will also allow for more inductive analysis whereby emergent themes are also identified. This approach will help explain the experience and views of teachers and headteachers.
In the analysis, ‘keyness’ of themes does not relate to the frequency of occurrence but to whether a theme captures information relevant to the research questions, in this case relating to a range of experiences and views about behaviours and contexts relevant to TCM strategies [38]. The Framework Approach [39] will be used to manage data and aid systematic analysis (description and summary of key themes, patterns and links in the data), allowing the researcher to move between levels of abstraction during analysis and between a theory driven and more inductive approach, whilst also displaying the relevant data sources. This approach will help maintain a focus on the process evaluation objectives for the different phases of the study. Data analysis will be carried out separately for each stage of the pilot and main trial phases and findings will be synthesized to provide feedback at appropriate stages within the trial.
Rigour and reliability
Two researchers will be supported by an experienced qualitative researcher who will provide interview training and on-going supervision during data collection and analysis. A number of methods will be adopted to enhance rigour during the study including: Purposive sampling to increase representation and transferability; multiple researchers; peer debriefing which allows for discussion and identification of themes and concepts to increase representation; recording of analytical discussions; checks for thematic saturation and consistency; use of the framework approach and an audit trail of analysis decisions to help minimise bias and track variability between researchers [40].
Quantitative data
Based on experience during our feasibility work [41], the distribution of attendance data is predicted to be highly negatively skewed, with most teachers attending most sessions, which will necessitate the use of non-parametric approaches to summarise and test them for differences between control and intervention arms.
Satisfaction with each session will be explored by reporting the proportions that endorse each response for each facet of the session (overall, videos, group leader, group discussion) by group and by session to look for systematic differences in how teachers experience the course. These can then be explored further in the data from focus groups and interviews. We will seek systematic differences between the teacher reported experience of each session across the whole trial, and between each of the eight groups that will be run using chi-squared. Finally, we will summarise satisfaction at the end of the course by the proportion reporting that they found the course “very helpful” and that they would “strongly recommend it” to a colleague. Similarly, we will explore the responses on the five-point Likert scale in relation to the different teaching techniques to seek strategies that were particularly popular/unpopular and triangulate these results with themes that subsequently emerge from the focus groups and interviews.
Fidelity to course content will be assessed by reporting the proportion of the prescribed curriculum covered in each session and over all from the group leader checklists, and examining whether this increased over time as group leaders gained experience.
We will examine whether the effectiveness of the intervention for improving child behaviour outcomes (based on the teacher-reported Strengths and Difficulties Questionnaire) is greater for pupils of those teachers that attended more TCM sessions. Because teacher attendance of TCM sessions is likely to be associated with factors that impact on pupil outcomes we will account for this confounding using instrumental variable methods described by Dunn and Bentall [42].
Ethics and consent
Ethical approval for this study has been obtained from the Peninsula College of Medicine and Dentistry Research Ethics Committee (now under the auspices of the University of Exeter Medical School Research Ethics Committee), reference Mar12/05b/141.
The STARS study is likely to elicit sensitive and confidential data and attention to ethics and participant confidence in and acceptance of researchers is crucial. Information about the process evaluation will be included in trial information sheets, and consent for participation in focus groups or individual interviews will be obtained at the same time as consent to take part in the trial. A reminder and additional information on focus groups will be given to teachers during the TCM course. Verbal consent will be taken for participation at the beginning of each interview or focus group. It will be made clear that participants have the right to withdraw their personal data from the study at any time.