A large landmark intervention study, using RCT design, necessitates an article dedicated to the description of methodological issues and their complexity, which requires more space than is usually allowed in the Methods section of an ordinary article in the majority of scientific journals. This paper describes the complex methodological issues in the SEYLE study, which will allow for adequate interpretation of study’s results generated over-time, as well as appropriate replication and development of the study in the future.
SEYLE is a multi-site RCT of interventions to promote mental health and prevent risk behaviours and suicide in European schools. Very few RCTs have been conducted on youth mental health and most of them have focused on a single intervention or treatment method on a small sample or at only one site or within only one country, or alternatively with a clinical population [26–35]. SEYLE was designed to evaluate three different active intervention methods that respectively empower students, teachers and professionals, compared to controls, to identify early mental health problems and risk behaviours, while facilitating appropriate referral to the healthcare system. The interventions were performed on a large sample (N=12,395) in eleven sites, located in eleven different European countries. Extensive procedures were implemented in order to guarantee a homogeneous methodology across sites, including high quality forward and back-translations of manuals, instruments, standardized interventions, as well as cultural adaptation for each participating country and expert review of all ethical issues related to the investigation.
The SEYLE project achieved the sampling size objective of enrolling at least 1,000 participating pupils at each site (except Austria; n=960), for a total sample of N=12,395 school-based adolescents. Female participants (55.2%) exceeded the number of male participants. It may be hypothesized that girls are more interested and/or collaborative in participating in a study dealing with psychological issues than males, leading to a higher participation. However, in most countries, there were no significant differences between the gender proportion in the school and the gender proportion in our sample.
Analysis of representativeness indicates that the study sites are reasonably representative of their respective countries, thus allowing for in-country and between-country comparisons. The overall response rate of schools was high (67.8%). Only Israel had a low response rate of schools (37.5%). Without Israel, response rate of schools was 72%. It can be hypothesized that the low response rate of schools in Israel was attributed to the nearly uniform attitude of school principals’ against using school time for additional non-educational activities, in view of the many such activities already taking place. Israel, along with Cork, Ireland and Oviedo, Spain, were the only sites where a majority of the participating adolescents were male. The Cork study site had the lowest pupil participation rate (64.6%), which can possibly be attributed to factors outside the scope of the study, as an environmental emergency affecting the region (flooding) at the time of the SEYLE study, thus preventing many pupils attending school when the baseline questionnaire was administered. However, overall pupil participation rates in SEYLE were high and thus assure adequate external validity of the collected data.
Drop-out rates at follow-up were low: 20.6% at 12-months, including 12.7% at three months follow-up, indicating broad acceptance of the interventions and questionnaires by both schools and pupils. Drop-out rates did not vary significantly among countries. Importantly, the study methodology required that the school randomization include all eligible schools in the area. This allowed for comparability of study Arms within and across sites. The main demographic indicators at baseline, such as mean age, family structure and parental unemployment did not differ significantly between the Active interventions and the Control Arm.
Internal reliability of each scale administered in each country also provides reassuring results. Cronbach’s alpha values were measured for both instruments translated for the purposes of SEYLE, as well as for instruments already available officially, in the respective languages. As reported in Table 6, Cronbach’s alpha values were quite homogenous across countries with very small variations and can be considered good or very good for all administered scales. The lowest internal reliability was reported for the SDQ (alpha=0.740). This result is more than acceptable and in agreement with previous studies .
The major strength of the SEYLE RCT is its application of a robust and homogenous methodology applied across eleven study sites in eleven different countries, selected to provide a broad geographical representation of Europe. Due to extensive collaboration across sites through Work Packages, that required cross-site cooperation of all participating sites throughout the study, uniform adherence to the study methodology was assured. Moreover, the standardized translation methodology and cultural adaptation allowed for the fine-tuning of interventions to be responsive to local cultural contexts, thus ensuring that the project was meaningful and useful data were collected at each site. Another major strength of the project is the inclusion of a control group and the selection of outcome measures, which are related to mental health and wellness, as well as risk behaviours, thus allowing for the study outcomes to be associated with three distinct interventions. Finally, the SEYLE interventions are able to be tested on a combined, large sample of European adolescents, generating the first such findings from a large-scale RCT of adolescent well-being in Europe, providing an important cohort that can be followed over time.
In any large-scale multi-site study using a complex methodology, securing sufficient funding is always an important challenge. In the case of SEYLE, there were two major limitations due to funding: namely, the funding duration precluded a long follow-up after the intervention ended. It would have been of greater value to identify the long-term effects of the SEYLE interventions by having a longer follow-up, as many preventive effects may only be observed after a longer time post-intervention. In fact, a five-year instead of a three-year timetable for SEYLE would probably have allowed for more knowledge to be gained regarding the study’s outcomes. In SEYLE, one site per country was chosen for study participation. Sufficient funds to allow the inclusion of more than one site per country would significantly have improved representation of the urban and rural areas and therefore understanding of different populations. Moreover, the analysis of representativeness of the recruited sample in relation to the respective country was limited by the availability of sociodemographic indicators in Eurostat at the local level (NUTS2). It was not possible to directly compare the SEYLE data and the same indicators at the country level because these were not available for the adolescent population or were collected with different methodologies, ultimately being incompatible.
Consent rates of schools and pupils varied across countries. The consent rates of pupils were very good in eight countries and lower in the three countries where extended consent procedures were imposed by the local ethics committees. However, it has been reported that response rates between 30% and 70% are, at most, only weakly associated with bias . Available indicators such as school size did not differ significantly between participating and non-participating schools with the exception of Slovenia, where more small schools participated in the study. The study was necessarily performed during school hours and consequently there was limited opportunity to collect other than questionnaire data regarding pupil’s behaviour. This school-based approach necessarily required a very limited number of outcome measures. Another limitation is that all data were collected through self-report questionnaires.