- Study protocol
- Open Access
Remote population-based intervention for disruptive behavior at age four: study protocol for a randomized trial of Internet-assisted parent training (Strongest Families Finland-Canada)
BMC Public Health volume 13, Article number: 985 (2013)
Oppositional Defiant Disorder (ODD) is characterized by angry and noncompliant behaviour. It is the most common disruptive behaviour disorder (DBD), with prevalence estimates of 6-9% for preschoolers and is closely linked to several long-term difficulties, including disorders of conduct, mood, anxiety, impulse-control, and substance abuse. ODD in children is related to parental depression, family dysfunction, and impairments in parental work performance. Children displaying early DBDs exhibit more symptoms of greater severity, more frequent offences, and commit more serious crimes later in life. The goal of the Strongest Families™ Finland Canada (SFFC) Smart Website intervention research program is to develop and evaluate an affordable, accessible, effective secondary prevention parent training program for disruptive behaviour in preschoolers to prevent the negative sequelae of ODD. Strongest Families is an 11-session program with two booster sessions that focuses on teaching skills to: strengthen parent–child relationships; reinforce positive behaviour; reduce conflict; manage daily transitions; plan for potentially problematic situations; promote emotional regulation and pro-social behaviour and decrease antisocial behaviour.
This protocol paper describes an ongoing population-based randomized controlled trial (RCT) of high-risk 4 year-olds attending well-child clinics in Turku, Finland and environs to examine the effectiveness of the Strongest Families Smart Website intervention compared to an Education Control condition. Randomization consists of a 1:1 ratio for intervention versus the education group, stratified by the child’s sex. The participants randomized to the intervention group receive access to the Strongest Families Smart Website and weekly telephone coaching sessions. The participants randomized to the Education Control condition receive access to a static website with parenting tips. Children are followed using parental and daycare teacher measures at 6 and 12 months after randomization.
The Strongest Families Smart Website intervention is hypothesized to improve parenting skills, reduce child disruptive behaviour, reduce parental distress and improve family functioning. These results will likely inform subsequent investigations, public policy, and early treatment of childhood disruptive behaviour problems.
ClinicalTrials.gov # NCT01750996
Overview of childhood disruptive behaviour and parent training
Oppositional Defiant Disorder (ODD)  is the most common disruptive behaviour disorder (DBD) , with prevalence estimates of 6-9% for preschool children (with a higher percentage displaying symptoms that do not meet diagnostic criteria) [2, 3]. ODD is characterized by angry/irritable mood, argumentative/defiant behaviour, and vindictiveness. Population-based birth cohort studies have shown that childhood psychiatric problems are developmental precursors for a wide range of negative outcomes indicating risk of marginalization including peer rejection, school failure, psychopathology, substance abuse and criminality , and the prognoses are often poor [5–14]. Approximately half of those children whom have been identified as aggressive with externalizing behaviour at preschool age eventually develop persistent problems [15, 16] and there seems to be a developmental trajectory of early onset ODD that leads to Conduct Disorder (CD) in a proportion of these children . In children, ODD has also been linked to parental depression [17, 18], family dysfunction , and impairments in parental work performance .
Disruptive behaviour disorders such as ODD are among the most costly of early childhood psychiatric disorders . In children aged 3–8 years, DBDs cause substantial annual costs . Despite an established knowledge base, few strategies have been developed to prevent this trajectory at an early stage and at the population level.
Many parents react to defiant, oppositional behaviour with an increase in controlling strategies and a decrease in positive responses . Forty years ago, Patterson and Reid described the mechanisms via which this coercive pattern contributes to an escalation in disruptive behaviour and more serious antisocial behaviour . This pattern has been confirmed in longitudinal studies [17, 18], and leads to diminishing emotional regulation and poor peer relationships [22–24]. Elgar and colleagues  examined 4,184 parents and 6,048 ten to fifteen year-old children and youth enrolled in the 1998 and 2000 cycles of the Canadian National Longitudinal Survey of Children and Youth (NLSCY), and found that parenting style predicted both internalizing and externalizing behaviour in children and youth. Although genetics can play a role in disruptive behaviour [25, 26], an important implication of the research points to the possibility that early parenting interventions may have epigenetic effects [27, 28] that alter the genetic contribution to child outcomes.
Parent training has been shown to be the most effective approach to the prevention and treatment of disruptive behaviour [29–32] and it represents one of the most well-validated therapeutic techniques . The effectiveness of parent training has been established in small groups [34, 35], large groups [36, 37], and home-based coach-supported distance formats .
Parent training is a heterogeneous mode of treatment [39, 40], differing in theoretical orientation, amount of intervention, qualifications of the training administrator, mode of delivery, therapeutic components provided, and targeted recipients (e.g., parent only or with the child receiving therapy). In parent training interventions, parents typically learn to identify, define and observe problem behaviours in new ways, as well as learn strategies to prevent and respond to oppositional behaviour . A recent meta-analytic review of parenting programs found that the following program components were consistently related to larger effects: (1) increasing emotional communication skills and positive parent–child interactions, (2) teaching parents to utilize time-out and about the importance of consistent parenting, and (3) requiring practice of new skills between parents and children during parent training sessions .
Despite its promise, traditional parent training programs have significant limitations. In Finnish , Canadian , and American utilization studies , most (i.e., about 80%) children with externalizing problems do not receive timely treatment. Moreover, logistical barriers such as child care, transportation time, work schedules, stigma, or discomfort with services delivered in groups prevent many parents from enrolling in or completing parent training programs [39, 45–50]. Lastly, low income, limited education, maternal stress, and parental depression , can interfere with program completion, limiting the effectiveness of currently available models for families at greatest risk.
The Finnish universal health care system, with its high participation rate in child check-ups and recent emphasis on psychosocial well-being of families (including parenting), represents an exceptional opportunity to study the outcome of parenting skills programs targeted to families with children who present with high levels of oppositional behaviour problems.
The Strongest Families Finland Canada (SFFC) Smart Website Intervention Program
Based on the reviewed research, we concluded that an ideal intervention for disruptive behaviour problems should, (1) target behaviour problems emerging in the preschool years, (2) be effective, (3) be affordable enough to be implemented widely, (4) appeal to parents who do not use or have ready access to available, traditional programs, (5) be flexible enough to customize the intervention to meet the child’s and family’s needs, and (6) facilitate an integrated system that includes early identification at the population level and management of difficulties, with follow-up after intervention to maintain the effect.
The goal of the research program is to translate and replicate the Strongest Families™ telephone-based program in a Finnish population trial using a Smart Website delivery system. A population-based randomized controlled trial (RCT) is currently being conducted with 4 year-old children displaying disruptive behaviour. Specific objectives of the SFFC Smart Website intervention research program are to:
develop a collaborative network that builds partnerships among investigators that will inform the content of our existing program by harnessing the advances of technology to customize care using videos, audio-clips and written exercises that demonstrate skill implementation;
develop methods for early identification of challenging behaviour in primary health care in Finland;
evaluate the prevalence of early signs of disruptive behaviour in Finnish 4 year-old children;
develop the Strongest Families Smart Website intervention based on the Strongest Families telephone-based program  for the prevention and treatment of disruptive behaviour in 4 year-old children;
evaluate the effectiveness of the Strongest Families Smart Website intervention for the early identification and treatment of disruptive behaviour in 4 year-old children, compared to an Education Control condition; and
examine the possible moderating influence of program utilization using tracking reports of website activities (e.g., time on task, number of screens viewed, sessions completed).
The primary hypothesis of this study is that the Strongest Families Smart Website intervention will reduce child disruptive behaviour symptoms and impairment scores on the The Child Behavior Checklist-Parent Report Form (CBCL) 1½–5 years of the Achenbach System of Empirically Based Assessment (ASEBA)  after treatment and at one year follow-up, compared to the Education Control condition.
To supplement this primary hypothesis, it is also hypothesized that after treatment, and at one year follow-up, those participants randomized to the Strongest Families Smart Website intervention will exhibit improved scores on parenting style (The Parenting Scale [PS]) [53, 54], parental distress (Depression Anxiety and Stress Scale Short Form [DASS-21]) , and teacher evaluation of school-related child behaviour scores (The Child Behavior Checklist-Teacher Report Form [TRF] of the Achenbach System of Empirically Based Assessment [ASEBA]) , compared to the participants randomized to the Education Control condition.
Design of the study
The study design is a two parallel group RCT stratified by sex, with 1:1 individual allocation comparing the Education Control condition and the Strongest Families Smart Website intervention (Intervention). Best practice guidelines for conducing RCTs will be followed in accordance with the CONsolidated standards of Reporting Trials statement (CONSORT), European International Conference on Harmonisation (ICH) Good Clinical Practice Guidelines, and Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (trial registry: ClinicalTrials.gov # NCT01750996).
Our study protocol was approved by our Research Ethics Boards (i.e., Intermunicipal Hospital District of Southwest Finland Ethics Committee, IWK Health Centre Research Ethics). All data is collected with voluntary consent. The voluntary consent forms were formulated according to guidelines set by the Ethics Committee and they were approved in the process of the Ethics approval.
Screening and inclusion criteria
Inclusion criteria for the RCT are as follows: (1) the child meets the screening criteria (i.e., the age of 4 years); (2) native language of Finnish or Swedish for at least one of the parents; (3) residence in any of the participating municipalities; (4) in the screening phase, the child has had behavioural challenges for the last six months (score of 5 points or more on the Conduct subscale of the Strengths and Difficulties Questionnaire [SDQ]  and with some perceived problems by the parent in the impact section); (5) the parent has access to a telephone, computer, and an internet connection in their home (a computer with an internet connection is provided to families if needed); (6) and the ability of one of the parents to speak and read Finnish.
Participant recruitment began on October 1st, 2011 in Turku, Raisio, Kaarina and Naantali cities located in Southwestern Finland (total population of 254,974 at the end of 2012). On October 1st, 2012, seven smaller municipalities were enrolled in the study (total population of 76,915 at the end of 2012) .
The participants are selected from the Finnish National Population Register . These participants are mailed a study information package approximately one month in advance of the four-year clinic visit. This package contains a brochure reminding the participants of the upcoming extended child health check-up, information about the study required by the Ethics Regulations, and a health questionnaire containing a Finnish translation of the SDQ . The participants are asked to complete the health questionnaire and are encouraged to bring the completed questionnaire to the clinic. If the questionnaire is absent, the health nurses ask the participants to complete the questionnaire during the appointment. In the event that a participant is not interested in the study, the nurse notes “declined to participate”. All of the families of 4 year-old children attending the check-up receive a small token of appreciation from the study team (i.e., a children’s book). After the appointment, the health nurses mail the completed questionnaires to the study site, data is entered into a study database and the SDQ’s are scored.
Those participants with a child meeting the screening inclusion criteria are enrolled in the next phase of the study. The study staff complete a recruitment telephone call introducing the study in more detail and screening for eligibility. During this telephone call, if the participant expresses their preliminary interest in participation, they are registered in the Smart Website electronic platform called IRIS (Intelligent Research and Intervention Software). If the participant declines to participate or is ineligible for the study, a record of this is entered into the system. If recruitment is successful, the data submitted by study staff generates an automated email trigger from IRIS to the participant containing instructions to set up a password for IRIS. On first login, the participating parents are presented with the online consent form. A separate consent telephone call with the parents is scheduled a few days after the recruitment telephone call. During the consent telephone call, the staff confirms the parents’ online consent, and, if consent is provided, the online baseline measures in IRIS are released to be completed by the participating parent. Once the online baseline questionnaires are completed, IRIS sends a task prompt to the study staff to complete the randomization process.
Exclusion criteria for the child are as follows: is not speaking in full sentences; is deaf or blind; has received or is receiving behavioural treatment (i.e., parent training); or has a diagnosis of Autism or a Pervasive Developmental Disorder (PDD), Down’s syndrome, Fetal Alcohol Syndrome, mental retardation, genetic diagnosis that will lead to mental retardation, or a major mental health disorder (e.g., depression, psychosis). Exclusionary criteria for parents include: current involvement with child protection services (i.e., removal of child custody, investigation of child abuse or neglect); child not living at home; a major mental health disorder, physical or other severe illness; and long-term hospital visits or care that would interfere with study participation. The exclusion criteria for the child and family are based on parent report during the recruitment call.
As per Cohen’s  suggestion that standardized effect sizes of 0.20, 0.50 and 0.80 be considered as small (detectable), medium and large, respectively, we expect to observe medium to small incremental effects in the order of 0.30 to 0.35 standard units between the intervention and control groups, accounting for multiple comparisons using hierarchical linear modelling (HLM) analysis. This would require 250 participants per group for the trial, allowing for 30% attrition over time. It is expected that adding data from one year follow-up will increase the power to detect differences. An alpha of 0.05 was chosen on an a priori basis for the statistical analyses in order to decrease the probability of type II errors.
The two randomization sequences (i.e., sex stratification) are generated with a 1:1 ratio (i.e., intervention versus Education Control condition) using a computerized random permuted block sequence generator (Random Allocation Software ) with concealed block sizes to ensure study staff blinding. A sequential, double envelope system is utilized to conceal individual placement. The sequential envelopes are labeled and colour-coded according to sex. The delegated study staff complete randomization by selecting the next sequential envelope per applicable stratification, affix the sticker that reveals the condition placement on the participant’s source document and enter the group placement information into IRIS. The next study steps released by IRIS are dependent on the group placement entered; IRIS unlocks the appropriate user interface (i.e., Strongest Families Smart Website intervention with weekly telephone coaching-Intervention condition; Static website with parenting tips-Education Control condition). After randomization, the participants receive an email informing them of the randomization results and a link to the relevant website. In addition, the participants are not restricted from seeking other assistance for problems that they encounter.
This is a minimal risk trial. Staff are trained in risk management reporting protocols to identify and report any suspicion of abuse and neglect according to the local legal requirements.
The internet traffic between the participant and the website is protected by a HyperText Transfer Protocol Secure (HTTPS) communications protocol. The website is hosted on a secure server maintained by the University of Turku IT Services in Finland.
Education control condition
The participants randomized to the Education Control condition receive access to a static website with parenting tips as well as a single 45-minute telephone call with a coach reviewing the parenting tips. The Education Control was chosen as the Control condition as opposed to a waitlist control or standard care for the following reasons: (1) a waitlist control condition does not allow for a long-term follow-up, (2) for the majority of families in the Finnish population, standard care constitutes no care at all, and (3) the Education control was deemed as an ethical alternative for parents with a challenging child.
The participants randomized to the Intervention condition receive a Smart Internet version of the Strongest Families telephone-based program [38, 61, 62], an 11-session, evidence-based parenting program, that focuses on skills for strengthening parent–child relationships; reinforcing positive behaviour; reducing conflict; managing daily transitions; planning for potentially problematic situations; and promoting pro-social behaviour. The Strongest Families telephone-based program was initially designed to target 3–12 year-old children. For the current study, the program content was modified into a web-based format and in accordance with the developmental level of 4 year-old children. Minor culturally appropriate changes were made throughout the program to accommodate for issues such as differences in the schooling and daycare systems between Finland and Canada.
The participating parents are asked to complete the 11 Strongest Families sessions online. The participants are encouraged to complete one session per week and at times most suitable to their daily life using a computer with an internet connection. Only one parent is the active participant during the program; however, the participant is encouraged to share the program content with the other partner when possible. The child does not take part in the telephone calls or use the website. The participant works through the online material for each session including exercises, instructional videos and skill demonstration audio and video clips. During the sessions, the participants are required to complete specific knowledge-based and experience-related questions. The participants are encouraged to complete each session in approximately one week after which the session concludes with a weekly telephone call with a trained coach (with health care professional certification) who provides support, responds to participants’ questions and reviews successful implementation of the program skills. If session skill adoption is considered adequate based on the coach’s professional experience, the next session is introduced at the end of the telephone call. Otherwise, the participant is encouraged to spend additional time with session content to acquire a necessary skill level.
Description of strongest families components
The Strongest Families Smart Website intervention has four components. First, the parenting skills curriculum is based on our Strongest Families telephone-based program, an approach derived from programs developed by members of this team [36, 37]. There are also two booster sessions to encourage maintenance of skills and outcome gain (i.e., 2–3 months and 4–5 months after intervention). Second, it is a personalized website that tracks and uses activities and interactions to modify the intervention. Each participant’s interactions with the website are personalized by using such factors as the child’s name, child’s problems and strengths, and preferred activities. Each weekly session’s examples and homework tryout pages are populated with the personalized content . The participants are reminded of upcoming appointments or prompted if they had visited the website infrequently. Third, during the initial 11-week phase and booster sessions, the participants are assisted and monitored on their progress by a coach. The coach contacts the participant each week by telephone (45-minute call), reviews their progress, introduces new skills, facilitates the solution of problems, and provides support and encouragement. Prior to each session, the coach reviews the participant’s use of the website and any automatic messages sent to the participants.
In order to ensure treatment fidelity, a number of steps are followed, including the recording of telephone calls, strict adherence to standard study protocol procedures, cultural adaptation of the program material to Finnish, and training with coaches. Finnish coaches are health care professionals or semi-professionals with a background in children’s services. They are trained by an experienced Strongest Families clinician to ensure proper application of the program in Finland. Specifically, in addition to ongoing coaches’ meetings to discuss cases conducted initially every two weeks and then once a month with the Canadian team, three training and question/answer sessions for the Finnish group are conducted to ensure correct administration of the Strongest Families Smart Website intervention: (1) one week of intensive program content review, (2) one week of revised curriculum and research protocol training and regulation review per Good Clinical Practice according to the ICH guidelines, and (3) a final process review after the RCT had begun. The Finland study team has weekly study meetings, and coach supervision meetings are held weekly both individually as well as with all of the coaches to review and discuss the cases. All of the telephone calls are recorded and a portion are audited by the Finland coach supervisor and scored for competency evaluation. If scores do not reach competency, additional training is completed with the coach using additional telephone call evaluations.
Recruitment for the RCT is currently underway, with 337 participants on August 27th, 2013. The Strongest Families Smart Website-Finland was launched in January, 2012. We aim to recruit a sample of 500 families and to complete the one year follow-up in 2015 (see Figure 1 for SFFC Smart Website Intervention Consort Flow Diagram).
The child and parent applicable measures are completed online using secured access to the Strongest Families Smart Website intervention. The teachers are asked to complete the SDQ and TRF on paper and submit the completed questionnaires to the study team. With the exception of the screening measure, all post-screening measures are administered at baseline, at 6 months and at 12 month follow-up.
Screening for the RCT is conducted using the Strengths and Difficulties Questionnaire (SDQ) . The SDQ is a brief behavioural screening questionnaire for use with 3–16 year-old children and adolescents. The measure is composed of 25 items that are divided between 5 scales: (1) emotional symptoms, (2) conduct problems, (3) hyperactivity/inattention, (4) peer relationship problems, and (5) prosocial behaviour. Research has found that the SDQ displayed adequate validity in Finnish samples of school-aged children and adolescents, with one study of Finnish school-aged children and adolescents reporting inter-rater agreement for total scores on the parent, teacher, and self-report measures of 0.38-0.44 and internal consistency of 0.71 [64, 65]. Another study of Finnish adolescents found correlations between the items and their respective subscales ranging from moderate to high (r = 0.47–0.73), and subscale internal consistency ranging from alpha = 0.53–0.71 . The conduct problem scale of the SDQ is used for screening purposes. Recent research with the parent-rated SDQ in preschool children found that the measure has internal consistency of alpha = 0.58 for the conduct problems scale and acceptable concurrent validity .
Background information on participant native language, marital and socioeconomic status (SES), parental education, and child developmental and medical history is collected.
Primary outcome measure
The Externalizing subscale of the Child Behavior Checklist-Parent Report Form (CBCL) for ages 1½-5 of the Achenbach System of Empirically Based Assessment (ASEBA) , the most widely employed, validated and age-normed measure of disruptive behaviour [68–74], is used as the primary outcome measure. The Externalizing subscale of the CBCL consists of 34-items that measure both the angry, irritable, defiant, uncooperative behaviour associated with ODD as well as a broader range of closely related externalizing problems . The CBCL has good test-retest reliability (e.g., 0.81) and criterion validity (e.g., 0.56-0.87) .
Secondary outcome measures
The Parenting Scale (PS)[53, 54] is used to measure parenting skills. The PS is a 30-item parent/caregiver report for children ages 1–12 years. It measures parenting and discipline styles, particularly those that are found to be related to the development and/or maintenance of child disruptive behaviour problems. The PS has adequate internal consistency (e.g., 0.84) and convergent validity (e.g., 0.53) .
The Parent Problem Checklist (PPC) is a 16-item scale measuring parental conflict. Interparental conflict is recognized as a risk factor for many childhood behavioural and emotional problems. This scale has good reliability and validity. High scores on the PPC have been associated with parental reports of increased child problems, general conflict, and lower marital satisfaction .
Parent’s sense of coherence
The Sense of Coherence Scale (SOC-13) is used to assess parents’ sense of coherence (i.e., a global view of the world and individual environment as comprehensible, manageable, and meaningful) [76, 77]. The SOC-13 consists of 13 questions with two anchoring phrases. The questions are rated between 1 and 7 on a Likert-type scale, with five items reverse-scored. The sum of all items yields a score from 13 to 91. The reliability and validity of the scale have been established in many studies, and the score of the SOC-13 has been found to correlate positively with various aspects of health and well-being, and negatively with perceived stressors and poor coping skills [76, 78]. A systematic review examining 127 studies using SOC-13 found Cronbach alphas ranging from 0.70 to 0.92 and means ranging from 35.39 (SD 0.10) to 77.60 (SD 13.80) points .
The Depression Anxiety and Stress Scale Short Form (DASS-21) is used to measure parental distress. It consists of three subscales (i.e., depression, anxiety and stress) combined into a measure of general distress . The DASS-21 demonstrates strong internal consistency (e.g., 0.93 for the total scale) and adequate construct validity (e.g., 0.69 for the total scale)  and is sensitive to the effects of parenting interventions [81–83].
Child’s callous and unemotional traits
The Inventory of Callous-Unemotional Traits (ICU) is a 24-item questionnaire designed to provide a comprehensive assessment of traits that have been proven to be important for designating a distinct subgroup of antisocial and aggressive youth. The ICU has acceptable internal consistency (e.g., 0.77) and convergent validity (e.g., 0.19-0.44) [84, 85].
Parent attention-deficit/hyperactivity symptoms
The Barkley Adult ADHD Rating Scale–IV (BAARS-IV-Quick Screen) is used to determine whether adult attention-deficit/hyperactivity disorder (ADHD) symptoms moderate the response of parents to the intervention . This 14-item measure takes 3–5 minutes to administer and is linked to DSM-IV diagnostic criteria. It is a reliable and valid measure of current and childhood ADHD symptoms .
Program utilization measure
Each participant’s time on the website is downloaded using appropriate time-out values and the percentage of primary screens the participants observed. Program satisfaction and therapeutic alliance questions are released at the end of treatment for the intervention group only. Program satisfaction is also measured at 12 months.
All aspects of the trial (e.g., data collection, telephone calls to the participants) are monitored to ensure that data is valid. Staff are centralized in a small call centre and carefully supervised to ensure the system’s integrity.
Data management and analysis
Latent growth analysis, mixed effects regression, structured covariance analysis, or hierarchical linear modeling (HLM) will be used. The primary analysis will model the pattern of differences between the groups using growth curve analysis and the statistical software MLwinN . With the primary outcome of CBCL externalizing scores, the analysis will treat repeated measures within individuals as Level 1, and differences in the coefficients between the Strongest Families Smart Website intervention and Education Control conditions as Level 2. In effect, the approach will be to estimate parameters of growth (i.e., the slope of the outcome scores over time) for each individual, and compare these between the Strongest Families Smart Website intervention and Education Control conditions. In addition to comparing trajectories of response among children allocated to the two arms of the trial, net differences in response will be estimated by transposing the intercept to the end of the trial . This will allow a formal test of outcome differences. In addition, aggregate anonymous analysis of the characteristics of eligible children whose families did not consent to participate in the trial will be conducted. This will be important in understanding to whom the results of the trial can be generalized. The next step will be to test the possible moderating influence of program utilization using tracking reports of website activities (e.g., time on task, number of screens viewed, sessions completed).
Disruptive behaviour disorders, such as ODD, are among the most costly of early childhood psychiatric disorders. We expect to have an efficient method to improve access to parenting services that are conveniently delivered to families in the comfort and privacy of their home, removing typical access barriers [39, 45–50]. The intervention builds on earlier RCTs, as well as the knowledge of decision makers and parents about what will work in the real world environment. The Strongest Families Smart Website intervention developed through the SFFC program was designed to reduce the prevalence of child disruptive behaviour symptoms through early detection and accessible treatment using this innovative web-based delivery system. Our consumer preference studies [49, 50, 61, 90–92] and our experience with Strongest Families  show that personalized coaching is critical to the success of the program.
We are conducting a population-based RCT study of 4 year-olds exhibiting elevated levels of ODD to examine a preventative intervention in the real world rather than in a more controlled situation. The information collected from this study will provide a detailed understanding of how the Strongest Families Smart Website intervention functions, how it can be improved, translated and disseminated. It is anticipated that these results will inform subsequent investigations, public policy, and the treatment of childhood disruptive behaviour problems.
Strengths and implications
Few interventions to date have employed a population sample and a solid understanding of long-term treatment effects has not been achieved. Importantly, few studies have examined interventions that incorporate the power of interactive web technology to provide a personalized and sustainable intervention for the public health system. This trial will allow for many individuals, including those at greatest risk (e.g., isolated and low income families) who often do not obtain services, to receive evidence-based specialist care. In doing so, this trial will extend the research supporting the effectiveness of parent training for the treatment of ODD by conducting a longitudinal evaluation of an evidence-based, family-oriented web intervention in a Finnish population sample. The incorporation of a Finnish sample allows for a multitude of data to be gathered, as over 99% of the families participate in the health check-ups in maternity and well-baby clinics when invited .
Achenbach System of Empirically Based Assessment
- BAARS-IV-Quick Screen:
Barkley Adult ADHD Rating Scale–IV
Child Behavior Checklist
CONsolidated Standards Of Reporting Trials Statement
Depression Anxiety and Stress Scale Short Form
Disruptive Behaviour Disorder
Hierarchical Linear Modelling
HyperText Transfer Protocol Secure
International Conference on Harmonisation
Inventory of Callous-Unemotional Traits
Intelligent Research and Intervention Software
National Longitudinal Survey of Children and Youth
Oppositional Defiant Disorder
Pervasive Developmental Disorder
Parent Problem Checklist
Randomized Controlled Trial
Strengths and Difficulties Questionnaire
Strongest Families Finland Canada
Sense of Coherence Scale
Teacher Report Form.
American Psychiatric Association: Diagnostic and statistical manual of mental disorders. 2000, Washington: American Psychiatric Association, 4
Nock MK, Kazdin AE, Hiripi E, Kessler RC: Lifetime prevalence, correlates, and persistence of oppositional defiant disorder: results from the National Comorbidity Survey Replication. J Child Psychol Psychiatry. 2007, 48: 703-713. 10.1111/j.1469-7610.2007.01733.x.
Maughan B, Rowe R, Messer J, Goodman R, Meltzer H: Conduct disorder and oppositional defiant disorder in a national sample: developmental epidemiology. J Child Psychol Psychiatry. 2004, 45: 609-621. 10.1111/j.1469-7610.2004.00250.x.
Burke JD, Loeber R, Birmaher B: Oppositional Defiant Disorder and Conduct Disorder: A review of the past 10 years, part II. J Am Acad Child Adolesc Psychiatry. 2002, 41: 1275-1293. 10.1097/00004583-200211000-00009.
Achenbach TM, Howell CT, McConaughy SH, Stanger C: Six-year predictors of problems in a national sample: III. Transitions to young adult syndromes. J Am Acad Child Adolesc Psychiatry. 1995, 34: 658-669. 10.1097/00004583-199505000-00018.
Caspi A, Moffitt TE, Newman DL, Silva PA: Behavioral observations at age 3 years predict adult psychiatric disorders. Longitudinal evidence from a birth cohort. Arch Gen Psychiatry. 1996, 53: 1033-1039. 10.1001/archpsyc.1996.01830110071009.
Loeber R, Burke JD, Lahey BB, Winters A, Zera M: Oppositional defiant and conduct disorder: A review of the past 10 years, part I. J Am Acad Child Adolesc Psychiatry. 2000, 39: 1468-1484. 10.1097/00004583-200012000-00007.
Hofstra MB, van der Ende J, Verhulst FC: Child and adolescent problems predict DSM-IV disorders in adulthood: A 14-year follow-up of a Dutch epidemiological sample. J Am Acad Child Adolesc Psychiatry. 2002, 41: 182-189. 10.1097/00004583-200202000-00012.
Moffitt TE, E-Risk Study Team: Teen-aged mothers in contemporary Britain. J Child Psychol Psychiatry. 2002, 43: 727-742. 10.1111/1469-7610.00082.
Kim-Cohen J, Caspi A, Moffitt TE, Harrington H, Milne BJ, Poulton R: Prior juvenile diagnoses in adults with mental disorder: Developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry. 2003, 60: 709-717. 10.1001/archpsyc.60.7.709.
Simonoff E, Elander J, Holmshaw J, Pickles A, Murray R, Rutter M: Predictors of antisocial personality. Continuities from childhood to adult life. Br J Psychiatry. 2004, 184: 118-127. 10.1192/bjp.184.2.118.
Fergusson DM, John Horwood L, Ridder EM: Show me the child at seven: the consequences of conduct problems in childhood for psychosocial functioning in adulthood. J Child Psychol Psychiatry. 2005, 46: 837-849. 10.1111/j.1469-7610.2004.00387.x.
Sourander A, Multimäki P, Nikolakaros G, Haavisto A, Ristkari T, Helenius H, Parkkola K, Piha J, Tamminen T, Moilanen I, Kumpulainen K, Almqvist F: Childhood predictors of psychiatric disorders among boys: A prospective community-based follow-up study from age 8 years to early adulthood. J Am Acad Child Adolesc Psychiatry. 2005, 44: 756-767. 10.1097/01.chi.0000164878.79986.2f.
Sourander A, Elonheimo H, Niemelä S, Nuutila AM, Helenius H, Sillanmäki L, Piha J, Tamminen T, Kumpulainen K, Moilanen I, Almqvist F: Childhood predictors of male criminality: A prospective population-based follow-up study from age 8 to late adolescence. J Am Acad Child Adolesc Psychiatry. 2006, 45: 578-586. 10.1097/01.chi0000205699.58626.b5.
Richman N, Stevenson J, Graham PJ: Pre-school to school: A behavioural study. 1982, London: Academic Press
Fischer M, Rolf JE, Hasazi JE, Cummings L: Follow-up of a preschool epidemiological sample: Cross-age continuities and predictions of later adjustment with internalizing and externalizing dimensions of behavior. Child Dev. 1984, 55: 137-150. 10.2307/1129840.
Cunningham CE, Boyle MH: Preschoolers at risk for attention-deficit hyperactivity disorder and oppositional defiant disorder: Family, parenting, and behavioral correlates. J Abnorm Child Psychol. 2002, 30: 555-569. 10.1023/A:1020855429085.
Elgar FJ, McGrath PJ, Waschbusch DA, Stewart SH, Curtis LJ: Mutual influences on maternal depression and child adjustment problems. Clin Psychol Rev. 2004, 24: 441-459. 10.1016/j.cpr.2004.02.002.
Hyde JS, Else-Quest NM, Goldsmith HH, Biesanz JC: Children’s temperament and behavior problems predict their employed mothers’ work functioning. Child Dev. 2004, 75: 580-594. 10.1111/j.1467-8624.2004.00694.x.
Romeo R, Knapp M, Scott S: Economic cost of severe antisocial behaviour in children - and who pays it. Br J Psychiatry. 2006, 188: 547-553. 10.1192/bjp.bp.104.007625.
Patterson GR, Reid JB: Reciprocity and coercion: Two facets of social systems. Behavior modification in clinical psychology. Edited by: Neuringer C, Michael JL. 1970, New York: Appleton-Century-Crofts, 133-177.
Elgar FJ, Mills RSL, McGrath PJ, Waschbusch DA, Brownridge DA: Maternal and paternal depressive symptoms and child maladjustment: The mediating role of parental behavior. J Abnorm Child Psychol. 2007, 35: 943-955. 10.1007/s10802-007-9145-0.
Joussemet M, Vitaro F, Barker ED, Côté S, Nagin DS, Zoccolillo M, Tremblay RE: Controlling parenting and physical aggression during elementary school. Child Dev. 2008, 79: 411-425. 10.1111/j.1467-8624.2007.01133.x.
Kiernan KE, Huerta MC: Economic deprivation, maternal depression, parenting and children’s cognitive and emotional development in early childhood. Br J Sociol. 2008, 59: 783-806. 10.1111/j.1468-4446.2008.00219.x.
Caspi A, McClay J, Moffitt TE, Mill J, Martin J, Craig IW, Taylor A, Poulton R: Role of genotype in the cycle of violence in maltreated children. Science. 2002, 297 (5582): 851-854. 10.1126/science.1072290.
Caspi A, Moffitt TE, Morgan J, Rutter M, Taylor A, Arsenault L, Tully L, Jacobs C, Kim-Cohen J, Polo-Thomas M: Maternal expressed emotion predicts children’s antisocial behaviour problems: Using monozygotic-twin differences to identify environmental effects on behavioral development. Dev Psychol. 2004, 40: 149-161.
Moffitt TE: The new look of behavioral genetics in development psychopathology: Gene-environment interplay in antisocial behaviors. Psychol Bull. 2005, 131: 533-554.
Tremblay RE: Understanding development and prevention of chronic physical aggression: towards experimental epigenetic studies. Philos Trans R Soc Lond B Biol Sci. 2008, 363: 2613-2622. 10.1098/rstb.2008.0030.
Dretzke J, Frew E, Davenport C, Barlow J, Stewart-Brown S, Sandercock J, Bayliss S, Raftery J, Hyde C, Taylor R: The effectiveness and cost-effectiveness of parent training/education programmes for the treatment of conduct disorder, including oppositional defiant disorder, in children. Health Technol Assess. 2005, 9: 50-
Lundahl B, Risser HJ, Lovejoy MC: A meta-analysis of parent training: Moderators and follow-up effects. Clin Psychol Rev. 2006, 26: 86-104. 10.1016/j.cpr.2005.07.004.
Waddell C, Hua JM, Garland OM, Peters RD, McEwan K: Preventing mental disorders in children: A systematic review to inform policy-making. Can J Public Health. 2007, 98: 166-173.
Dretzke J, Davenport C, Frew E, Barlow J, Stewart-Brown S, Bayliss S, Taylor RS, Sandercock J, Hyde C: The clinical effectiveness of different parenting programmes for children with conduct problems: a systematic review of randomised controlled trials. Child Adolesc Psychiatry Ment Health. 2009, 3: 7-10.1186/1753-2000-3-7.
Kazdin AE: Bridging the enormous gaps of theory with therapy research and practice. J Clin Child Psychol. 2001, 30: 59-66. 10.1207/S15374424JCCP3001_7.
Pisterman S, McGrath P, Firestone P, Goodman JT, Webster I, Mallory R: Outcome of parent-mediated treatment of preschoolers with attention deficit disorder with hyperactivity. J Consult Clin Psychol. 1989, 57: 628-635.
Pisterman S, Firestone P, McGrath P, Goodman JT, Webster I, Mallory R, Goffin B: The role of parent training in treatment of preschoolers with ADHD. Am J Orthopsychiatry: Mental Health & Social Justice. 1992, 62: 397-408.
Cunningham CE, Bremner R, Boyle M: Large group community-based parenting programs for families of preschoolers at risk for disruptive behaviour disorders: Utilization, cost effectiveness, and outcome. J Child Psychol Psychiatry. 1995, 36: 1141-1159. 10.1111/j.1469-7610.1995.tb01362.x.
Cunningham CE: COPE: Large group, community based, family-centred parent training. Attention deficit hyperactivity: A handbook for diagnosis and treatment. Edited by: Barkley RA. 2006, New York: Guilford Press, 480-498. 3
McGrath PJ, Lingley-Pottie P, Thurston C, MacLean C, Cunningham C, Waschbusch DA, Watters C, Stewart S, Bagnell A, Santor D, Chaplin W: Telephone-based mental health interventions for child disruptive behavior or anxiety disorders: Randomized trials and overall analysis. J Am Acad Child Adolesc Psychiatry. 2011, 50: 1162-1172. 10.1016/j.jaac.2011.07.013.
Bourke ML, Nielsen BA: Parent training: Getting the most effective help for the most children. J Psych Prac. 1995, 1: 142-152.
Kazdin AE: Parent management training: Evidence, outcomes, and issues. J Am Acad Child Adolesc Psychiatry. 1997, 36: 1349-1356. 10.1097/00004583-199710000-00016.
Kaminski JW, Valle LA, Filene JH, Boyle CL: A meta-analytic review of components associated with parent training program effectiveness. J Abnorm Child Psychol. 2008, 36: 567-589. 10.1007/s10802-007-9201-9.
Pihlakoski L, Aromaa M, Sourander A, Rautava P, Helenius H, Sillanpää M: Use of and need for professional help for emotional and behavioral problems among preadolescents: A prospective cohort study of 3- to 12-year-old children. J Am Acad Child Adolesc Psychiatry. 2004, 43: 974-983. 10.1097/01.chi.0000127574.41662.fa.
Offord DR, Boyle MH, Szatmari P, Rae-Grant NI, Links PS, Cadman DT, Byles JA, Crawford JW, Munroe Blum H, Byrne C, Thomas H, Woodward CA: Ontario Child Health Study II: Six-month prevalence of disorder and rates of service utilization. Arch Gen Psychiatry. 1987, 44: 832-836. 10.1001/archpsyc.1987.01800210084013.
U.S. Department of Health and Human Services: Mental Health: A report of the surgeon general. 1999, Rockville: U.S. Department of Health and Human Services
Cunningham CE, Boyle M, Offord D, Racine Y, Hundert J, Secord M, McDonald J: Tri-ministry study: Correlates of school-based parenting course utilization. J Consult Clin Psychol. 2000, 68: 928-933.
Kazdin AE, Holland L, Crowley M: Family experience of barriers to treatment and premature termination from child therapy. J Consult Clin Psychol. 1997, 65: 453-463.
Barkley RA: Commentary on the multimodal treatment study of children with ADHD. J Abnorm Child Psychol. 2000, 28: 595-599. 10.1023/A:1005139300209.
Lingley-Pottie P, McGrath PJ, Andreou P: Barriers to mental health care: Perceived delivery system differences. ANS Adv Nurs Sci. 2013, 36: 51-61. 10.1097/ANS.0b013e31828077eb.
Cunnigham CE, Deal K, Rimas H, Buchanan DH, Gold M, Sdao-Jarvie K, Boyle M: Modeling the information preferences of parents of children with mental health problems: A discrete choice conjoint experiment. J Abnorm Child Psychol. 2008, 36: 1123-1138. 10.1007/s10802-008-9238-4.
Cunningham CE, Chen Y, Deal K, Rimas H, McGrath P, Reid G, Lipman E, Corkum P: The interim service preferences of parents waiting for children’s mental health treatment: A discrete choice conjoint experiment. J Abnorm Child Psychol. 2013, 41: 865-877. 10.1007/s10802-013-9728-x.
Reyno SM, McGrath PJ: Predictors of parent training efficacy for child externalizing behavior problems – a meta-analytic review. J Child Psychol Psychiatry. 2006, 47: 99-111. 10.1111/j.1469-7610.2005.01544.x.
Achenbach TM, Rescorla LA: Manual for the ASEBA preschool forms & profiles. 2000, Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families
Arnold DS, O’Leary SG, Wolff LS, Acker MM: The parenting scale: a measure of dysfunctional parenting in discipline situations. Psychol Assess. 1993, 5: 137-144.
Rhoades KA, O’Leary SG: Factor structure and validity of the parenting scale. J Clin Child Adolesc Psychol. 2007, 36: 137-146. 10.1080/15374410701274157.
Lovibond SH, Lovibond PF: Manual for the Depression Anxiety Stress Scales. 1995, Sydney: Psychology Foundation, 2
Goodman R: The Strengths and Difficulties Questionnaire: A research note. J Child Psychol Psychiatry. 1997, 38: 581-586. 10.1111/j.1469-7610.1997.tb01545.x.
Statistics Finland. http://www.stat.fi/tup/index_en.html,
The Finnish Population Register Centre. http://vrk.fi/default.aspx?id=39,
Cohen J: A power primer. Psychol Bull. 1992, 112: 155-159.
Random Allocation Software. http://mahmoodsaghaei.tripod.com/Softwares/randalloc.html#Random,
Lingley-Pottie P, McGrath PJ: Distance therapeutic alliance: The participant’s experience. ANS Adv Nurs Sci. 2007, 30: 353-366. 10.1097/01.ANS.0000300184.94595.25.
McGrath PJ, Lingley-Pottie P, Emberly DJ, Thurston C, McLean C: Integrated knowledge translation in mental health: Family help as an example. J Can Acad Child Adolesc Psychiatry. 2009, 18: 30-37.
Cunningham CE, Rimas H, Chen Y, Mielko S, Ratcliffe J, Holding A, McGrath P: Reducing children’s mental health waiting times: A conjoint analysis modeling the response of parents to self-paced programs combining therapist and computerized telephone coaching assistance. Psychologie Canadienne: 50(2a) for the Annual Meeting of the Canadian Psychological Association. 2009, Montreal: QC: Canadian Psychological Association, 169-
Obel C, Heiervang E, Rodriguez A, Heyerdahl S, Smedje H, Sourander A, Guðmundsson ÓÓ, Clench-Aas J, Christensen E, Heian F, Mathiesen KS, Magnússon P, Njarðvik U, Koskelainen M, Rønning JA, Morten Stormark K, Olsen J: The Strengths and Difficulties Questionnaire in the Nordic countries. Eur Child Adolesc Psychiatry. 2004, 13: 32-39.
Koskelainen M, Sourander A, Kalijonen A: The Strengths and Difficulties Questionnaire among Finnish school-aged children and adolescents. Eur Child Adolesc Psychiatry. 2000, 9: 277-284. 10.1007/s007870070031.
Koskelainen M, Sourander A, Vauras M: Self-reported strengths and difficulties in a community sample of Finnish adolescents. Eur Child Adolesc Psychiatry. 2001, 10: 180-185. 10.1007/s007870170024.
Klein AM, Otto Y, Fuchs S, Zenger M, Von Klitzing K: Psychometric properties of the parent-rated SDQ in preschoolers. Eur J Psychol Assess. 2013, 29: 96-104. 10.1027/1015-5759/a000129.
Webster-Stratton C: Randomized trial of two parent-training programs for families with conduct-disordered children. J Consult Clin Psychol. 1984, 52: 666-678.
Webster-Stratton C, Kolpacoff M, Hollinsworth T: Self-administered videotape therapy for families with conduct-problem children: Comparison with two cost-effective treatments and a control group. J Consult Clin Psychol. 1988, 56: 558-566.
Webster-Stratton C: Enhancing the effectiveness of self-administered videotape parent training for families with conduct-problem children. J Abnorm Child Psychol. 1990, 18: 479-492. 10.1007/BF00911103.
Webster-Stratton C: Individually administered videotape parent training: “who benefits?”. Cognit Ther Res. 1992, 16: 31-52. 10.1007/BF01172955.
Webster-Stratton C, Hammond M: Treating children with early-onset conduct problems: A comparison of child and parent training interventions. J Consult Clin Psychol. 1997, 6,5: 93-109.
Irvine AB, Biglan A, Smolkowski K, Ary DV: The value of the Parenting Scale for measuring the discipline practices of parents of middle school children. Behav Res Ther. 1999, 37: 127-142. 10.1016/S0005-7967(98)00114-4.
Behan J, Fitzpatrick C, Sharry J, Carr A, Waldron B: Evaluation of the Parents Plus Programme. The Irish Journal of Psychology. 2001, 22: 238-256.
Morawska A, Thompson E: Parent Problem Checklist: measure of parent conflict. Aust N Z J Psychiatry. 2009, 43: 260-269. 10.1080/00048670802653281.
Antonovsky A: The structure and properties of the sense of coherence scale. Soc Sci Med. 1993, 36: 725-733. 10.1016/0277-9536(93)90033-Z.
Eriksson M, Lindström B: Validity of Antonovsky’s sense of coherence scale: a systematic review. J Epidemiol Community Health. 2005, 59: 460-466. 10.1136/jech.2003.018085.
Feldt T, Leskinen E, Kinnunen U, Ruoppila I: The stability of sense of coherence: comparing two age groups in a 5-year follow-up study. Pers Individ Dif. 2003, 35: 1151-1165. 10.1016/S0191-8869(02)00325-2.
Henry JD, Crawford JR: The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. Br J Clin Psychol. 2005, 44: 227-239. 10.1348/014466505X29657.
Antony MM, Bieling PJ, Cox BJ, Enns MW, Swinson RP: Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychol Assess. 1998, 10: 176-181.
Sanders MR, Markie-Dadds C, Tully LA, Bor W: The Triple P-Positive Parenting Program: A comparison of enhanced, standard, and self-directed behavioral family intervention for parents of children with early onset conduct problems. J Consult Clin Psychol. 2000, 68: 624-640.
Bor W, Sanders MR, Markie-Dadds C: The effects of the Triple P-Positive Parenting Program on preschool children with co-occurring disruptive behavior and attentional/hyperactive difficulties. J Abnorm Child Psychol. 2002, 30: 571-587. 10.1023/A:1020807613155.
Morawska A, Sanders MR: Self-administered behavioral family intervention for parents of toddlers: Part I. Efficacy. J Consult Clin Psychol. 2006, 74: 10-19.
Essau CA, Sasagawa S, Frick PJ: Callous-unemotional traits in a community sample of adolescents. Assessment. 2006, 13: 454-469. 10.1177/1073191106287354.
Fanti KA, Frick PJ, Georgiou S: Linking callous-unemotional traits to instrumental and non-instrumental forms of aggression. J Psychopathol Behav Assess. 2009, 31: 285-298. 10.1007/s10862-008-9111-3.
Barkley RA: Barkley Adult ADHD Rating Scale–IV (BAARS-IV). 2008, New York: Guildford
Sonuga-Barke EJS, Daley D, Thompson M: Does maternal ADHD reduce the effectiveness of parent training for preschool children’s ADHD?. J Am Acad Child Adolesc Psychiatry. 2002, 41: 696-702. 10.1097/00004583-200206000-00009.
Rasbash J, Browne W, Goldstein H, Yang M, Plewis I, Healy M, Woodhouse G, Draper D, Langford I, Lewis T: A user’s guide to MLwiN, Version 2.1. 2000, London: Multilevel Models Project, Institute of Education, University of London
Singer JD, Willett JB: Applied longitudinal data analysis: Modelling change and event occurrence. 2003, New York: Oxford University Press
Lingley-Pottie P, McGrath PJ: A therapeutic alliance can exist without face-to-face contact. J Telemed Telecare. 2006, 12: 396-399. 10.1258/135763306779378690.
Lingley-Pottie P, McGrath PJ: Telehealth: a child and family-friendly approach to mental health-care reform. J Telemed Telecare. 2008, 14: 225-226. 10.1258/jtt.2008.008001.
Lingley-Pottie P, McGrath PJ: A paediatric therapeutic alliance occurs with distance intervention. J Telemed Telecare. 2008, 14: 236-240. 10.1258/jtt.2008.080101.
Ministry of Social Affairs and Health: Maternity and child health clinic services, school and student health services and preventive oral health services for children and youth. 2009, Helsinki, Finland: Publications of Ministry of Social Affairs and Health, 20-
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/13/985/prepub
The authors acknowledge the assistance of the following funding sources: The Jane and Aatos Erkko Foundation; the Finska Läkaresällskapet Research Foundation; the Sigrid Juselius Foundation; the Association of Friends of the University Children’s Hospitals; and Turku University Hospital Expert Responsibility Area Research Funding. The authors would also like to acknowledge Hannele Kallio and Riitta Liuksila as well as all of the physicians and health nurses at the child health clinics in Finland who assisted in the study. The participation of CC was supported by the Jack Laidlaw Chair in Patient-Centered Health Care.
PM may benefit indirectly from the study as he holds the trademark and copyright rights outside of Finland for the Strongest Families materials.
PM and AS led the study, were responsible for securing funding, and made substantive contributions to conceptualization and design of the study, as well as the writing and revision process. PM has given final approval of the manuscript. MA contributed to the planning of the study and manuscript revision. PC and AU assisted in the conceptualization and writing process, and have been involved in the oversight of data collection. PC and AU will also be involved in the writing and publication process. CW assisted in the conceptualization of the study and provided guidance in the use of computers in health care. All of the authors have contributed to the conceptualization, design and coordination of the study and helped to draft the manuscript. All of the authors have read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
McGrath, P.J., Sourander, A., Lingley-Pottie, P. et al. Remote population-based intervention for disruptive behavior at age four: study protocol for a randomized trial of Internet-assisted parent training (Strongest Families Finland-Canada). BMC Public Health 13, 985 (2013). https://doi.org/10.1186/1471-2458-13-985
- Disruptive behaviour disorders
- Children and youth
- Distance treatment
- Health service access
- Population-based study
- Web-based parent training