- Research article
- Open Access
- Open Peer Review
Web-based interventions to decrease alcohol use in adolescents: a Delphi study about increasing effectiveness and reducing drop-out
BMC Public Healthvolume 15, Article number: 340 (2015)
Web-based computer-tailored (CT) interventions have a high potential to reach a large number of people and effectively change health risk behaviors and their determinants. However, effect studies show small and variable effect sizes, and these interventions also suffer from high drop-out. In this study we explored how Web-based CT interventions can be used effectively to reduce binge drinking in 16- to 18-year-old adolescents.
A three-round Delphi study was conducted. We invited experts to identify strategies to be used in Web-based CT interventions that can effectively decrease binge drinking in adolescents and to rate these strategies by importance. We asked to discriminate between interventions targeted for adolescents and those targeted for parents. Furthermore, we asked experts to suggest strategies for reducing drop-out and to indicate their importance.
Important strategies mentioned by the experts were: encouraging parents to set appropriate rules, encouraging consistent communication, and training refusal skills among adolescents. Concerning the reduction of drop-out from Web-based CT interventions experts came up with suggestions involving the content of the intervention (e.g., relevant material, use of language, tailored messages) but also involving the use of reminders and incentives.
The results of this explorative study provide useful strategies to increase effectiveness and decrease drop-out in future interventions.
Binge drinking, particularly for adolescents, is associated with a variety of negative consequences, such as fighting, being injured and injuring others , dating violence, attempting suicide, smoking, and using other (illicit) drugs . Furthermore, binge drinking (drinking ≥4/5 glasses per occasion for girls/boys)  is associated with brain damage and neurocognitive deficits  and can impair learning and school performance , and causes high societal costs, like health-care and law-enforcement costs, as well as costs for property damage and social work services . Binge drinking is prevalent in Europe with an average of 39% of adolescents having at least one binge drinking occasion in the previous 30 days . Therefore, interventions are needed to reduce binge drinking among adolescents [1,2], which should be targeted at their personal determinants (e.g., socio-cognitive variables) as well as their environment (e.g., parenting and peer influences) . The 16- to 18-year-old adolescent group has been largely understudied, with only a few studies focusing on this age group [8,9]. Most studies focused on either younger adolescents [10-14] or young adults who are 18 years and older [15-17].
An effective way to reduce binge drinking in adolescents could be through Web-based CT interventions. Web-based interventions have the potential to reach a large number of people, as access to the Internet is growing worldwide . Most often these interventions use the Social Cognitive Model (SCT) , Transtheoretical Model (TTM) , or Theory of Reasoned Action/Planned Behavior (TPB) [21,22] and its determinants to develop the intervention . Interventions built on TPB, however, led to substantially larger effects compared to the other theories .
The CT messages are developed by analyses of cognitive determinants of behavior and formulating feedback messages tailored to these determinants . Furthermore, individual characteristics of a person can be taken into account (i.e., demographics), which results in relevant and highly individualized information that is more likely to attract attention . Personalization and feedback have been shown to be effective working mechanisms of CT interventions . CT interventions have been proven to be efficacious in changing health risk behaviors and their determinants [27,28], but effect sizes, although statistically significant, are often only small to medium . This raises the question whether the right strategies were used to target the health behavior and this specific group. There are methods available to target determinants at several levels , but relatively little is known regarding how to translate these methods into strategies incorporated into Web-based CT interventions. A review has shown that the most commonly used behavior change techniques included providing information about consequences of behavior, prompting self-monitoring, identifying barriers, and providing problem solving skills, but those associated with the largest effects on behavior in Web-based interventions were stress management and general communication skills training . Other effective strategies were modeling, relapse prevention/coping planning, facilitating social comparison, goal setting, action planning, and feedback on performance . Furthermore, Web-based CT interventions can reach many people, but tend to have high drop-out rates [30-32]. This problem is common in eHealth effect studies and results in less power to reveal potential effects . Some studies investigated the effects of invitations to and incentives of surveys to reduce drop-out rates [34-36] and suggest that using incentives, short questionnaires and personalization of the invitation might be effective in increasing response rates. Yet, there is scarce knowledge about how to design an intervention to target that problem. Therefore, this issue should be addressed when developing Web-based CT interventions.
Although previous research has clearly identified determinants of adolescent binge drinking [16,37,38], to change these determinants some methods are more or less suitable depending on the target group and the way the intervention is delivered . Therefore, this study has two goals: first, we aim to identify the most suitable strategies for Web-based interventions aimed to change determinants and to reduce binge drinking among 16- to 18-year-old adolescents; strategies that may target adolescents’ personal factors, as they have been found to be important determinants of binge drinking [12,39,40], as well as their parents, who still have considerable influence on their children’s alcohol intake during this age period [14,41,42]. Second, we aim to identify strategies that can reduce drop-out of adolescents and parents in Web-based CT interventions.
We conducted a three-round Delphi study during a five-month period (Figure 1). A Delphi study is a method used to structure a group communication process in order to reach consensus to a complex problem . Although the number of rounds required is disputed, it appears that the majority of studies prefer either two or three rounds [43,44]. A three-round method is advantageous, since factors for which no clear consensus has been reached in the second round are offered another time to respondents for a critical review concerning their importance. During each round experts were invited to respond to a specific set of questions. The rounds were iterative in nature and each round took about 10 to 15 minutes to complete. Experts received an e-mail inviting them to participate in an Internet Delphi study. The e-mail contained a link to the online questionnaire. Two weeks after the first invitation a reminder was sent to non-responders, followed by a second reminder if needed after three weeks. Invited experts came from both research and practice backgrounds, to get a broad overview of the existing knowledge from both fields. These experts had experience with alcohol prevention projects or projects to reduce alcohol use for adolescents and young adults. They were invited to indicate whether their expertise involved interventions and studies about binge drinking adolescents targeting adolescents, those targeting parents, or both (Table 1).
In order to facilitate examination of the data and analyses and support accurate replication attempts to contribute to future meta-analyses [45,46], all study materials (i.e., questionnaires, data, syntax, and output of the analyses) are available at www.sciencerep.org/14.
The first round consisted of open-ended questions. Four main topics were covered with two to three sub-questions per topic. In order to prompt the experts to think of successful strategies, we first asked for all possible factors that determined binge drinking in adolescents. This technique, called elicitation, is often used to identify salient beliefs underlying behavioral determinants  (a list of the identified determinants can be found in the tables in the Additional file 1). The first three topics were about parenting practices/styles/actions, environmental factors, and motivational factors that influence binge drinking and had an identical structure. We then asked the following question: “What are, according to your expertise, effective parenting practices/styles/actions to reduce binge drinking in 16- to 18-year-old adolescents?” Subsequently, we asked: “How would you translate your knowledge about parenting practices/styles/actions into strategies to be used in a Web-based intervention aimed at parents to reduce adolescents’ binge drinking?” If participants indicated that they had experience with interventions targeting parents to reduce adolescents’ alcohol use, they were asked: “According to your expertise, changing which factors have been shown to be especially effective in an intervention aimed at parents to reduce alcohol consumption in adolescents?” The last topic was related to factors to reduce drop-out in interventions targeting adolescents and interventions targeting parents.
We invited 66 international experts to answer the questions from the first round of the Delphi study. Experts with research backgrounds were identified through a search using Google Scholar, PsycINFO and PubMed. If researchers previously published at least three articles on topics that we considered relevant (alcohol, prevention, adolescents, parents, or interventions), they were considered experts in this field. We then conducted a Google search to obtain further information about them on their institute Web sites (e.g., field of expertise, e-mail address) and invited them to participate. We also invited experts with practical backgrounds because they are often actively involved in implementing and conducting interventions, and have thus more experience in the application of interventions in the field. We reached them by approaching established national institutes that are very active in preventing alcohol and drug use (e.g., Trimbos Institute and Mondriaan Verslavingzorg). Eventually, 22 experts from six countries (Australia, Iceland, Sweden, The Netherlands, the United Kingdom, the United States) (33% response rate) filled out the questionnaire (Figure 1).
All answers given by the experts in the first round were categorized into a list of factors and strategies. First, all answers to one question were listed. Second, all double items were deleted. Finally, semantically similar items were taken together. The first step was done by one researcher only. Two more researchers were involved in the second and third steps. Consensus on the final list of items was reached through discussion . The entire research team approved the final questionnaire.
All experts from the first round were invited to participate in the second round. In addition, we searched Google Scholar, PsycINFO and PubMed to identify more experts in the relevant fields, as well as abstract books from relevant conferences (e.g., European Health Psychology Society (EHPS) and Kettil Bruun Society (KBS)). Of the 189 identified and invited experts, 64 from 11 countries (Australia, Brazil, Canada, Germany, Norway, Portugal, Sweden, Switzerland, The Netherlands, the United Kingdom, and the United States) responded to our request to participate in the second round (Figure 1).
Because we were mainly interested in the strategies to change determinants, we only asked for determinants in the first round to elicit the eligible strategy.
Experts from the second round were presented a list with all strategies to reduce binge drinking and strategies to reduce drop-out that were identified during the first round and to indicate the importance of each strategy using a seven-point Likert scale ranging from 1 (not important at all) to 7 (extremely important).
The data were analyzed by calculating the median score (Mdn), to indicate the importance of every strategy, and the interquartile deviations (IQD), to get an impression of the degree of consensus of the experts on the strategy . The median score can be defined as the score that falls exactly in the middle of a group of scores, meaning that exactly one half of all obtained scores lies above and the other half of all scores lies below this median score. In this study a median score of ≥6 is considered important. The IQD is a measure used to express the degree of consensus obtained, with a higher IQD referring to a smaller degree of consensus. When using a seven-point scale, IQDs with a value of ≤1 (more than 50% of the opinions fall within one point of the scale) indicate good consensus .
All experts that participated in the second round were invited to take part in the third and final round of the Delphi study. One expert refused participation in the final round and seven experts did not provide their names in the questionnaire. We therefore invited 56 experts in the final round, 39 of whom completed the questionnaire (Figure 1).
The questionnaire, including the feedback about median and IQD for each item from the second round, was sent to the participants to re-rate their answers from the prior round. Of all items, 17.9% had an IQD ≤ 1 and were taken out of the questionnaire. This resulted in the third round questionnaire consisting of 101 questions.
Ethical approval of the Regional Medical Ethics committee in the Netherlands was not necessary because participants in this study were not “subjected to procedures or required to follow certain rules of behavior” (http://www.ccmo.nl/en/your-research-does-it-fall-under-the-wmo).
During the first round a number of determinants of adolescent binge drinking were identified. For these determinants, the experts defined strategies to change that determinant in order to reduce binge drinking. These strategies, including the results of the second and third rounds, are listed in Tables 2, 3, 4 and 5.
In the first round, 70 factors that determine binge drinking in adolescents, 40 strategies targeting parents, and 47 strategies targeting adolescents were identified. With regard to reducing drop-out, 17 and 19 strategies were identified for adolescents and parents, respectively. In the second round, experts reached consensus (IQD ≤ 1) on 16 strategy items that were considered important (Mdn ≥ 6). In the third round, experts reached consensus on 33 important items. Both rounds taken together delivered a consensus of 49 out of 123 items (39%).
Strategies targeting parents
Concerning strategies for interventions that target parents, experts agreed on eight important items during the second round and another 10 items during the third round (Table 2). The strategies considered relevant can be categorized into the following: 1) parenting practices like setting rules, communication about alcohol, and monitoring of the child, and 2) parenting styles, such as being responsive and interested in the child.
Strategies targeting adolescents
A further goal of this study was to identify strategies to reduce binge drinking in adolescents in an intervention aimed at adolescents; these results are depicted in Table 3. Experts agreed on three important strategies during the second round and on another four during the third round. Most prominent were strategies to increase refusal skills. Other strategies were related to coping with negative emotions, dealing with drinking in social situations, and developing decision-making skills.
Strategies to reduce drop-out of adolescents
Concerning drop-out of adolescents, experts agreed on one important strategy during the second round and another 12 strategies in the third round. During the second round, experts only agreed on the importance of incentives. During the third round, strategies related to design and content of the intervention and the importance of reminders were emphasized (Table 4).
Strategies to reduce drop-out of parents
Concerning drop-out of parents, experts agreed on four important strategies in the second round and seven in the third round. Here the important strategies were related to the content and design of the intervention (e.g., usability, feasible recommendations, tailoring the intervention) and the use of reminders (Table 5).
The aim of this Delphi expert study was to gather expertise on effective strategies to be used in Web-based CT interventions to reduce binge drinking in 16- to 18-year-old adolescents. Some of the important strategies that could be used in an intervention targeted at parents are already described in the literature like specific parenting practices such as monitoring the adolescents’ whereabouts and friends [50-52], and being a responsive and interested parent . Results from previous studies regarding communication about alcohol, which was also considered important by our experts, are more heterogeneous. Some studies found a positive effect of communication on alcohol consumption [41,42], others show no effect of communication or even detrimental effects . Our experts highlighted specific aspects of communication other than just frequency. They recommend that communication should be consistent, kind, open, and of positive quality. One study  found indeed a difference in quality and frequency of communication about alcohol use, with quality being negatively associated with alcohol use and frequency being positively associated. To obtain more insight about which aspects of communication are useful in this context, more research is recommended . Nevertheless, when advising parents about communication with their children about alcohol, attention should be paid to communication being of good quality, open, kind, and consistent, as indicated by the experts in this study, rather than very frequent. Furthermore, experts in this study placed a high importance on setting clear and consistent rules which is in accordance with previous research [14,42,55]. However, what kinds of rules (e.g., zero tolerance rules, rules that are in line with the health guidelines, rules that are self-set by parents) have a different effect in reducing alcohol use in a target group that is legally allowed to purchase alcohol in comparison with other countries where this is illegal is still unclear. To our knowledge, no research has yet been conducted to entangle this problem. Finally, our experts reached consensus on the importance of emphasizing that the family continues to have influence throughout adolescence and young adulthood and that parents’ self-efficacy toward making agreements and setting rules should be strengthened. Applying these recommendations in Web-based interventions could mean creating a Web site for parents where they can obtain information about the importance of setting rules and communicating with their adolescent about alcohol and how to do this. To make this information as personalized and relevant as possible, computer-tailoring strategies could be used, thus assessing current communication and rule setting with the help of questionnaires and then providing the parents with personalized feedback based on their answers. These computer-tailored feedback messages could be either text-based or video-based, as both have been proven to be effective; however, video messages are preferred as they have been shown to be slightly more effective compared to text-based messages .
Regarding effective strategies targeting adolescents to reduce binge drinking, experts agreed on the importance of giving adolescents the opportunity to try out different reactions and their consequences in social situations, increasing refusal skills and perceived control in adolescents, and providing opportunities to cope with negative emotions in other ways than drinking. The Social Cognitive Theory (SCT)  assumes that self-efficacy is a very important factor that influences whether people perform a specific behavior. Self-efficacy can be increased by, for example, enactive mastery experience, modeling, or verbal persuasion . In enactive mastery experience individuals are confronted with different situations that increase in difficulty. They try to master them and receive feedback on their performance. One possibility is to confront adolescents with social situations in which alcohol is available and increase the difficulty by adding people and pressure to drink alcohol to the situations. This could be simulated in a Web-based intervention by using animations or videos that allow users to make choices that lead to different scenarios that result from their choices. Using modeling as a technique in Web-based interventions could be implemented by using short videos that show how other adolescents successfully refuse alcoholic drinks. Verbal persuasion techniques could be implemented by showing videos of adolescents that explain how they refuse drinks and encourage the adolescent that he or she also has the capability of refusing drinks and resisting peer pressure. Interventions that focused on preventing alcohol use in young adolescents (11 to 14 years of age) found that teaching techniques to manage social influences and pressure to drink and offering alternatives to alcohol  are effective in reducing alcohol use . Several other studies have shown that making coping plans was predictive for long-term lifestyle change in rehabilitation patients after discharge  and increased abstinence rates in quitters from smoking . These studies indicate that coping plans might be helpful in maintaining a healthy lifestyle and preventing unhealthy behavior, like binge drinking, if adolescents formulate coping plans for situations that are difficult for them.
Another goal of this Delphi study was to identify effective strategies to reduce drop-out as this constitutes a major problem to Web-based interventions. Gathering this expertise is very important, as there is little research available on effective strategies to reduce drop-out. For interventions aimed at adolescents, experts reached consensus on different strategies that can be divided into three categories: 1) providing incentives (non-monetary and monetary); 2) creating an appealing content (setting small goals to achieve during the intervention; using engaging graphics; offering self-assessment with personalized feedback; using highly relevant material, attractive designs, and inspiring topics; using language that relates to the adolescents; providing as little text and as much interaction as possible; designing an engrossing Website); and 3) sending reminders (e-mail and text messages). Generally, we can discriminate two kinds of drop-out. The first is intervention drop-out, meaning that participants drop-out during the intervention, and thus are not fully exposed to the intervention content which can negatively affect public health impact of the intervention . The other form is drop-out at follow-up assessment, thus participants not returning to a follow-up assessment. This form of drop-out diminishes the possibility to reveal possible effects . Both forms are problematic to intervention trials. Some of the strategies mentioned by the experts can either be used to reduce both forms of drop-out (e.g., providing incentives for completion of the intervention and for returning to the follow-up assessments), but other strategies work better to reduce one form of drop-out (e.g., using engaging graphics in the intervention to reduce intervention drop-out). The results regarding intervention content, which would be useful to reduce intervention drop-out, are of particular importance as limited experimental research has been conducted to test the effect of the content and layout of an intervention on drop-out rates. Most importantly, when creating a Web-based intervention, developers and researchers should collaborate closely with the target group to ensure that the chosen material is attractive, inspiring, and relevant. Methods to do this could be focus group interviews or a panel of the target group that evaluates all materials and provides feedback. Web-based interventions should be pilot tested and usability tests should be conducted in order to see how the intervention is used and understood and to get immediate feedback regarding which parts are appreciated and which not. Using computer-tailoring strategies will make the intervention much more personalized and relevant; however, in order to use the right language, motivational interviewing (MI) techniques  could be more appropriate. Although motivational interviewing usually is provided through personal contact between a professional therapist and a client, motivational interviewing techniques have already been successfully used in Web-based interventions to promote physical activity [63,64]. Through the use of computer tailoring, where responses are tailored to the answers given in the program, a dialog between the program and the user can be simulated [65,66]. One of the recommendations is to use a combination of open-ended and multiple choice questions for the MI questions. Open-ended questions can stimulate simple reflection and enable autonomy support while automated feedback messages to multiple choice questions can stimulate skillful reflections . An experiment with an avatar to strengthen the social relationship with the user was not associated with higher intervention impact . Recently, games for education and health promotion purposes, so-called “serious” games, have been developed and tested. The results concerning knowledge acquisition and attitude and behavior change are promising [67,68]. They further seemed to increase intrinsic motivation in adolescents [69,70], which is an important factor for continued intervention use.
When it comes to reducing drop-out in interventions aimed at parents, similar strategies can be used as experts again came up with many strategies relating to the content of the intervention and emphasized the importance of using reminders (e-mail or text messages). With regard to the content, experts mentioned the following strategies: ensuring parents realize following the intervention would make a difference; providing realistic and feasible recommendations; tailoring the intervention; using highly relevant material; providing interesting topics; using language that does not sound pompous or may be interpreted as condescending; ensuring parents they are the experts when it comes to their children; acknowledging that being a parent can be extremely difficult; and making the intervention salient to the parents. In addition to the earlier suggestions (i.e., collaborate closely with target group, use tailoring or motivational interviewing techniques), parents might also benefit from an approach based on goal-setting theory  in order to make realistic and feasible recommendations. This could be designed by creating a tool in which parents can choose from a series of sub-goals (e.g., have a first conversation with my child about alcohol, come to an agreement with my child about the amount of alcohol he/she is allowed to drink, make my rules clear to my child and explain consequences of noncompliance). For every sub-goal, further guidance can be provided regarding how to reach the goal, either text-based or, preferably, video-based . Other research thus far has shown that using a tunneled Web site, where visitors are more guided and have less control, increased the time spent on the Web site, number of pages visited, and knowledge gained compared to a Web site where the visitor could move freely . Furthermore, there is some literature available about methods to increase response to postal and electronic questionnaires. A review  of this literature identified some effective strategies: giving non-monetary incentives; offering survey results; using shorter questionnaires; personalizing electronic questionnaires by addressing the participants by name, using a picture and white background on the invitation; using interesting (relevant to participants) questions; sending reminders after the initial invitation; including a statement that others had responded; and setting a response deadline. Providing the incentive together with the questionnaire, rather than after the questionnaire was completed, increased response rates . Despite these effective methods, many methods to increase response rates have not proven to be effective, including monetary incentives for online questionnaires , contingent versus unconditional incentives , or offering cash lotteries (big and small amounts) as incentive . However, these results relate to survey research and not intervention research. Most of the strategies that have been evaluated on effectiveness are related to invitations or reminders to respond to a questionnaire, or incentives that participants received. More research is needed to test whether attractive, relevant, and interesting content can also reduce drop-out during an intervention. In particular, experimental research is needed, as most studies thus far were based on observational research [34-36], which does not allow conclusions about causal relationships.
Given the vast amount of mentioned strategies by the experts it becomes clear that there are many possibilities to decrease drop-out rates in Web-based interventions, but only few have been proven to be effective. Although it may seem wise to combine several strategies in order to increase their impact on retention rates, more experimental research is also needed to test unique and potential interaction effects of these strategies.
Limitations and strengths
Finally, we noticed that experts only agreed on a few important strategies to reduce binge drinking in adolescents in interventions targeting adolescents (7 out of 47) compared to interventions targeting parents (18 out of 40). We checked whether this could be an artifact of the sample selection, but researchers of parent-based interventions were not oversampled. Therefore, this finding could indicate that adolescents are a particularly difficult target group and that only a few strategies have proven to be effective. We included experts with research and practice background to get a broad overview of the existing knowledge from both fields. It would be interesting to compare strategies from researchers and practitioners to look for similarities and differences. Unfortunately, our sample of practitioners was too small to make meaningful comparisons, but we would recommend this for future research.
We also want to mention that the response rate from the first two rounds compared to the last round was relatively low (33% and 34% compared to 70%, respectively). Yet, comparable response rates have been reported in other Delphi studies [73,74]. The increase from the second to the third round may indicate that once experts agreed to participate in this study, this was likely to predict continued participation. Furthermore, there is no clear indication about decent panel sizes or acceptable response rates , so our goal was to reach saturation of information in the first round. We reached this goal with the answers provided by 22 experts participating in the first round.
This Delphi study identified strategies that can be used in a Web-based CT intervention to reduce binge drinking in 16- to 18-year-old adolescents and strategies to reduce drop-out rates from these interventions. The results of this explorative study can be used to inform future interventions.
Swahn MH, Simon TR, Hammig BJ, Guerrero JL. Alcohol-consumption behaviors and risk for physical fighting and injuries among adolescent drinkers. Addict Behav. 2004;29(5):959–63.
Miller JW, Naimi TS, Brewer RD, Jones SE. Binge Drinking and Associated Health Risk Behaviors Among High School Students. Pediatrics. 2007;119(1):76–85.
Verdurmen J, Monshouwer K, Dorsselaer S, Lokman S, Vermeulen-Smit E, Vollebergh W. Jeugd en riskant gedrag 2011. Utrecht: Trimbos-instituut; 2011.
Zeigler DW, Wang CC, Yoast RA, Dickinson BD, McCaffree MA, Robinowitz CB, et al. The neurocognitive effects of alcohol on adolescents and college students. Prev Med. 2005;40(1):23–32.
Rehm J, Mathers C, Popova S, Thavorncharoensap M, Teerawattananon Y, Patra J. Global burden of disease and injury and economic cost attributable to alcohol use and alcohol-use disorders. Lancet. 2009;373(9682):2223–33.
Hibell B, Guttormsson U, Ahlström S, Balakireva O, Bjarnason T, Kokkevi A, Kraus L: The 2011 ESPAD Report: Substance Use Among Stundents in 36 European Countries. 2011.
Green LW, Kreuter MW. Health Promotion Planning: an educational and environmental approach 3rd edn. CA: Mayfield: Mountain View; 1999.
Voogt CV, Kleinjan M, Poelen EA, Lemmers LA, Engels RC. The effectiveness of a web-based brief alcohol intervention in reducing heavy drinking among adolescents aged 15–20 years with a low educational background: a two-arm parallel group cluster randomized controlled trial. BMC Public Health. 2013;13(1):694.
Kuntsche E, Knibbe R, Gmel G, Engels R. Who drinks and why? A review of socio-demographic, personality, and contextual issues behind the drinking motives in young people. Addict Behav. 2006;31(10):1844–57.
Jones DJ, Hussong AM, Manning J, Sterrett E. Adolescent alcohol use in context: The role of parents and peers among African American and European American youth. Cult Divers Ethn Minor Psychol. 2008;14(3):266–73.
MacPherson L, Magidson JF, Reynolds EK, Kahler CW, Lejuez CW. Changes in sensation seeking and risk-taking propensity predict increases in alcohol use among early adolescents. Alcohol Clin Exp Res. 2010;34(8):1400–8.
Marcoux BC, Shope JT. Application of the Theory of Planned Behavior to adolescent use and misuse of alcohol. Health Educ Res. 1997;12(3):323–31.
Marsden J, Boys A, Farrell M, Stillwell G, Hutchings K, Hillebrand J, et al. Personal and social correlates of alcohol consumption among mid-adolescents. Br J Dev Psychol. 2005;23(3):427–50.
Van der Vorst H, Engels RCME, Meeus W, Dekovi M, Van Leeuwe J. The role of alcohol-specific socialization in adolescents’ drinking behaviour. Addiction. 2005;100(10):1464–76.
Clapp JD, Shillington AM. Environmental Predictors of Heavy Episodic Drinking. Am J Drug Alcohol Abuse. 2001;27(2):301–13.
Courtney KE, Polich J. Binge Drinking in Young Adults: Data, Definitions, and Determinants. Psychol Bull. 2009;135(1):142–56.
Ham LS, Hope DA. College students and problematic drinking: A review of the literature. Clinical Psychological review. 2003;23:719–59.
Bewick BM, Trusler K, Barkham M, Hill AJ, Cahill J, Mulhern B. The effectiveness of web-based interventions designed to decrease alcohol consumption–A systematic review. Preventive Medicine: An International Journal Devoted to Practice and Theory. 2008;47(1):17–26.
Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall; 1986.
Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: toward an integrative model of change. J Consult Clin Psychol. 1983;51(3):390–5.
Fishbein M. A theory of reasoned action: Some applications and implications. Nebr Symp Motiv. 1979;27:65–116.
Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179–211.
Webb LT, Joseph J, Yardley L, Michie S. Using the Internet to Promote Health Behavior Change: A Systematic Review and Meta-analysis of the Impact of Theoretical Basis, Use of Behavior Change Techniques, and Mode of Delivery on Efficacy. J Med Internet Res. 2010;12(1):e4.
Dijkstra A, De Vries H. The development of computer-generated tailored interventions. Patient Educ Couns. 1999;36(2):193–203.
De Vries H, Brug J. Computer-tailored interventions motivating people to adopt health promoting behaviours: Introduction to a new approach. Patient Educ Couns. 1999;36(2):99–105.
Dijkstra A. Working mechanisms of computer-tailored health education: evidence from smoking cessation. Health Educ Res. 2005;20(5):527–39.
Krebs P, Prochaska JO, Rossi JS. A meta-analysis of computer-tailored interventions for health behavior change. Prev Med. 2010;51(3–4):214–21.
Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull. 2007;133(4):673–93.
Bartholomew LK, Parcel GS, Kok G, Gottlieb NH, Fernandez ME. Planning Health Promotion Programs: An Intervention Mapping Approach. 3rd ed. San Francisco, CA: Jossey-Bass; 2011.
Elfeddali I, Bolman C, Candel M, Wiers R, Vries Hd. Preventing Smoking Relapse via Web-Based Computer Tailored Feedback: A Randomized Controlled Trial. J Med Internet Res 2012, 14(4).
De Vries H, Logister M, Krekels G, Klaasse F, Servranckx V, van Osch L. Internet-Based Computer Tailored Feedback on Sunscreen Use J Med Internet Res 2012, 14(2).
Kohl LFM, Crutzen R, Vries NK. Online Prevention Aimed at Lifestyle Behaviors: A Systematic Review of Reviews. J Med Internet Res. 2013;15(7):e146.
Eysenbach G. The Law of Attrition. J Med Internet Res 2005, 7(1).
Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires. Cochrane Database of Systematic Reviews 2009(3):1-527
Göritz AS. Contingent versus unconditional incentives in WWW-studies. Metodolosky Zvezki. 2005;2(1):1–14.
Göritz AS. Cash lotteries as incentives in online panels. Soc Sci Comput Rev. 2006;24(4):445–59.
Jander A, Mercken L, Crutzen R, Vries H. Determinants of binge drinking in a permissive environment: focus group interviews with Dutch adolescents and parents. BMC Public Health. 2013;13:882–95.
Wilks J, Callan VJ, Derek AA. Parent, Peer and Personal Determinants of Adolescent Drinking. Br J Addict. 1989;84(6):619–30.
Cooper ML. Motivations for alcohol use among adolescents: Development and validation of a four-factor model. Psychol Assess. 1994;6(2):117–28.
Kuntsche E, Knibbe R, Gmel G, Engels R. Why do young people drink? A review of drinking motives. Clin Psychol Rev. 2005;25(7):841–61.
Turrisi R, Jaccard J, Taki R, Dunnam H, Grimes J. Examination of the short-term efficacy of a parent intervention to reduce college student drinking tendencies. Psychol Addict Behav. 2001;15(4):366–72.
Spijkerman R, Van den Eijnden RJJM, Huiberts A. Socioeconomic Differences in Alcohol-Specific Parenting Practices and Adolescents’ Drinking Patterns. Eur Addict Res. 2008;14(1):26–37.
Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. MA: Addison-Wesley: Reading; 1975.
Mullen P. Delphi: myths and reality. J Health Organ Manag. 2003;17(1):37–52.
Crutzen R, Peters G-JY, Abraham C: What about trialists sharing other study materials? BMJ 2012, 345.
Peters G-JY, Abraham C, Crutzen R. Full disclosure: doing behavioural science necessitates sharing. The European Health Psychologist. 2012;14(4):77–84.
Fishbein M, Ajzen I. Predicting And Changing Behavior The Reasoned Action Approach. New York: Psychology Press Taylor & Francis Group; 2010.
Crutzen R, De Nooijer J, Brouwer W, Oenema A, Brug J, De Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res. 2008;23(3):427–39.
Jones J, Hunter D. Consensus methods for medical and health services research. Br Med J. 1995;311:376–80.
Kim Y-M, Neff JA. Direct and indirect effects of parental influence upon adolescent alcohol use: A structural equation modeling analysis. Journal of Child & Adolescent Substance Abuse. 2010;19(3):244–60.
Beck KH, Boyle JR, Boekeloo BO. Parental monitoring and adolescent alcohol risk in a clinic population. Am J Health Behav. 2003;27(2):108–15.
Borawski EA, Ievers-Landis CE, Lovegreen LD, Trapl ES. Parental monitoring, negotiated unsupervised time, and parental trust: The role of perceived parenting practices in adolescent health risk behaviors. J Adolesc Health. 2003;33(2):60–70.
Baumrind D. Current patterns of parental authority. Dev Psychol. 1971;4(1):1–103.
Ennett ST, Bauman KE, Foshee VA, Pemberton M, Hicks KA. Parent–child Communication about Adolescent Tobacco and Alcohol Use: What Do Parents Say and Does It Affect Youth Behavior? J Marriage Fam. 2001;63(1):48–62.
Van Der Vorst H, Engels RCME, Meeus W, Deković M. The impact of alcohol-specific rules, parental norms about early drinking and parental alcohol use on adolescents’ drinking behavior. J Child Psychol Psychiatry. 2006;47(12):1299–306.
Stanczyk N, Bolman C, Adrichem M, Candel M, Muris J, Vries H. Comparison of Text and Video Computer-Tailored Interventions for Smoking Cessation: Randomized Controlled Trial. J Med Internet Res. 2014;16(3):e69.
Bandura A. SOCIAL COGNITIVE THEORY: An Agentic Perspective. Annu Rev Psychol. 2001;52(1):1–26.
Perry CL, Williams CL, Veblen-Mortenson S, Toomey TL, Komro KA, Anstine PS, et al. Project Northland: Outcomes of a Communitywide Alcohol Use Prevention Program during Early Adolescence. Am J Public Health. 1996;86(7):956–65.
Komro KA, Perry CL, Williams CL, Stigler MH, Farbakhsh K, Veblen-Mortenson S. How did Project Northland reduce alcohol use among adolescents? Analysis of mediating variables. Health Educ Res. 2001;16(1):59–70.
Sniehotta FF, Schwarzer R, Scholz U, Schüz B. Action planning and coping planning for long-term lifestyle change: theory and assessment. Eur J Soc Psychol. 2005;35(4):565–76.
Van Osch L, Lechner L, Reubsaet A, Wigger S, De Vries H. Relapse prevention in a national smoking cessation contest: Effects of coping planning. Br J Health Psychol. 2008;13(3):525–35.
Miller WR, Rollnick S. Motivational Interviewing: Helping People Change, 3rd Edition (Applications of Motivational Interviewing). New York, NY: The Guilford Press; 2013.
Friederichs S, Bolman C, Oenema A, Guyaux J, Lechner L. Motivational Interviewing in a Web-Based Physical Activity Intervention With an Avatar: Randomized Controlled Trial. J Med Internet Res. 2014;16(2):e48.
Friederichs SAH, Oenema A, Bolman C, Guyaux J, van Keulen HM, Lechner L. Motivational interviewing in a web-based physical activity intervention: questions and reflections. Health Promotion International. 2013;28(4):1–13.
Bickmore T, Giorgino T. Health dialog systems for patients and consumers. J Biomed Inform. 2006;39(5):556–71.
Del Hoyo-Barbolla E, Kukafka R, Arredondo MT, Ortega M. A new perspective in the promotion of e-health. Stud Health Technol Inform. 2006;124:404.
Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic literature review of empirical evidence on computer games and serious games. Comput Educ. 2012;59(2):661–86.
DeSmet A, Van Ryckeghem D, Compernolle S, Baranowski T, Thompson D, Crombez G, et al. A meta-analysis of serious digital games for healthy lifestyle promotion. Prev Med. 2014;69:95–107.
Tüzün H, Yılmaz-Soylu M, Karakuş T, İnal Y, Kızılkaya G. The effects of computer games on primary school students’ achievement and motivation in geography learning. Comput Educ. 2009;52(1):68–77.
Papastergiou M. Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation. Comput Educ. 2009;52(1):1–12.
Locke EA, Latham GP. Goal setting theory. In: Jr HFON, Drillings M, editors. Motivation: Theory and research. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc; 1994. p. 13–29.
Crutzen R, Cyr D, Vries NK. The Role of User Control in Adherence to and Knowledge Gained from a Website: Randomized Comparison Between a Tunneled Version and a Freedom-of-Choice Version. J Med Internet Res. 2012;14(2):e45.
De Vet E, Brug J, De Nooijer J, Dijkstra A, De Vries NK. Determinants of forward stage transitions: a Delphi study. Health Educ Res. 2005;20(2):195–205.
Schneider F, Osch Lv, Vries Hd: Identifying Factors for Optimal Development of Health-Related Websites: A Delphi Study Among Experts and Potential Future Users. J Med Internet Res 2012, `14(1).
We thank all experts for their participation in this study and Roos Jansen for her support in analyzing the data. This work was supported by ZonMW, the Netherlands Organization for Health Research and Development [grant number: 200120006]. ZonMW had no role in the study design, collection, analysis or interpretation of the data, writing the manuscript, or the decision to submit the paper for publication.
Hein de Vries is scientific director of Vision2Health, a company that licenses evidence-based innovative computer-tailored health communication tools. All other authors declare that they have no competing interests.
AJ contributed in the development of the questions, the analysis and interpretation of all rounds and writing the manuscript. RC was involved in the development of the questions, analysis and interpretation of all rounds and revising the manuscript. LM contributed to the development of the questions and revising the manuscript. HdV contributed to the development of the questions and revising the manuscript. All authors approved the final manuscript.
Effective parenting practices/styles/actions. Table S2. Social environmental factors. Table S3. Motivational factors.