This article has Open Peer Review reports available.
Using a mobile health application to reduce alcohol consumption: a mixed-methods evaluation of the drinkaware track & calculate units application
© The Author(s). 2017
Received: 2 October 2016
Accepted: 28 April 2017
Published: 17 May 2017
Smartphone applications (“apps”) offer promise as tools to help people monitor and reduce their alcohol consumption. To date, few evaluations of alcohol reduction apps exist, with even fewer considering apps already available to the public. The aim of this study was to evaluate an existing publically available app, designed by Drinkaware, a UK-based alcohol awareness charity.
We adopted a mixed-methods design, analysing routinely collected app usage data to explore user characteristics and patterns of usage. Following this, in-depth interviews were conducted with a sub-sample of app users to examine perceptions of acceptability, usability and perceived effectiveness, as well as to provide recommendations on how to improve the app.
One hundred nineteen thousand seven hundred thirteen people downloaded and entered data into the app over a 13-month period. High attrition was observed after 1 week. Users who engaged with the app tended to be “high risk” drinkers and to report being motivated “to reduce drinking” at the point of first download. In those who consistently engaged with the app over time, self-reported alcohol consumption levels reduced, with most change occurring in the first week of usage. Our qualitative findings indicate satisfaction with the usability of the app, but mixed feedback was given regarding individual features. Users expressed conflicting views concerning the type of feedback and notifications that the app currently provides. A common preference was expressed for more personalised content.
The Drinkaware app is a useful tool to support behaviour change in individuals who are already motivated and committed to reducing their alcohol consumption. The Drinkaware app would benefit from greater personalisation and tailoring to promote longer term use. This evaluation provides insight into the usability and acceptability of various app features and contains a number of recommendations for improving user satisfaction and the potential effectiveness of apps designed to encourage reductions in alcohol consumption.
Excessive alcohol consumption is a leading cause of death worldwide . There is now clear evidence that screening and brief intervention to encourage reductions in alcohol consumption prove effective, at least in the short term [2, 3]. Increasingly, digital tools are being used to deliver such interventions, including via the internet or through smart phone applications (“apps”). These newer platforms for intervention delivery offer a number of benefits over and above more traditional approaches, such as face-to-face intervention in clinical settings. For example, not only does near ubiquitous smart phone ownership and internet access mean greater potential for wide intervention reach, but digital tools also allow for discrete delivery of intervention content, high intervention fidelity and can ensure that participant anonymity is retained . Apps in particular also permit recording of alcohol consumption and risk feedback in real time . These are particularly pertinent benefits in the context of alcohol reduction interventions given that accurately determining alcohol consumption levels based on retrospective recall is acknowledged to be challenging, especially given that high levels of intake can interfere with memory formation .
Despite widespread availability of alcohol reduction apps on both iTunes and Google Play stores, clear evidence of their effectiveness so far appears lacking . This likely owes to variability in app content and quality . For example, while the behaviour change techniques (BCTs; smallest observable and replicable components of an intervention that have potential to bring about behaviour change) effective in reducing alcohol consumption are known , and interventions based on theoretical underpinnings are generally shown to elicit greater effect sizes than those that are not , a recent investigation of publically available alcohol reduction apps has demonstrated that very few include all key BCTs . This represents a missed opportunity, not only to optimise the content of these offerings, but also to enhance user engagement given that inclusion of evidence-based content has previously been shown to be an important user preference .
The issue of user engagement is particularly relevant to health interventions delivered via app as drop-off in usage is common (e.g. up to 95% of apps are disengaged with after 1 month) . Such low retention rates are problematic from the perspective of intervention delivery as they preclude provision of longer term support, as well as disallowing the follow-up measurement necessary to determine effectiveness. We are, therefore, currently in a position of needing to understand more about exactly what users of alcohol reduction apps are seeking in terms of these offerings, and which features, designs or functionalities appear to promote longer term engagement.
To date, the few available evaluations that explore user preferences in relation to alcohol reduction apps tend to be limited in their generalisability, owing to the recruitment of either small or targeted sample groups (e.g. college age students) [12, 13], or rely on analyses of user ratings of existing publically available apps  (which may be biased towards more extreme views) . The findings of these evaluations seem to point towards preferences for personalised content, social networking capability and ease of functionality, especially in terms of entering drinking data [12, 15]. The ability to set own drinking goals is also highly valued . Similarly, in the broader mobile health (mhealth) literature beyond apps alone, comparable conclusions are also drawn; for example, text message based interventions have demonstrated effectiveness in both decreasing intentions to consume alcohol and in actual drinking behaviour [16, 17], with this type of intervention liked by users on the basis of ease and convenience . Other forms of digital intervention, including email, additionally seem to be accepted by recipients, primarily given their scope to tailor and personalise content , but may elicit lower levels of actual engagement compared to similar information and BCTs delivered via mobile phone .
The present study intends to add to this evidence base by presenting a mixed-methods evaluation of a publically available alcohol reduction app. The app in question was developed by the Drinkaware Trust and is listed on both iTunes and Google Play stores as the “Drinkaware: Track and Calculate Units” app (hereafter, the “the Drinkaware app”). The Drinkaware Trust is an independent UK-based charity which “aims to reduce alcohol-related harm by helping people make better choices about their drinking” . While this evaluation is app-specific, lessons learnt in this context may prove valuable to others wishing to develop effective alcohol reduction apps or to improve upon their existing offering.
Through exploratory analyses of routinely captured app usage data, this study aimed to examine typical patterns of app usage over time, and to understand how these patterns differ according to user characteristics, including pre-existing self-reported alcohol consumption levels and user demographics. In addition, we intended to explore app users’ views of the acceptability, usability and potential effectiveness of the Drinkaware app in qualitative interviews with a sub-sample of users. These interviews aimed to explain and augment the findings of the initial quantitative analysis, to further explore how and why users engaged with specific features contained in the app, and to offer recommendations to improve the design of this and similar products.
Our analysis of the Drinkaware app utilised a mixed-methods approach, specifically, a sequential explanatory design. This involved collection and analysis of quantitative data followed by a collection and analysis of qualitative data. This approach was chosen given that our quantitative data was uncontrolled, meaning that there are a number of limitations in reporting this data in isolation. The findings of the in-depth qualitative data are, therefore, used in the present study to further explain and aid interpretation of the quantitative findings.
Overview of the Drinkaware app
As outlined by Drinkaware, their app aims to enable users to 1) calculate the units and calories in drinks and to track alcohol consumption over time, 2) gain feedback on how drinking impacts health and to understand trends in drinking patterns, 3) set goals to reduce drinking that are relevant to their lifestyle and to receive supportive notifications when specific achievements are met, and 4) define geographic locations (e.g. a “local bar” or “supermarket”) where users may feel that additional support to regulate alcohol consumption is needed (referred to throughout as drinking “weak spots”). The app then sends users supportive messages when they reach these locations (e.g. “You are near one of your designated weak spots. Remember, drinking less has many feel-good benefits”) in an attempt to break alcohol consumption habits.
Behaviour Change Techniques included in the Drinkaware app
Behaviour Change Technique 
Recording the number of alcoholic drinks consumed per day
Self-monitoring of behaviour
Feedback of the number of alcoholic drinks consumed and associated risk level
Feedback on behaviour
Information about health consequences
Feedback of the cost and calorie equivalent of alcoholic drinks consumed
Feedback on outcomes of behaviour
Setting goals to reduce alcohol consumption (e.g. no drink day, drink within guidelines, drink one less)
Goal setting (behaviour)
Identification of drinking “weak spots” based on geolocation
Avoidance/ reducing exposure to cues for behaviour
Reductions in alcohol consumption are awarded via notifications
App usage data was collected and analysed over 13-months (between 10th August 2014 and 8th September 2015). Consent to use this data was acquired from all users through a mobile license agreement presented in the app upon first use. Information was collected on user demographic characteristics (age and gender), month of first access, initial motivations for downloading the app (five forced-choice response options including “to reduce drinking”, “to lose weight”, “to be healthier”, “just curious” and an option to provide no information), usage of primary features contained in the app (e.g. goal setting (Fig. 1e), “weak spots” (Fig. 1d) and alcohol consumption self-monitoring and feedback (Fig. 1b)). As Fig. 1a shows, the app interface allowed users to record (for each day of the week) the 1) type (i.e. beer/wine/spirit), 2) measure (e.g. pint, single) and 3) brand of alcohol consumed. Units referred to in the app are based on UK definitions (10 ml or 8 g of pure alcohol). Exploratory analyses of drinking behaviour presented in the results section are based on these self-reported measures only, with no additional alcohol use questionnaires deployed.
Data files were first anonymised and imported into excel where they were processed into a format suitable for analysis by statistical package SPSS. Anonymisation involved converting geo-location data from the app, specifying users ‘home’ location, into an Index of Multiple Deprivation (IMD) score (an area-level indicator of socio-economic status) via home postcode. This process was undertaken by the internal research team at Drinkaware who had no involvement in data analysis. No other personal sensitive data were extracted from the app. Only after anonymisation were the data were passed to the external researchers (SA, KM) for analysis purposes.
Given the vast amount of usage data available, and in expectation of a large drop-off in usage within the first month, a decision was made to analyse drinking related outcomes reported at pre-specified time points only. These were during ‘on-boarding’ into the app (i.e. information input into the app on typical weekly alcohol consumption at the point of registration), and then at weeks one, two, three, four and twelve thereafter.
Specification of user risk levela
Higher risk: consumption of more than twice the upper limit of the lower risk daily guidelines (6 units for women, 8 units for men), on four or more occasions within 1 week. Alternatively, consumption of more than 50 units within 1 week for a man, or more than 35 units within 1 week for a woman, regardless of the regularity of drinking
Increasing risk: consumption of more than the upper limit of the lower risk daily guidelines (3 units for women, 4 units for men), on four or more occasions during the week. Alternatively, consumption of more than 21 units within 1 week for a man, or more than 14 units within 1 week for a woman, regardless of the regularity of drinking.
Lower risk: consumption of more than the upper limit of the lower risk daily guidelines (3 units for women, 4 units for men), on no more than three occasions during the week. Alternatively, consumption of no more than 21 units within 1 week for a man, or no more than 14 within 1 week units for a woman, regardless of the regularity of drinking.
Existing app users who had previously expressed an interest in receiving further information from Drinkaware, and who had provided a contact email address, were invited to complete an anonymous feedback survey. The survey collected information on app usage patterns and user satisfaction (for more details, see Additional file 1). As part of this, users were asked to indicate whether they would be interested to take part in a telephone interview to give more detailed feedback on their experiences using the app.
Aged between 18 to 65 years
Resident in the UK
Able to consent to and complete a telephone interview in English.
Individuals who agreed to be interviewed were emailed an information sheet and consent form. Following receipt of a signed consent form via post or email, a member of the research team at Drinkaware (JL, HP) who had experience in conducting qualitative research then contacted respondents, by telephone, at a time of their convenience and completed a 30–40 min semi-structured interview. All interviews were audio-recorded, with interviewees verbally consenting to this process prior to the start of the interview. Interviewees were made aware that they were free to withdraw from interviews (and to withdraw disclosed information) at their discretion. Ethics approval for this study was granted by St. Mary’s University, Twickenham ethics committee.
The interview schedule was developed by external researchers KM and SA (see Appendix 1) to probe any questions or issues that arose during the quantitative analysis and to explore users’ experiences of the app, perceptions of usability, acceptability, effectiveness and to obtain suggestions for improvements to this product.
Framework analysis was used to analyse interview transcripts . This approach was chosen as is permits key themes to be explored across the whole data set, whilst also ensuring that the views of each research participant remain connected to others that they express during interviews, thereby taking into account the wider context surrounding a specific statement.
External researchers KM and SA coded anonymized interview transcripts (50% each) and created themes linking individual codes. To ensure rigour of the coding process, 25% of the interview transcripts were double coded. Decisions on final themes and their constituent codes was an iterative process: using the first five transcripts, an initial analytical framework was co-created by KM and SA, informed in part by the journey that users take through the app (i.e. from download, through the process of “on-boarding” to weekly usage etc.). This ‘analytical framework was refined following discussion with the Drinkaware researchers (HP and JL) and then used by KM and SA to index subsequent transcripts and was expanded as the data demanded. This led to the eventual creation of a detailed analytic framework (developed in excel) summarizing all themes to emerge across the entire data set in a coherent structure. The approach enabled comparison of data across cases as well as within individual cases. Throughout the analysis process, KM and SA kept separate notes detailing their interpretations and other potential themes to emerge.
User demographic characteristics, motivations to download the app and baseline drinking data
Considering motivations in relation to alcohol consumption at the point of downloading the app, users classified as “low risk” most frequently stated that they were “just curious” in the app (n = 14,930, 39.4%), those at “increasing risk” most commonly wanted “to lose weight” (n = 12,111, 31.3%), while “to reduce drinking” was the modal choice for those classified as “high risk” (n = 12,733, 49.4%) (Fig. 3b). Differences in motivation preference by baseline risk level were statistically significant (χ2 = 13,995.78, df = 8, p < 0.001).
Of all users who downloaded the app, 102,367 provided baseline drinking values for a complete week (86.9%). Of these, 22 outliers were excluded from subsequent analyses as these users reported consuming in excess of 300 units per week. In total, therefore, 102,345 users provided valid on-boarding baseline data for further analyses.
On average, male users reported consuming significantly more units of alcohol than female users (34 units per week versus 26 units per week (t = 48.34, df = 100,499, p < 0.001)). In context, the UK Chief Medical Officer advises consumption of no more than 14 units of alcohol per week to keep associated health risks low . A relatively large proportion of app users (6.8%) also reported “0” unit consumption during a “typical week” at on-boarding. This sample were found to be slightly younger (36.1 compared to 39.1 years) and contained a higher proportion of male users (49.0% compared to 42.8%) than those entering >0 units per week consumed at on-boarding (p < 0.001).
Regarding drinking patterns, users self-reported engaging in a drinking “binge”, on average, 1.9 times per week and remained alcohol-free on 3.5 days per week. Approximately one quarter of users were classified as “higher risk” based on self-reported “typical week” consumption at on-boarding (n = 10,442 users (24%) of male users, n = 14,794 (25.9%) of female users). This compares to only 5% of men and 3% of women in the UK population classified as at “higher risk” based on these definitions .
Pattern of app use
In those who completed on-boarding, a large drop-off in app usage occurred over the follow-up period. For example, after 1 week the number of app users (i.e. those recording alcohol consumption at least once during the week) dropped to n = 51,027 at week one (42.6% of those to download the app). This further reduced to n = 17,257 users (14.4%) by week four and n = 6025 users (5%) by week twelve.
Drinking behaviour over time
Drinking behaviour over time was examined in a subgroup of users who downloaded the app at least 12-weeks before the end of the follow-up period (e.g. those who proved valid drinking data at baseline and who entered the study period in sufficient time to allow for data collection at all follow-up time points; n = 73,538). Half of these users entered drinking data during week one (n = 37,065 users, 50.4%), with data entry diminishing thereafter. In total, 5045 (6.9%) users entered drinking data at all pre-specified follow-up time points.
We examined drinking patterns of users who consistently engaged with the Drinkaware app over time. These “engaged users” were defined as those entering data on their alcohol consumption, covering at least 6 days per week, for 12-weeks (n = 3401 users; 4.6% of the 73,538 users cited above). Compared to the 70,147 users who supplied 12-weeks of data but who did not engage consistently with the app, engaged users were older (42.6 compared to 38.7 years), contained a higher proportion of male to female users (57.4% compared to 43.9%) and reported higher levels of baseline alcohol consumption (31.6 compared to 29.7 units per week; p < 0.001 in all cases). Engaged users also most commonly reported “to reduce drinking” as their motivation for using the app (30.6%), compared to non-engaged users who more frequently cited that they were “just curious” (p < .001).
Exploring determinants of drinking behaviour over 4 weeks in engaged users
In order to explore in greater detail the demographic and motivational characteristics of engaged app users and patterns of usage associated with change in drinking behaviour during the first month of use (i.e. prior to significant drop off in use), multiple linear regression analyses were conducted on the sub-sample of engage users (n = 7785). Gender, age, season of download and baseline drinking were entered into the model initially, explaining 28.4% of the variance in unit consumption at 1 month follow-up (adjusted R2 = 0.284, p = <0.001). Male gender and self-reported alcohol consumption at the point of on-boarding were the only significant predictors in this step. Controlling for the abovementioned variables, user motivation was then entered into the model, explaining an additional 0.004% of the variance in unit consumption at 1 month (adjusted R2 = .288, F change = 10.54, p < 0.001). Finally, variables that represent usage of different app features were added to the model, explaining an additional 0.002% of the variance in unit consumption at this point (adjusted R2 = 0.289, F change = 9.12, p < 0.001). Use of goal setting was associated with lower unit consumption at 1 month follow up (1.9 fewer units per week, compared to those who did not use the goal setting feature, p < 0.001). Use of the “weak spot” feature did not however predict unit consumption at 1 month (p > 0.10).
Characteristics of Drinkaware app users who completed an interview
Age bracket (years)
Motivation for downloading the app
To reduce drinking
To reduce drinking
To be healthier
To reduce drinking
To lose weight
To be healthier
To be healthier
To reduce drinking
To be healthier
To reduce drinking
To reduce drinking
To reduce drinking
To reduce drinking
To reduce drinking
Getting started with the app
The on-boarding process
“But initially, I thought, this is quite a strange thing to do. And I was thinking back to my week, and thinking, oh God, I can’t remember. I was just throwing some random figures and drinks in there, that I thought were roughly accurate, but I wasn’t completely sure.” (C10)
This statement may help to explain the relatively high number of users reporting “0” units consumed at on-boarding. This value either accurately reflects consumption of no alcohol in those who may be seeking the app to assist in maintaining abstinence, or alternatively, represents users who do not wish to, or who are unable to, disclose their typical alcohol consumption levels. This raises questions as to the accuracy of the drinking data obtained during on-boarding as a valid baseline for comparison against further weeks within the quantitative evaluation.
Motivations for downloading the app
“It was health reasons, there wasn’t really any one, just one thing, it was a series of things….it’s says it’s no good for your health and then if you don’t have good health then it affects your appearance” (C9)
“I’m not sure if this is sort of a usual thing to download it for, but as well as sort of improving just my physical wellbeing, it was also an experiment for me, to see if it affected my mental wellbeing as well. So the whole reason I actually downloaded it, is because I thought, I wondered if certain difficulties I was experiencing at the time, were linked to excessive drinking. So it was actually more for the mental wellbeing, than the physical wellbeing” (C10)
Patterns of app usage
Self-monitoring alcohol consumption
“I do it daily most of the time. If I haven’t done it for a few days then I will go back and add it in retrospectively. I’m not in a particular routine with it I would say. I don’t know 80 per cent of the time I do it daily but then I might forget for a few days and just go back and do it”. (C8)
“I’d use it in the evening after…like bedtime sort of time”. (B2)
“Because, I’ve got a couple of other little apps that I look at on a daily, not all apps, but a little regime of four or five, you know, I check the weather and I look at my drink app, and various things like that, a little routine, so pretty much daily” (B1)
“When I say every day, I tend to fill them in the next day. If I’m out having a drink in the pub, I don’t sit there and each time put it into the app. Either when I come home, depending on the time like. If it was after say, midnight, then you’d have to go back a day so I would leave it ‘til the following morning” (C12)
“But often on the heavier nights, I have to guess because I don’t always do it when I’m out. So when I go to the pub or sometimes when I’m not having very many, I can easily remember so I’ll either do it at the time or after. But to be honest, when I’ve had more, I’m guessing roughly at how much I’ve had”. (C17)
“And I think…..even when I stopped all together, there was something still really…something to be said for still logging that, even though I was looking at a whole month and it would just be all green, and it would be like nothing, nothing, nothing, nothing, it was still quite satisfying to log that somewhere, if that makes sense….And then you’ve got somewhere to actually log that and you’ve got something that shows you the progress that you’ve made. And so that’s really, really helpful, and it’s sort of a little motivator for yourself (C10)
App feature usage
Goal setting feature experience
“To an extent I was already doing it, because…I don’t know why, I just was, I thought, right, okay, you can’t drink on a school night, obviously” (C2)
“… if I happened to have had no drinks on a day that I didn’t set as a no drink day and then it kind of racks up that way, I swear it [the app] gets confused sometimes. My goal is to not drink Monday to Thursday, but say I didn’t drink Sunday to Wednesday I think sometimes it goes, oh yes, congratulations, you’ve met your goal and I’m a bit unclear whether I have or not.” (C8)
“I felt that I probably couldn’t really pick a day when I’d drunk one less or I thought that would be really easy to do because I don’t drink a set amount on a certain day. I don’t like go to the pub on a certain day and think, right, I’ll have three pints today so this time I’ll just have two instead.” (C17)
“I only set one goal because I was very keen to kind of remain focused on one thing. I didn’t want to come and get lost in the app using it like a game. You know, I wanted to use it for one very specific thing.....I think I set it to drink probably within guidelines.” (C13)
“No, it didn’t appeal - probably because I thought if I put some goals in I’m probably not going to stick to it, which probably makes me sound a bit naughty.” (C7)
“Weak spot” feature experience
“It doesn’t really work for me…There’s not really any one place where I would go to consume alcohol. (C8)
“I think the weak spot is more not a physical place, the weak spot is because you suddenly think, oh god, I’ll go and have a drink.....Yes, your general circumstances are much more accurate, or much more relevant.” (B1)
“Yes. I did. I set up a few. I’m just wondering if it was meant to give me a notification, but I don’t recall ever getting one.” (C10)
Feedback and notifications from the Drinkaware app
“I like the numbers. I like to track stuff and have some figures behind it rather than just like, oh, I’ll go for a run today. I’ll be like, well, I’ll go for a run today but what’s my time from last time and how can I beat it? And I think that’s why this kind of app appeals to me. If I just put the drinks in and it just said you’re drinking too much but didn’t give any numbers behind it, I’d probably delete it within a few days”. (C17)
“I guess because you were filling out a diary it’s a bit more passive and the fact is that when you enter your figures on the Drinkaware app that first notification and ongoing notifications made you more aware of what you were doing and you are more inclined to do something about it”. (C13)
Type of feedback received
“The most surprising thing actually was the amount of calories……the calories was something I hadn’t expected, so I think when I’ve looked back on it, that’s probably been quite useful to see.... You know, if you think about having a massive pizza that might be a 1,000 calories, I’d think twice about eating a really big pizza. But having ten pints in any night, I would link it to be unhealthy and costly and feeling a bit bad the next day, but I wouldn’t have thought before that it was also calorific. (C17)
“…it just feels a bit too generic. Whereas with the MyFitnessPal it’s very much, you know, one can of Carling you know you’re going to consume, you know, 210 calories and that’s it for Carling. If you drink Carlsberg it could be 220 or, you know”. (C5)
“I think putting things in terms of money. So when it works out how much you’ve saved. I think that means a lot to people and that could really hit home as well.” (C9)
“The cost for me personally isn’t so much of a driver; it’s much more just about bringing down the level of drinking and about my health as opposed to what I’m spending”. (C11)
“I probably had it switched off, I tend to switch notifications off.” (B1)
“I think because they were just pinging… and I was just thinking, I don’t really want to read this right now. Obviously, and I don’t know whether they do but I guess most people check their phone when something pings in and you can be with your friends and actually maybe you wouldn’t want to be saying to your friends, I’ve just got a notification from Drinkaware”. (C14)
Perceived effectiveness of the Drinkaware app
Although the quantitative findings highlight a general reduction in self-reported alcohol consumption over time in engaged app users only, the uncontrolled nature of this evaluation means that no causality can be attributed (i.e. we are unsure if this reduction is a result of app usage prompting behaviour change, or simply that this change is recorded by the app over the course of the follow-up period). Hence, to know more about the potential impact of the app on drinking behaviour, perceptions of effectiveness were explored in qualitative interviews directly.
“Because, just like the name of it, Drinkaware, it made me aware of my alcohol intake….rather than sat there with a calculator, which is really boring, nobody wants to do it, all you have to do is write, large glass of wine, boom boom, you’ve got all your statistics there”. (C9)
“Well, I guess like all these things, I was hoping that it would change my behaviour. I think in reality it just made me more thoughtful about how much I’m drinking, which is probably a good thing”. (C17)
“So what I’ve learnt about my drinking habits is that probably I can do 5 days a week and not drink but on a Friday or a Saturday or a Saturday and a Sunday then I’ll drink more, so I might have half a bottle of wine on a Saturday and half a bottle of wine on a Sunday, but that almost puts me into, well it puts me into a red; it puts me in towards binge drinking”. (C14)
“I attribute a lot of it [reduction in drinking] to the app I guess because it’s a tool isn’t it? I mean obviously I wouldn’t have achieved it without the wish to do something as well personally, the drive to cut down my drinking, but I think that app is a great tool to help you achieve that”. (C11)
“It’s very, very easy to just slip back into how you were before or to just, yeah, go on, I’ll just have a another, and things like that. And so it’s nice to have something completely external from that, that sort of just keeps you on track. (C10)
“No, no [app has not changed drinking]….having found that I am not drinking to excess or to a bad problematic level, it’s been quite reassuring to know the amount I’m drinking is you might say fine, so I haven’t had to change anything”. (C16)
“To be honest, in a way, I think it’s probably made things worse, because while I might have been interested in pursuing a programme, like, you know, following that and trying to do something about it, it’s just forced me to go off the app, delete it, and forget all about the idea. Do you know what I mean? So, I might even be worse off after the app, rather than better off”. (B1)
Recommendations for improvements to the Drinkaware app
“Maybe a graph of the whole 28 days, might be a really good overview.....so you can see if it’s gone down, if it’s gone up across the month, where maybe at what point in the month it goes up and down, and then, yeah, that might be quite helpful”. (C10)
“It might give you a real big overview, say if you’ve been doing it for a year, it might give you a whole overview of how you’ve done in that year and how much you’ve saved, and how much…things like that. That might be quite helpful, to really look at it quite broadly.” (C10)
“Because it’s almost like you expect it to be kind of saved on a…because I’ve got the same email address and the same everything…I just literally swapped the device. So I was quite surprised to sort of lose all that. When you’ve done well that was quite gutting.” (C3)
“Yeah, not just generically saying you shouldn’t drink because in the long-term it’s bad for you. It’s like, well, tell me something I don’t know. But if it was personalised and said, this is the long-term effect. I mean, I would even go as far as… I know some people wouldn’t like it but I’d even go as far as saying, if you drink this amount, the risk to your health is like minus 3 years off your life.” (C17)
“…but the education side, how do you try and get out of the yellow [indicating ‘increasing risk’]. Like what ways can you actually…how can you change your behaviour rather than just see that you’re in the yellow.” (C5)
Summary of main findings
This mixed-methods evaluation of the Drinkaware app found that the app is disproportionately downloaded by higher risk drinkers (compared to UK averages), with high attrition observed after 1 week. Users who specifically intended to reduce their drinking, or those who self-reported consuming higher levels of alcohol prior to download, were more likely to engage with the app over time (i.e. to record drinks consistently over multiple weeks) and to utilise the main BCTs contained in the app (e.g. setting goals, self-monitoring drinking and setting up “weak spot” alerts).
Based on this uncontrolled quantitative analysis of routinely collected app usage data, we remain unable to conclude with any certainty that the alcohol consumption of engaged users actually reduced as a result of continued interaction with the app. Among users who completed interviews, however, and in line with existing qualitative findings in this area , a view was clearly expressed that the Drinkaware app had helped to raise awareness of the amount of alcohol consumed, facilitated self-monitoring of intake and encouraged more mindful drinking. A number of users explicitly stated that the app had facilitated changes in their drinking behaviour. We do, however, acknowledge that these findings are limited in their generalisability and do not necessarily represent the views and usage patterns of non-engaged users or of those not willing to participate in qualitative interviews.
Main findings in context
We intended to explore how and why users engaged with specific features contained in the Drinkaware app, and to offer recommendations to improve the design of this and similar products. Generally, for those who engaged with the goal setting feature, a preference was voiced for goal options that were simple and that built on existing patterns of behaviour, with the majority of users setting one rather than multiple goals. These findings stand in support of the results of existing evaluations of app based behaviour change interventions [24, 25], including those focussing on alcohol consumption specifically . Additionally, these findings also support text message based interventions in this area; for example, in a recent mobile phone based alcohol reduction intervention, greater decreases in intentions to consume alcohol were found in recipients who clearly set a goal to reduce drinking prior to the intervention . This finding, in combination with the results of our study, imply that any goal setting features incorporated into mhealth offerings like the Drinkaware app may be best viewed as tools to support users to achieve pre-existing health goals, rather than to prompt formation of new goals.
Related to this, our study also indicates that more information on the link between patterns of consumption and potential health harms may need to be included in the Drinkaware app to ensure that “no drink day” goals are not being used to compensate for, or to permit, subsequent binge drinking. Indeed, as a recent randomized controlled trial of a different alcohol reduction app suggests , care needs to be taken that content is used as intended, and not as a means to accurately quantify reductions in drinking at one point in time in order to give license to engage in different, yet equally damaging, drinking patterns at a later date .
In many cases, users indicated a desire for the app to include options to specify personal goals, while others wanted goal setting to automatically adapt to their progress. This finding fits with a growing body of literature to show that tailoring content to individual needs, and the ability to “learn” from data previously entered, are particularly sought-after features, and may even enhance effectiveness [12, 28]. As is demonstrated in both our study and within the existing literature, alcohol reduction apps, and indeed other forms of digital intervention that include goal setting features, may benefit from functionality that allows users to change their goals over time or in different contexts, so ensuring that the app takes into account deviations from an originally set goal if it ultimately result in the same or similar behavioural outcome .
Our analysis of usage of the “weak spot” geolocation feature does, however, emphasize the fact that any such tailoring needs to be thoughtfully designed and implemented; for example, both the quantitative and qualitative findings presented here suggest that highlighting individual physical environment-specific triggers to drinking is not necessarily seen as useful, and was generally poorly understood by users. Interestingly, this finding reflects that of Crane et al. (2015), who also found that alcohol reduction apps that contain features to enable users to identify and restructure environmental triggers to consumption were associated with lower user ratings . Other existing evaluations of apps that contain geolocation features to prompt change in the context of alternative health behaviours (e.g. smoking cessation, reduction in time spent sedentary) do, however, report somewhat more positive responses to geolocation based features, although also document lower than optimal levels of engagement [29, 30].
As with many health related apps, drop-off in usage of the Drinkaware app was large during the first month following download . From the routine data analysed, we were unable to follow-up the drinking behaviour of those who downloaded but subsequently disengaged with the app. This represents a missed opportunity given evidence that screening and brief interventions to reduce alcohol related harms (akin to the on-boarding process into the Drinkaware app) prove effective in reducing alcohol consumption in other contexts . Interestingly, interviewees in our study not only discussed the value of longer term self-monitoring to encourage behaviour change, but frequently mentioned that the app had raised awareness, both of the actual amount of alcohol consumed and of the associated health risks. This finding is supported by an existing qualitative study of a similar alcohol reduction app , and would suggest that future research may benefit from exploring whether brief, single interventions to reduce alcohol consumption, delivered via app, prompt behaviour change despite subsequent disengagement.
In users who engaged with the app for more than 1 month, the general pattern was to enter drinking data in one session, usually at the start or end of a day. This is a particularly useful finding for those who may be attempting to develop new app features, as it would suggest that these must not rely on real-time data entry (e.g. developing alert systems when a certain consumption limit is reached). Moreover, our qualitative interviews, like others in this area , found that discretion in terms of data entry and feedback was particularly important to users. For example, reflecting wider literature in this area , interviewees in our study voiced clear concerns about what other people would think if they saw the Drinkaware app icon on their phone or tablet, suggesting that private feedback is desirable.
We note that our quantitative analysis did show a reduction in alcohol consumption between on-boarding and subsequent weeks in users who regularly entered drinking data into the app, although this sample is very small compared to those who originally downloaded the app (i.e. <5% of all users). This finding is also interpreted cautiously given that “typical week” consumption estimates were based on retrospective recall (i.e. at the point of on-boarding), which differs from the diary entry approach adopted thereafter. As existing research shows, individuals’ assessments of the amount that they habitually drink tend to underestimate intake . Hence, our findings may suggest that a maximum change in drinking is observed after one week of app use and maintained thereafter, or that the reduction in consumption seen during week one may be an artefact of the measurement tools used and that the app needs to incorporate more features that are effective in prompting reductions in alcohol intake over time. The next section outlines a number of recommendations to this effect.
When asked for suggestions for improvements to the app, interviewees commonly mentioned the feedback that it provided. Specifically, comments centred upon a wish for the app to incorporate a visually appealing means to overview trends in consumption over time. For example, frustration was often reported if the app had “lost” data previously input (e.g. as a result of an update), suggesting that more work may need to be conducted to optimise how information is stored, shared and presented between apps and other online dashboards or interfaces .
Regarding presenting longer term trends in drinking, many interviewees mentioned that this information would give them a more meaningful way to judge progress, and could be augmented by linking drinking patterns to relevant contextual information to better understand influences on consumption. To achieve this linkage, the Drinkaware app and others like it may benefit from incorporating ecological momentary assessment techniques (EMA) (e.g. ‘real-time’ data capture in natural contexts) . Indeed, a number of studies are already underway to explore the utility of this technique for understanding contextual triggers to alcohol consumption [38, 39]. We do, however, draw attention to the fact that users frequently appeared to switch off notifications from the app, with a number of interviewees suggesting that they found these intrusive - a finding that is also supported within the wider literature on this topic . As such, when developing app features based on EMA, the added benefit of collecting real-time data on determinants of drinking will need to be carefully weighed against data collection burden [40, 41]. In all cases, we recommend that those developing and evaluating apps in this area take into account “APEASE” criteria , which highlight the importance of developing digital health interventions that not only contain features that are effective, but also acceptable, practical, affordable, safe and equitable.
An additional recommendation put forward by a number of interviewees was for the app to provide more information on the effect of alcohol on well-being or daily functioning, and to give a broader holistic picture of how drinking can impact individual mood patterns. While favoured by users, it remains to be determined whether provision of this type of information will be effective in prompting actual behaviour change. That this preference was expressed would, however, suggest there are likely to be significant numbers who download alcohol reduction apps not necessarily to reduce drinking directly, but who instead wish to manage a number of non-health related outcomes. Expanding initial “motivation” options present in the Drinkaware app to include improving mental health or well-being may be a good starting point to understand this need more clearly, as would allowing users to specify multiple, possibly interacting motives for downloading this or similar products (e.g., to reduce drinking in order to lose weight and so feel better).
One further, consistent theme to emerge across interviews was that the current version of the Drinkaware app offers little in the way of maintenance support once drinking had successfully reduced. This is a clear limitation given that continued monitoring of consumption (including zero consumption) is documented within the literature as an effective relapse prevention technique . Future alcohol reduction apps should, therefore, ensure that they incorporate BCTS that not only assist current drinkers to reduce consumption levels, but also help abstainers to maintain this over time.
Strengths and limitations
This analysis of routinely captured app usage data enabled a large sample to be analysed, with our findings complemented by the qualitative study that permitted clarification of data abnormalities and provided further information on app usage patterns. That we studied a sample of existing users, rather than individuals who were recruited into a trial, enhances the ecological validity of our results. This study presents novel insights into how real app users perceived this product, interacted with the features that it contains and explored what is sought by those who actually download alcohol reduction apps.
We do, however, acknowledge a number of limitations to this study: firstly, a reliance on routinely captured app usage data meant that we were unable to draw firm conclusions regarding the effect of app usage on drinking behaviour. This is both due to the observational nature of the quantitative analysis and to the fact that a separate, validated measure of alcohol consumption was not incorporated (i.e. the act of recording drinks served as both a BCT and also an outcome measure in our quantitative evaluation). As such, we recommend that future studies in this area include alternative, validated measures of alcohol consumption, obtained independently of the app itself (e.g., an AUDIT tool could be sent to users at various intervals; see Garnett et al., 2016 for an example of a rigorous evaluation design adopting a separate ‘outcome’ measure) . This may also permit follow-up of drinking behaviour in users who disengage with the app.
Related to this point, we also recognise that conclusions based on analyses of routinely collected data are likely to reflect the structure of the app and app promotion activities run by Drinkaware over the course of the designated follow-up period. For example, we note that the “no drink day” goal was the first option provided to users on a vertical list (see Fig. 1), suggesting that order effects may, to some extent, account for the frequency with which it was chosen. In order to overcome this limitation, it may be beneficial to employ study designs that counterbalance or randomise the order in which app content is presented to users, and to analyse a sample of data collected during a period in which no additional promotional activities are being run.
For the qualitative study, we were unable to recruit more than two interviewees who had since disengaged with the app. Our interviews may therefore underrepresent the views of those who didn’t find the app acceptable or effective. Indeed, this investigation is limited to analysing data provided by those who continued to engage with the app over time, meaning that the results can only be generalised to longer terms users of alcohol reduction apps, and not to those who discontinue usage or who never download these products in the first place.
The Drinkaware app appears to be a useful tool for raising awareness about drinking and potentially reducing consumption in individuals already committed to making such changes. The evaluation sheds light on issues surrounding usability and acceptability of the various features of the app, and provides recommendations that generally focus on user requirements for greater personalisation and tailoring. Future evaluations should seek to adopt more rigorous designs, including independent outcome measures to draw stronger conclusions about the effectiveness of apps for reducing alcohol consumption.
The Drinkaware independent Medical Advisory Panel offered guidance on the measures to be used as part of the data collection tool and the Drinkaware Research and Insights Manager Andrew Russell assisted in collating data analysed for this study.
Drinkaware is primarily funded by voluntary and unrestricted donations from UK alcohol producers, retailers and supermarkets, based on an agreement between the UK government and the alcohol industry.
Availability of data and materials
The data that support the findings of this study are available from Drinkaware but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Drinkaware.
Drinkaware provided routinely collected app usage data. SA and KM designed the study, analysed the data and drafted the manuscript. JL and HP offered feedback to inform the study design, recruited interview participants and conducted interviews. All authors read and approved the final manuscript.
SA and KM received a consultancy fee for their ongoing work as independent evaluators of the Drinkaware app. HP and JL supported the study and conducted the interviews as employees at Drinkaware and members of the Research and Impact team.
Consent for publication
Ethics approval and consent to participate
Consent to use the quantitative data was acquired from all users through a mobile license agreement presented in the app upon first use. Ethics approval for the qualitative component was provided by St. Mary’s University Twickenham ethics committee. Written and audio-recorded consent were provided by all participants prior to their interview.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- World Health Organisation. Global status report on alcohol and health. (2014). http://www.who.int/substance_abuse/publications/global_alcohol_report/en/. (Accessed 12 Sept 2016).
- Kaner E, Bland M, Cassidy P, Coulton S, Dale V, Deluca P, Gilvarry E, Godfrey C, Heather N, Myles J, et al. Effectiveness of screening and brief alcohol intervention in primary care (SIPS trial): pragmatic cluster randomised controlled trial. BMJ. 2013;346:e8501.View ArticlePubMedPubMed CentralGoogle Scholar
- Anderson P, Bendtsen P, Spak F, Reynolds J. Improving the delivery of brief interventions for heavy drinking in primary health care: outcome results of the Optimizing Delivery of Health Care Intervention (ODHIN) five-country cluster randomized factorial trial. Addiction 2016; doi: 10.1111/add.13476.
- Capon H, Hall W, Fry C, Carter A. Realising the technological promise of smartphones in addiction research and treatment: an ethical review. Int J Drug Policy. 2016;36:47–57.View ArticlePubMedGoogle Scholar
- Cohn AM, Hunter-Reel D, Hagman BT, Mitchell J. Promoting behavior change from alcohol use through mobile technology: the future of ecological momentary assessment. Alcohol Clin Exp Res. 2011;35(12):2209–15.View ArticlePubMedPubMed CentralGoogle Scholar
- White AM. What happened? Alcohol, memory blackouts, and the brain. Alcohol Res Health. 2003;27(2):186–96.PubMedGoogle Scholar
- Payne EH, Lister C, West HJ, Bernhardt MJ. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth. 2015;3(1):e20.View ArticlePubMedPubMed CentralGoogle Scholar
- Garnett C, Crane D. Identification of behavior change techniques and engagement strategies to design a Smartphone app to reduce alcohol consumption using a formal consensus method. JMIR Mhealth Uhealth. 2015;3(2):e73.View ArticlePubMedPubMed CentralGoogle Scholar
- Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, Eccles MP, Cane J, Wood CE. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Annals Behav Med. 2013;46(1):81–95.10.View ArticleGoogle Scholar
- Vandelanotte C, Muller AM, Short CE, Hingle M, Nathan N, Williams SL, Lopez ML, Parekh S, Maher CA. Past, present, and future of eHealth and mHealth research to improve physical activity and dietary behaviors. J Nut Ed Behavr. 2016;48(3):219–228.e211.View ArticleGoogle Scholar
- Crane D, Garnett C. Behavior change techniques in popular alcohol reduction apps: content analysis. J Med Internet Res. 2015;17(5):e118.View ArticlePubMedPubMed CentralGoogle Scholar
- Milward J, Khadjesari Z. User preferences for content, features, and style for an app to reduce harmful drinking in young adults: analysis of user feedback in app stores and focus group interviews. JMIR Mhealth Uhealth. 2016;4(2):e47.View ArticlePubMedPubMed CentralGoogle Scholar
- Gajecki M, Berman AH, Sinadinovic K, Rosendahl I, Andersson C. Mobile phone brief intervention applications for risky alcohol use among university students: a randomized controlled study. Addict Sci Clin Practe. 2014;9(1):1–12.View ArticleGoogle Scholar
- Koh NS, Hu N, Clemons EK. Do online reviews reflect a product’s true perceived quality? An investigation of online movie reviews across cultures. Electron Commer Res Appl. 2010;9(5):374–85.View ArticleGoogle Scholar
- Giroux D, Bacon S, King DK, Dulin P, Gonzalez V. Examining perceptions of a smartphone-based intervention system for alcohol use disorders. Telemedi J Ehealth. 2014;20(10):923–9.View ArticleGoogle Scholar
- O'Rourke L, Humphris G, Baldacchino A. Electronic communication based interventions for hazardous young drinkers: a systematic review. Neurosci biobehavioral rev. 2016;68:880–90.View ArticleGoogle Scholar
- Fowler LA, Holt SL, Joshi D. Mobile technology-based interventions for adult users of alcohol: a systematic review of the literature. Addict Behav. 2016;62:25–34.View ArticlePubMedGoogle Scholar
- Suffoletto B, Kristan J. Optimizing a text message intervention to reduce heavy drinking in young adults: focus group findings. JMIR Mhealth Uhealth. 2016;4(2):e73.View ArticlePubMedPubMed CentralGoogle Scholar
- Bendtsen M, Bendtsen P. Feasibility and user perception of a fully automated push-based multiple-session alcohol intervention for university students: randomized controlled trial. JMIR mHealth and uHealth. 2014;2(2):e30.View ArticlePubMedPubMed CentralGoogle Scholar
- Ivankova NV, Creswell JW, Stick SL. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods. 2006;18(1):3–20.View ArticleGoogle Scholar
- Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med ResMethodol. 2013;13(1):1–8.View ArticleGoogle Scholar
- Department of Health. UK Chief Medical Officers’ Alcohol Guidelines Review: Summary of the proposed new guidelines. (2016). https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/489795/summary.pdf. Accessed 9 Sept 2016.
- NHS Digital. Health Survey for England - 2013. Trend tables. (2014). http://content.digital.nhs.uk/catalogue/PUB16077. Accessed 1 Sept 2016.
- Tang J, Abraham C, Stamp E, Greaves C. How can weight-loss app designers' best engage and support users? A qualitative investigation. Brit J Health Psych. 2015;20(1):151–71.View ArticleGoogle Scholar
- Mendiola MF, Kalnicki M. Valuable features in mobile health apps for patients and consumers: content analysis of apps and user ratings. JMIR Mhealth Uhealth. 2015;3(2):e40.View ArticlePubMedPubMed CentralGoogle Scholar
- Suffoletto B, Chung T. Patterns of change in weekend drinking cognitions among non-treatment-seeking young adults during exposure to a 12-week text message intervention. J Stud Alcohol Drugs. 2016;77(6):914–23.View ArticlePubMedGoogle Scholar
- Prinsen S, Evers C, de Ridder D. Oops I did it again: examining self-licensing effects in a subsequent self-regulation dilemma. Appl psychol health and well-being. 2016;8(1):104–26.View ArticlePubMedGoogle Scholar
- Head KJ, Noar SM, Iannarino NT, Grant HN. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med. 2013;97:41–8.View ArticlePubMedGoogle Scholar
- Naughton F, Hopewell S. A context-sensing mobile phone app (Q sense) for smoking cessation: a mixed-methods study. JMIR Mhealth Uhealth. 2016;4(3):e106.View ArticlePubMedPubMed CentralGoogle Scholar
- Thomas JG, Bond DS. Behavioral response to a just-in-time adaptive intervention (JITAI) to reduce sedentary behavior in obese adults: implications for JITAI optimization. Health Psych. 2015;34s:1261–7.View ArticleGoogle Scholar
- Michie S, Atkins L, West R. The behaviour change wheel: a guide to designing interventions. Great Britain: Silverback Publishing; 2015.Google Scholar
- Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psych Bull. 2007;133(4):673–93.View ArticleGoogle Scholar
- Kaner EF, Beyer F, Dickinson HO, Pienaar E, Campbell F, Schlesinger C, Heather N, Saunders J, Burnand B. Effectiveness of brief alcohol interventions in primary care populations. Cochrane Database Syst Rev. 2007;18(2):Cd004148.Google Scholar
- McKay FH, Cheng C, Wright A, Shill J, Stephens H, Uccellini M. Evaluating mobile phone applications for health behaviour change: A systematic review. J telemed telecare 2016; doi:10.1177/1357633X16673538
- Ekholm O, Strandberg-Larsen K, Gronbaek M. Influence of the recall period on a beverage-specific weekly drinking measure for alcohol intake. Eur J Clin Nut. 2011;65(4):520–5.View ArticleGoogle Scholar
- Berman AH, Gajecki M, Sinadinovic K, Andersson C. Mobile interventions targeting risky drinking among university students: a review. Curr Addic Rep. 2016;3:166–74.View ArticleGoogle Scholar
- Lagoa CM, Bekiroglu K, Lanza ST, Murphy SA. Designing adaptive intensive interventions using methods from engineering. J Consul Clinl Psych. 2014;82(5):868–78.View ArticleGoogle Scholar
- Cohn A, Brandon T, Armeli S, Ehlke S, Bowers M. Real-time patterns of smoking and alcohol use: an observational study protocol of risky-drinking smokers. BMJ Open. 2015;5(1):e007046.View ArticlePubMedPubMed CentralGoogle Scholar
- Trela CJ, Piasecki TM, Bartholow BD, Heath AC, Sher KJ. The natural expression of individual differences in self-reported level of response to alcohol during ecologically assessed drinking episodes. Psychopharm. 2016;233(11):2185–95.View ArticleGoogle Scholar
- Matthews M, Abdullah S, Murnane E, Voida S, Choudhury T, Gay G, Frank E. Development and Evaluation of a Smartphone-Based Measure of Social Rhythms for Bipolar Disorder. Assessment 2016; doi: 10.1177/1073191116656794.
- Asselbergs J, Ruwaard J. Mobile phone-based unobtrusive ecological momentary assessment of day-to-day mood: an explorative study. J Med Internet Res. 2016;18(3):e72.View ArticlePubMedPubMed CentralGoogle Scholar
- Klingemann H. Coping and maintenance strategies of spontaneous remitters from problem use of alcohol and heroin in Switzerland. Int J Addict. 1992;27(12):1359–88.View ArticlePubMedGoogle Scholar
- Garnett C, Crane D, Michie S, West R, Brown J. Evaluating the effectiveness of a smartphone app to reduce excessive alcohol consumption: protocol for a factorial randomised control trial. BMC Pub Health. 2016;16(1):1–10.View ArticleGoogle Scholar
- Health and Social Care Information Centre: Statistics on Alcohol England. (2015). http://content.digital.nhs.uk/catalogue/pub17712/alc-eng-2015-rep.pdf. Accessed 15 Nov 2016.