Validating estimates of problematic drug use in England
© Frisher et al; licensee BioMed Central Ltd. 2007
Received: 04 October 2006
Accepted: 10 October 2007
Published: 10 October 2007
UK Government expenditure on combatting drug abuse is based on estimates of illicit drug users, yet the validity of these estimates is unknown. This study aims to assess the face validity of problematic drug use (PDU) and injecting drug use (IDU) estimates for all English Drug Action Teams (DATs) in 2001. The estimates were derived from a statistical model using the Multiple Indicator Method (MIM).
Questionnaire study, in which the 149 English Drug Action Teams were asked to evaluate the MIM estimates for their DAT.
The response rate was 60% and there were no indications of selection bias. Of responding DATs, 64% thought the PDU estimates were about right or did not dispute them, while 27% had estimates that were too low and 9% were too high. The figures for the IDU estimates were 52% (about right), 44% (too low) and 3% (too high).
This is the first UK study to determine the validity estimates of problematic and injecting drug misuse. The results of this paper highlight the need to consider criterion and face validity when evaluating estimates of the number of drug users.
The UK Government's direct annual expenditure on combating illicit drug use is £1.48 billion  (2005/2006). By 2007/08, expenditure on treatment will increase to £478 million from £253 million in 2004 . By 2008, the Government's target is to have the capacity to treat 200,000 drug users . However, direct methods of ascertaining the number of problematic drug users in the country are unlikely to yield accurate figures. This is because of multiple response bias affecting the likelihood of problem drug users a) living in a household that would be included in a sample survey, b) agreeing to participate in a sample survey and c) reporting recent problem drug use [4, 5]. One alternative method which has been recommended for country wide prevalence estimates is the Multiple Indicator Method (MIM) . This method combines information on prevalence that is available only in a few areas (the 'anchor points') and drug indicators (e.g. number of crimes, seizures) that are available in all areas . We previously reported a MIM analysis for all Drug Action Team (DAT) areas in England for 2001 . The MIM involved a three stage process: a) factor analysis of the drug indicators, b) regression analysis linking factor scores to "known" prevalence estimates and c) imputation of estimates in all other DATs. The "known" estimates are obtained by a technique known as capture-recapture. The capture-recapture method (CRM) involves 'capturing' a random sample who are then 'marked' and returned to their habitat . Subsequently, a second random sample are 'recaptured' and the number of marked animals from the first sample are observed. The ratio of marked animals to the recaptured sample size is assumed to be the same as the ratio of the first captured sample to the total population. Thus, if a 'capture' sample of 200 animals are marked and released and a 'recapture' sample of 100 contains 10 animals which are marked, the estimate for the total population would be 2,000 (i.e. 10:100 = 200:2,000). In drug use studies the samples are lists of drug users whose identifiers are recorded by agencies such as treatment centres or the police. More sophisticated versions involving multiple samples have also been developed in order to overcome the stringent assumptions which are required when only two-samples are used .
In the MIM study, the regression model explained 32% of the variance for the problematic anchor points and 70% for the injecting anchor points. The median estimates were 287,670 problematic drug users and 93,185 injecting drug users in England in 2001. The definition of problematic drug use was 'current use of illicit opiates, crack-cocaine or benzodiazepines'. The definition of injecting drug use was 'current use of any illicit drug where injecting was the method of drug administration'. These definitions are based on the data collection procedures of those organizations whose data is used in capture-recapture studies. These organizations consist of drug treatment agencies, the police, HIV-test registers and needle and syringe exchanges. As most of these agencies categorise drug users simply as being current users at time of contact, it is not possible to give a more precise timeframe.
Validity of drug user estimates
As the MIM estimates are based on a statistical model it is important to determine the validity of the results. In other words, does the model correspond to how many problematic and injecting drug users there actually are in each DAT? The issue may be addressed in two ways. Firstly, by comparing the MIM estimates to other prevalence estimation techniques. This is known as criterion-related validity, which may be defined as "comparing values with one or more external criteria, known or believed to measure the attribute under study" . The second method is to ask experts in each of the DATs to evaluate the MIM estimates. This is referred to as face validity, which is the degree to which something appears to measure what it is supposed to measure. While in statistical terms, this is a weaker form of validity, in practical terms it is important to practitioners in the field. In 1993, Reuter argued that prevalence estimates in the United States were not widely used because of policy-makers' perception of their limited credibility . However, the situation has changed over the last decade with the introduction of more scientific approaches to the simulation of hidden population . In the UK prevalence estimates are increasingly used in the policy process, for example in relation to the introduction and placement of syringe exchange programmes  and modelling HIV prevalence . The Government's Updated Drug Strategy 2002 explicitly aims to increase the proportion of the "estimated 250,000 problematic drug users" who receive treatment by 2008 .
With regard to criterion-related validity, we have previously estimated the number of drug users in England, Wales and Scotland in 1996 using a variety of methods corresponding to different forms of drug use . Between 1996 and 2001, the MIM estimate for England increased by 27.6% while a range of drug indicators increased by an average of 18%. This suggests that the 2001 MIM estimates have acceptable criterion-related validity.
To the best of our knowledge there has only been one previous study into face validity of drug user estimates. Friedman et al  estimated the number of injecting drug users for 96 metropolitan areas in the USA in 1996 using a variety of multiplier methods similar to those described above. They assessed validity by asking 143 experts to comment on studies or estimates available for their areas. Of the valid responses, 81% endorsed the estimates, 8% though the estimates were too low and 11% thought they were too high. These results suggest that the indicator methods used in the American study had a high degree of construct and/or face validity.
The aim of the current study is to determine the face validity of the 2001 MIM estimates of problematic drug use (PDU) and injecting drug use (IDU) for the 149 DATS in England.
All DATs in England were sent a letter from the Home Office informing them of this study and that all DATs were being asked to respond to a questionnaire. Each DAT was provided with information specific to their area and told that the aim of the exercise was to gain a broad understanding of how DATs view the PDU and IDU estimates. The questionnaire was kept short, so as to ensure as high a response rate as possible.
The questionnaire, in Adobe PDF format was emailed to each DAT by the Home Office via the Regional Managers on 21st June 2004. It was requested that the questionnaire be completed by the person (or persons) best placed to assess the number of drug users in the DAT. The return date for the questionnaire was 9th July 2004; non-responders were followed up at the end of July.
90 of the 149 DATs returned the form by 29th July 2004, giving a response rate of 60.4%. Responders and non-responders did not differ significantly in terms of their mean DAT populations (320,674 vs. 313,707, t = 0.17, p = 0.86). Responders did not significantly differ from non-responders in terms of their estimated PDU rate (PDU 665 vs. 621 per 100,000 population, x2 = 1.51, p = 0.21).
Evaluation of PDU and IDU estimates
The first question asked was "In your view, is this estimated number of problematic users for your DAT in 2001 A) about right, B) too low, C) too high, D) don't know, E) other (specify) [Note: we did not provide definitions of "about right", "too low" or "too high". For this and subsequent questions, the percentages refer to responding DATs.]
Drug Action Teams' evaluation of Problematic Drug User (PDU) estimates
Percent excluding "don't know"
Mean PDU Estimate per 100,000 population
1.00 about right
2.00 too low
3.00 too high
4.00 don't know
Drug Action Teams' evaluation of Injecting Drug User (IDU) estimates
Percent excluding "don't know"
Mean IDU Estimate per 100,000 population
1.00 about right
2.00 too low
3.00 too high
4.00 don't know
Further analysis of Problematic Drug Use (PDU)
Of the 60 DATs who said their PDU estimates were either too low, too high or don't know, 33 (55%) provided some free text information to clarify this assessment.
Information provided includes:
• too low (n = 18): other research; data provided by the National Treatment Agency (NTA); own experience
• too high (N = 7): errors in model figures; too high relative to numbers on treatment; estimates derived from the British Crime Survey (BCS)
• don't know/other (N = 8): can't comment; no data; not in post in 2001
The basis of estimates included: research study (e.g. capture-recapture, economic modelling); need assessments; extrapolation from number in treatment, number of arrests; extrapolation from the British Crime Survey.
Further analysis of Injecting Drug Use (IDU)
Of the 69 DATs who said their IDU estimates were either too low, too high or don't know, 37 (53%) provided some free text information to clarify this assessment.
Information provided included:
• too low (n = 27): figures suggest service capacity rather than actual figure; perception, based on proportion of problematic users who are injectors; model figures wrong
• too high (N = 2): local research; perception that smoking more common
• don't know/other (N = 8): can't comment; no data; not in post in 2001
Comparison of DATs own PDU estimates and the MIM PDU estimates
6 of the 42 DATs (14%) who thought their MIM estimate were either 'too low' or 'too high' compared to the MIM estimate, actually had estimates within 10% of the MIM estimate.
Comparison of DATs own IDU estimates and the MIM IDU estimates
As with PDU, where DATs had their own IDU estimates (N = 38) they tended to be higher than the MIM estimates. Two DATs who thought that their MIM IDU estimate were either 'too high' or 'too low', provided an estimate of their own that was within 10% of the MIM estimate.
Evaluation of drug indicators
There were many reasons why DATs' own estimates/figures diverged from the MIM indicators. For example some DATs thought that indicator S31-refers to all people currently in treatment. This would account for the fact that for this indicator, 35 DATs thought the MIM figure was too low. In fact the MIM indicator refers to people entering treatment [Number of individuals aged 18+ with addresses in the DAT area recorded by RDMD as having entered treatment during 2000/2001].
In contrast, the majority of DATs thought that the MIM possession and supply figures were too high. One DAT suggested that their figure was a concatenation of 3 years figures. For five of the indicators, more DATs (who expressed an opinion) though that the indicators were "about right" as opposed too low/high.
Some DATs thought the methadone prescription figures were too low. These DATs had access to data on methadone prescriptions, but our data refer only to prescriptions issued in primary care. Where prescriptions are issued in secondary care, they would not be included in our data source.
These ambiguities could possibly have been reduced had the questionnaire provided more detail. However, there was concern that providing this information would have inhibited some DATs from responding. DATs were told that the further information on the indicators was contained in the full report which could be accessed online .
Further comments on the estimates and indicators
44 DATs provided comments on the form. These covered a wide range of themes. Some DATs were anxious that estimates were wrong because of wrong input data. As explained above, this might have been due to misinterpretation of which data were actually used in the estimation process. Some DATs were concern that definition of PDU did not cover PDU as perceived in DAT area. This is a fair comment, but needs to be seen in the context of providing uniform estimates across all English DATs. Some DATs felt that it was difficult to comment on the situation in 2001 or that the relevancy of 2001 estimates was questionable. Finally some DATs were appreciative of the estimates and wished to be more directly involved in future estimation work.
Main findings of this study
The MIM pilot study provided all DATs with estimates of the prevalence of problem drug use and injecting drug use in their area. Among responding DATs (N = 90, response rate = 60%), 33% thought the PDU estimates were "about right", while 24% were unable to make an assessment. A further 6 (7%) DATs who rated the MIM estimates as being too low/high provided their own estimate which was actually within 10% of the MIM estimate. By combining the three categories: "about right", "don't knows" and "within 10%", 64% thought the PDU estimates were about right or did not dispute them [The 'didn't know' DATs have been included on the basis that they had no local knowledge with which to make a comparison and therefore the MIM estimate is an improvement on no knowledge.]
Among responding DATs (n = 90, response rate = 60%), 23% thought the IDU estimates were "about right", while 27% were unable to make an assessment. A further 2 (2%) DATs who rated the MIM estimates as being too low/high provided their own estimate which was actually within 10% of the MIM estimate. Thus, by combining "about right", "don't knows" and "within 10%", 52% of DATs have potentially acceptable IDU estimates.
Of the 31 DATs who had their own PDU estimates, 24 of these were higher than the MIM estimate while 7 were lower. Of the 20 DATs who had their own IDU estimates, 17 of these were higher than the MIM estimate while 3 were lower. As noted above these estimates may not be comparable with the MIM estimates. Nevertheless, the reasons for so many DATs having higher estimates requires further investigation.
What this study adds
This is the first study to determine the validity of estimates of problematic drug use in the UK. 64%/52% of the DATs thought the PDU/IDU estimates were about right or did not dispute them. However, these figures are lower if the "don't knows" are excluded (40% and 25% respectively), and further work is needed to determine the reasons for this discrepancy.
Limitations of the study
Many comments were received on the validity and reliability of the indicators. For example, most DATs thought the indicator on the number in treatment was too low, that the possession and supply indicators were too high, that arrest referral and drug related deaths were about right, and did not know about methadone referrals or deprivation. Clearly, if there is heterogeneity in the completeness of accuracy of indicators this will add bias to the MIM estimates, and this may be a potential explanation for why the MIM estimates were considered too low or too high. For example, the MIM study indicated that in one particular DAT there were no IDUs, while the DAT itself reported in excess of 100 needle exchange attendances.
Given that public health expenditure on illicit drug use is based on estimates of the scale of the problem, validation should be an inherent feature of prevalence estimation studies. Recently, researchers have estimated problematic drug use for all DATs in England using capture-recapture for all DATs (except for a few where sufficient data were not available). The definition of problematic drug use was use of opiates and/or the use of crack cocaine which is narrower than the definition used in the MIM study (which also included benzodiazepines). The estimate for the median number of problematic drug users in England for 2004/5 was 327,466 . This figure represents a 14% increase on the 2001 MIM study. However, between 2001 and 2005 the rate of self-reported class A drug use in the last month among people aged 16–59 in the population dropped by 14%  and the number of drug related deaths declined by 10% . These figures suggest that problematic drug use in England declined between 2001 and 2005.
These findings have implications for future studies as there are two approaches to prevalence estimation for English DATs. The first approach would involve conducting capture-recapture studies for all DATs. The second approach would involve conducting capture-recapture studies in a small number of DATs and using the MIM to estimate prevalence for remaining DATs. The results of this paper highlight the need to consider criterion and face validity when evaluating estimates of the number of drug users.
This study was funded by the Home Office. We would like to thank: Nicola Singleton and Sara Skodbo of the Home Office for their help in administering this study and all the Drug Action Teams who took part. We are grateful to the two reviewers for their helpful comments. The views expressed in this paper are those of the authors, not necessarily those of the funders.
- Drug Strategy. Accessed Friday, 12 January 2007, [http://www.drugs.gov.uk/drug-strategy/funding]
- Drug Strategy. Accessed Friday, 12 January 2007, [http://www.drugs.gov.uk/treatment/strategy]
- Home Office: Updated Drug Strategy 2002. 2002, London: Home OfficeGoogle Scholar
- Kraus L, Augustin R, Frisher M, Kummler P, Uhl A, Wiessing L: Estimating prevalence of problem drug use at national level in countries of the European Union and Norway Addiction. 2003, 98 (4): 471-85.Google Scholar
- Gfroerer J, Brodsky M: The incidence of illicit drug use in the United States, 1962–1989. British Journal of Addiction. 1992, 87: 1345-1351. 10.1111/j.1360-0443.1992.tb02743.x.View ArticlePubMedGoogle Scholar
- European Monitoring Centre for Drugs and Drug Addiction: Study to obtain comparable national estimates of problem drug use prevalence for all EU member states. 1999, Lisbon: EMCDDAGoogle Scholar
- Rhodes W: Synthetic estimation applied to the prevalence of drug use. The Journal of Drug Issues. 1993, 23: 297-321.View ArticleGoogle Scholar
- Frisher M, Heatlie H, Hickman M: Prevalence of problematic and injecting drug use for Drug Action Team areas in England. Journal of Public Health. 2006, 28 (1): 3-9. 10.1093/pubmed/fdi073.View ArticlePubMedGoogle Scholar
- Laska EM: The use of capture-recapture methods in public health. Bull World Health Organ. 2002, 80 (11): 845-845. ISSN 0042-9686PubMedPubMed CentralGoogle Scholar
- Frisher M, Leyland A, Cormack R, Goldberg D, Bloor M, Green S, Taylor A, Covell R, McKeganey N, Platt S: Estimating population prevalence of injection drug use and HIV infection among injection drug users in Glasgow. American Journal of Epidemiology. 1993, 138: 170-181.Google Scholar
- Kerlinger FN: Foundations of behavioral research. 1973, New York: Holt, Rinehart, & WinstonGoogle Scholar
- Reuter P: Prevalence estimation and policy formulation. Journal of Drug Issues. 1993, 23 (2): 167-184.View ArticleGoogle Scholar
- Hickman M, Taylor C, Chatterjee A, et al: Estimating drug prevelance: Review of methods with special reference to developing countries. UN Bulletin of Narcotics. 2003, LIV (1 and 2): 15-32.Google Scholar
- Elliott L, Gruer L, Farrow K, Cameron J: Do Female injectors under-utilize needle exchanges?. Addiction. 1994, 89: 95-98. 10.1111/j.1360-0443.1994.tb00854.x.View ArticleGoogle Scholar
- Vickerman P, Hickman M, Rhodes T, Watts C: Model projections on the required coverage of syringe distribution to prevent HIV epidemics among injecting drug users. J Acquir Immune Defic Syndr. 2006, 42 (3): 355-61. 10.1097/01.qai.0000219788.73539.47.View ArticlePubMedGoogle Scholar
- Frisher M, Hickman H, Kraus L, Mariani F, Wiessing L: A comparison of different methods for estimating the prevalence of problematic drug misuse in Great Britain. Addiction. 2001, 96: 1465-1476. 10.1046/j.1360-0443.2001.9610146510.x.View ArticleGoogle Scholar
- Friedman SR, Tempalski B, Cooper H, et al: Estimating numbers of IDUs in metropolitan areas for structural analyses of community vulnerability and for assessing relative degrees of service provision for IDUs. Journal of Urban Health. 2004, 81: 377-400.View ArticlePubMedPubMed CentralGoogle Scholar
- Frisher M, Heatlie H, Hickman M: Estimating the prevalence of problematic and injecting drug use for Drug Action Team areas in England: a feasibility study using the Multiple Indicator Method. Online Report OLR 34/04. 2004, London: Home OfficeGoogle Scholar
- Hay G, Gannon M, MacDougall J, Millar T, Eastwood C, McKeganey N: Local and national estimates of the prevalence of opiate use and/or crack cocaine use (2004/05). Measuring different aspects of problem drug use: methodological developments. Edited by: Nicola Singleton, Rosemary Murray, Louise Tinsley. Home Office Online Report 16/06
- Roe S: Drug Misuse Declared: Findings from the 2004/05 British Crime Survey: England and Wales. 2005, London: Home OfficeGoogle Scholar
- Office for National Statistics: Deaths related to drug poisoning: results for England and Wales 2000–2004. Health Statistics Quarterly. 2006, 29: 69-76.Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/7/286/prepub