Analysis of inadequate cervical smears using Shewhart control charts
BMC Public Health volume 4, Article number: 25 (2004)
Inadequate cervical smears cannot be analysed, can cause distress to women, are a financial burden to the NHS and may lead to further unnecessary procedures being undertaken. Furthermore, the proportion of inadequate smears is known to vary widely amongst providers. This study investigates this variation using Shewhart's theory of variation and control charts, and suggests strategies for addressing this.
Cervical cytology data, from six laboratories, serving 100 general practices in a former UK Health Authority area were obtained for the years 2000 and 2001. Control charts of the proportion of inadequate smears were plotted for all general practices, for the six laboratories and for the practices stratified by laboratory. The relationship between proportion of inadequate smears and the proportion of negative, borderline, mild, moderate or severe dyskaryosis as well as the positive predictive value of a smear in each laboratory was also investigated.
There was wide variation in the proportion of inadequate smears with 23% of practices showing evidence of special cause variation and four of the six laboratories showing evidence of special cause variation.
There was no evidence of a clinically important association between high rates of inadequate smears and better detection of dyskaryosis (R2 = 0.082).
The proportion of inadequate smears is influenced by two distinct sources of variation – general practices and cytology laboratories, which are classified by the control chart methodology as either being consistent with common or special cause variation. This guidance from the control chart methodology appears to be useful in delivering the aim of continual improvement.
Each year some 3.6 million women in England have a cervical smear taken following an invitation from the national screening programme. The proportion of these that do not contain material suitable for analysis and, are therefore deemed inadequate, has risen from around 6% in the early 1990s to 9.7% in 2000–1. Women who have an inadequate smear must be retested, in line with national guidelines, and women who have three successive inadequate smears are referred for colposcopy. Inadequate smears are therefore a source of distress to women, and a waste of resources in general practices, clinics and cytology laboratories.
There is wide variation in the proportion of inadequate smears between providers. This paper analyses this variation using Walter Shewhart's theory of variation which classifies variation as either emanating from a "common cause" or a "special cause" and gives guidance on the appropriate action required to address these two types of variation and so deliver continual improvement.
Common cause variation is expected variation attributable to "chance". It is part of every process and affects everyone in that process. To reduce common cause variation we need to fundamentally change the underlying process. In contrast, special cause variation is exceptional variation not attributable to "chance", but arising from special circumstances and therefore not affecting everyone in that process. Special cause variation can produce exceptionally good or bad results. To reduce unfavourable special cause variation we need to identify the special cause and act on it. In the case of favourable special cause variation, we likewise need to investigate and learn the lessons therein for the benefit of others.
Shewhart's methods have been widely used in industry [3, 4], in laboratory settings , and in communicable diseases control [6, 7]. The use of control charts has been suggested in clinical governance and health care performance monitoring [8, 9]. This paper reports an investigation of variation in the proportion of inadequate smears in South Staffordshire Health Authority using control charts.
Cervical cytology data routinely collected by the Health Authority from the laboratories serving the district was obtained for all 100 general practices in South Staffordshire. The data included the number of smears taken and the number inadequate for the calendar years 2000 and 2001.
We investigated two distinct sources of variation – variation by general practice and variation by cytology laboratory. We undertook these analyses using P-charts. P-charts are one member of the family of control charts and are designed to be used with binomial data (inadequate smear – yes/no) which is expressed as a proportion of the sample size. We sorted our data by the total number of smears (ie sample size) and plotted our P-charts with respect to this order. This has two advantages. Firstly the resulting P-charts show the impact of sample size on the control limits, and secondly the resulting control limits are easy on the eye because they appear like a funnel. Such plots have been advocated in health care .
National comparative data, which included the proportion of inadequate smears, the proportion of smears classified as mild dyskariosis, severe dyskariosis and the positive predictive value of a smear, was obtained for all 156 laboratories in England for the years 2000–2001. The relationship between the proportion of smears classified as inadequate by a laboratory and the positive predictive value of a smear in that laboratory was investigated by linear regression analysis. The relationship between the proportion of smears classified as inadequate and the proportion of smears classified as negative (no dyskariosis), mild dyskariosis, moderate dyskariosis or severe dyskariosis was investigated by step-down multivariate regression. All analysis was carried out using SPSS 11.0. For the step-down analysis the proportion of inadequate smears was the dependent variable and the proportion categorised as negative, borderline, mild, moderate and severe dyskariosis were independent variables. The regression was repeated with non-significant variables eliminated in a stepwise fashion.
Analysis of inadequate smears by general practice
Inter-general practice variation in the proportion of inadequate smears was explored using a P-chart for our 100 general practices (Figure 1). There is wide variation in the proportion of inadequate smears between general practices, with 23% of general practices showing evidence of special cause variation – 12% showing low special cause variation and 11% showing high special cause variation. However, the majority of practices (77%) are consistent with common cause variation.
Analysis of inadequate smears by cytology laboratory
We also explored the inter-laboratory variation in the proportion of inadequate smears. The results are plotted on a P-chart (Figure 2). Laboratories 1 and 2 show evidence of special cause variation with exceptionally high proportions of inadequate smears. Laboratories 3 and 4 show special cause variation with very low proportions of inadequate smears.
Analysis of inadequate smears by general practice and cytology laboratory
To provide further insight into the interaction between the two sources of variation (general practice and cytology laboratory), we analysed the proportion of inadequate for each general practice by cytology laboratories. We produced six P-charts one for each laboratory (Figure 3). The mean proportion of inadequate smears ranged from 12.7% (laboratory 1) to 6.7% (laboratory 6).
Two of the six practices served by laboratory 1 show special cause variation. Six of the 40 practices served by laboratory 2 show special cause variation. One of the seven practices served by laboratory 3 shows special cause variation. Four of the 41 practices served by laboratory 4 show special cause variation. Practices served by laboratory 5 and by laboratory 6 show common cause variation.
Further analysis of inadequate smear rates
It not clear whether a high or low inadequate rate is preferable; either too many smears are being rejected unnecessarily or inadequate smears are being analysed inappropriately. To investigate this further data were obtained on the results of laboratories nationally.
No relationship was found between the proportion of smears reported as inadequate and the positive predictive value of a smear in all 156 English laboratories [P = NS]. The positive predictive value of a smear is the percentage of referrals following a smear that have grade 2 cervical intraepithelial neoplasia or worse. The step-down multivariate regression found only the proportion of smears classified as negative to predict the proportion classified as inadequate (beta coefficient = -0.319; P < 0.001). However, this explained little of the variation in the rates reported as inadequate (R squared = 0.082), suggesting that laboratories classifying larger proportions of smears as inadequate do not report higher proportions of moderate or severe dyskaryosis.
Our study shows that there is wide variation in the proportion of inadequate smears amongst our 100 general practices, with 23% showing evidence of special cause variation which merit further investigation to identify possible causes. However the vast majority of general practices (77%) are consistent with common cause variation, which, according to Shewhart's theory of variation, is best addressed by introducing fundamental changes to the underlying process.
The National Institute for Clinical Excellence (NICE) has recently recommended the adoption of liquid based cytology (LBC), in part because it is associated with significantly lower proportions of inadequate smears from 9% to 1.6% . This constitutes a fundamental change in the cytology process. However, the inadequate smear rates reported were based on only three pilot site laboratories, the evaluation of which states that the long term reduction of the rate of inadequate smears cannot be assessed from the data reported. The systematic review that informed the NICE decision also cites studies that show an overall mean inadequate rate of 1.4% (5% and 95% percentile 0.3% and 9.1%) for conventional cytology and 0.8% (5% and 95% percentile 0.1% and 5.5%) for LBC. These studies also show great heterogeneity with either method and were not consistent with the current mean inadequate rate of 9% in NHS practice. This is reflected in NICE's statement that although LBC offers a decrease in the proportion of inadequate specimens "the literature reveals a wide and overlapping range in this proportion with both conventional smears and liquid-based methods."
Although LBC technology is potentially associated with significant reductions in the proportion of inadequate smears, we cannot be confident that the impact of change will be readily discernable in general practices/laboratories because (a) it is not clear that the results of the pilot evaluations are generalisable to practices/laboratories with special cause variation, and (b) there is potential interaction between the new technology and the existing but as yet unidentified special cause variation. As a first step we would recommend that special causes at these general practices/laboratories be identified (and where appropriate eliminated) before introducing changes to the underlying process. The following guidance has been recommended  in investigating special cause variation – check the data, check the case-mix, check the process, check the resources and finally check the individuals involved. The full benefits of LBC will only be realised if its use in practice is optimised. Under the philosophy of continual improvement, control charts, even in the presence of very low inadequate rates, offer a way of doing this because a key feature of the control chart methodology is that "special cause" variation is, by definition, economic to find and remove.
Our approach differs from the current methods employed in the NHS to analyse variations in inadequate smear rates. At present, quality assurance targets are set from the 10th to the 90th percentiles of the distributions for laboratories nationally. However, this fails to recognise that results from different laboratories cannot be assumed to be consistent with common cause variation, even in the 10th–90th percentile, as we have shown. In addition, a target based approach seems to focus attention on those laboratories who have "missed the target", implying that there is no "need" for "on target" laboratories to take action. In sharp contrast Shewhart's approach requires action for both special and common cause variation and so is able to support efforts for continual improvement.
We have undertaken a comparative analysis between general practices and laboratories. It would be very useful to investigate how the proportion of inadequate smears change over time (eg quarterly), for each laboratory and for each general practice. This longitudinal analysis would complement the cross-sectional analysis and provide a more complete picture of the variation in the proportion of inadequate smears.
Control charts offer an action-oriented way of analysing cervical smear data. By highlighting areas for further investigation they act as a tool for hypothesis generation about the causes of variation. In this way they direct us towards the most efficient action that will achieve quality improvement.
This study has shown that control charts can be used to analyse routine cervical data at a health authority level. Two distinct sources of variation in the inadequate smear data can be identified; that associated with laboratories and that associated with GP practices.
Cervical Screening Programme, England: 2000–01. Statistical Bulletin 2001/22. 2001, London: Government Statistical Service
Johnson J, Patnick J: Achievable standards, benchmarks for reporting and criteria for evaluating cervical cytopathology. 2000, Sheffield: NHSCSP Publications, Second
Deming WE: Out of the crisis. 1986, Cambridge, Massachusetts: Massachusetts Institute of Technology
Deming WE: The New Economics. 1994, Cambridge, Massachusetts: Massachusetts Institute of Technology
Chesher D, Burnett L: Using Shewhart p control charts of external quality-assurance program data to monitor analytical performance of a clinical chemistry laboratory. Clinical Chemistry. 1996, 42: 1478-82.
Benneyan JC: Statistical quality control methods in infection control and hospital epidemiology. Part 1: introduction and basic theory. Infection Control and Hospital Epidemiology. 1998, 19: 194-214.
Benneyan JC: Statistical quality control methods in infection control and hospital. Part 2: chart use, statistical properties, and research issues. Infection Control and Hospital Epidemiology. 1998, 19: 265-277.
Mohammed MA, Cheng KK, Rouse A, Marshall T: Bristol Shipman: and clinical governance: Shewhart's forgotten lessons. Lancet. 2001, 357: 463-467. 10.1016/S0140-6736(00)04019-8.
Adab P, Rouse AM, Mohammed MA, Marshall T: Performance league tables: the NHS deserves better. British Medical Journal. 2002, 324: 95-5.
Spiegelhalter D: Funnel plots for institutional comparison. Qual Saf Health Care. 2002, 11: 390-a-391. 10.1136/qhc.11.4.390-a.
National Institute for Clinical Excellence: NHS Technology Appraisal Guidance 69. Guidance on the use of liquid-based cytology for cervical screening. 2003, London: National Institute for Clinical Excellence
Moss SM, Gray A, Legood R, Henstock E: Evaluation of HPV/LBC Cervical Screening Pilot Studies. First report to the Department of Health on evaluation of LBC. Sutton: Institute of Cancer Research. 2003
Karnon J, Peters J, Chilcot J, McGoogan E: Liquid-based cytology in cervical screening: an updated rapid and systematic review. Sheffield: The School of Health and Related Research. 2003
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/4/25/prepub
MW conceived the study based on an original idea by TM. WH carried out the initial analysis and drafted the paper. TM carried out further analysis and revised the draft text. MAM provided expert statistical advice and produced the revised final manuscript. All authors contributed to the final manuscript.
About this article
Cite this article
Harrison, W.N., Mohammed, M.A., Wall, M.K. et al. Analysis of inadequate cervical smears using Shewhart control charts. BMC Public Health 4, 25 (2004). https://doi.org/10.1186/1471-2458-4-25