This article has Open Peer Review reports available.
Epidemic features affecting the performance of outbreak detection algorithms
© Kuang et al. 2012
Received: 17 January 2012
Accepted: 8 June 2012
Published: 8 June 2012
Outbreak detection algorithms play an important role in effective automated surveillance. Although many algorithms have been designed to improve the performance of outbreak detection, few published studies have examined how epidemic features of infectious disease impact on the detection performance of algorithms. This study compared the performance of three outbreak detection algorithms stratified by epidemic features of infectious disease and examined the relationship between epidemic features and performance of outbreak detection algorithms.
Exponentially weighted moving average (EWMA), cumulative sum (CUSUM) and moving percentile method (MPM) algorithms were applied. We inserted simulated outbreaks into notifiable infectious disease data in China Infectious Disease Automated-alert and Response System (CIDARS), and compared the performance of the three algorithms with optimized parameters at a fixed false alarm rate of 5% classified by epidemic features of infectious disease. Multiple linear regression was adopted to analyse the relationship of the algorithms’ sensitivity and timeliness with the epidemic features of infectious diseases.
The MPM had better detection performance than EWMA and CUSUM through all simulated outbreaks, with or without stratification by epidemic features (incubation period, baseline counts and outbreak magnitude). The epidemic features were associated with both sensitivity and timeliness. Compared with long incubation, short incubation had lower probability (β* = −0.13, P < 0.001) but needed shorter time to detect outbreaks (β* = −0.57, P < 0.001). Lower baseline counts were associated with higher probability (β* = −0.20, P < 0.001) and longer time (β* = 0.14, P < 0.001). The larger outbreak magnitude was correlated with higher probability (β* = 0.55, P < 0.001) and shorter time (β* = −0.23, P < 0.001).
The results of this study suggest that the MPM is a prior algorithm for outbreak detection and differences of epidemic features in detection performance should be considered in automatic surveillance practice.
Infectious diseases remain the major causes of morbidity and mortality in China despite substantial progress in their control . The outbreaks of infectious diseases pose serious threats on public health. Early detection of aberration and rapid control actions, which the Chinese Ministry of Health has taken as important strategies for emergency infectious disease prevention and control , are prerequisites for preventing the spread of outbreaks and reducing the morbidity and death caused by diseases. Therefore, China Infectious Disease Automated-alert and Response System (CIDARS) was successfully implemented and began to operate nationwide in 2008 .
At the end of 2010, analysis results of the operation of CIDARS in nationwide showed that a large number of outbreaks of infectious diseases could be timely detected, but it was also found that there were many of false-positive signals; large differences existed between outbreak signal counts and final identified outbreaks in different diseases; the detection performance was poor in those diseases which had more case reports and fewer outbreaks . These issues prompted us that epidemic features of infectious disease may affect outbreak detection performance.
Several studies have described the determinants of outbreak detection performance, including: system factors (representativeness, outbreak detection algorithms and algorithm-specifics), outbreak characteristics (outbreak size, shape of the outbreak signal and time of the outbreak) [5–8]. Understanding the differences these determinants make in detection performance can help public health practitioners improve the automated surveillance system, thus raising detection capabilities. Recently, extensive researches have explored novel algorithms to improve the performance of outbreak detection [9–13], but evidence on how epidemic features impact on detection performance is still limited.
To address this limitation, our study aimed to explore the influence of epidemic features (incubation period, baseline counts and outbreak magnitude) on algorithms’ detection performance. Findings of this study may help public health surveillance practitioners understand the detection performance of algorithms under these epidemic features and improve the implementation of automated surveillance.
The infectious disease data were extracted from CIDARS. CIDARS was developed basing on the existing data from National Disease Surveillance Reporting and Management System on 28 diseases that are outbreak-prone and require prompt action. The 28 diseases were classified into two types according to severity, incidence rate and importance . Type 1 diseases includes nine infectious diseases characterized with higher severity but lower incidence and are analysed using fixed-threshold detection method. For type 2 diseases (19 more common infectious diseases) , we selected eight diseases (dysentery, scarlet fever, mumps, measles, malaria, typhoid, encephalitis B and hepatitis A) which represented three routes of transmission(respiratory, oral-fecal and vector-borne). Five provinces were sampled for eight diseases(dysentery in Hunan, scarlet fever, measles in Xinjiang, mumps in Chongqing, malaria, typhoid, encephalitis B in Yunnan, hepatitis A in Guizhou), where the respective disease had high incidence and became important local public health problems. Then we randomly sampled 10 counties from the selected provinces for each disease, and obtained their actual daily number of reported cases in 2005–2009. Data from 2005 to 2007 were used as baseline, while data from 2008 to 2009 were used to evaluate the algorithms.
Strategy of inserting simulated outbreaks was used to evaluate the detection performance. To prevent public health emergency confounding the evaluation, we got the records of public health emergency in the corresponding counties from Emergency Public Reporting System .
Outbreak detection algorithms
To date many outbreak detection algorithms can be used for temporal data [15–19]. Considering that we collected five years data, using the same periods’ historical data as baseline is appropriate, to some extent, can reduce the seasonal and day-of-week variation in the baseline. However, for using this, the regression and ARIMA models may subject to certain restrictions, as the steps in their processing require recent continuous time interval to calculate expected statistic. So we chose two most commonly used statistical process control algorithms (EWMA, CUSUM) and a non-parameter algorithm(MPM) which enable the application of the same periods’ historical data in theory.
In the algorithm, the λ (0 < λ < 1) was the weighting factor, k was the control limit coefficient .
If the CUSUM statistic is larger than h, then the current day is considered as a possible outbreak.
The MPM uses previous several years (such as 3–5 years) over the same period as baseline data, setting a percentile of baseline data as a detection parameter c. If the current day counts xt is greater than the detection parameter's corresponding percentile (Pc), outbreak signal is produced .
To obtain the optimized parameter values of the three algorithms, we used R software  generating two-year Poisson distribution sequences with five daily average counts levels (0.1, 0.5, 1.0, 2.0, 5.0). We set a fixed false alarm rate of 0.05 (an average of one false alarm every twenty days) by applying each algorithm to these five sequences without any added outbreak signals, and determining the parameters that would yield an average of one false alert every twenty days (see Additional file 1: Table S1).
Outbreak simulation and insertion
Summary the number of injected outbreak signals
Outbreaks should be injected
Public health emergencies
Actual injected outbreaks
Classification the incubation of diseases with K-means clustering
Performance comparisons were based on three indicators: sensitivity (the proportion of outbreaks the algorithm detected), timeliness (the difference between the date of the first true alarm and the beginning date of the outbreak) and false alarm rate (the proportion of non-outbreak days on which the algorithm signal an alarm).
For more informative comparisons of the performance in different determinants, we plotted sensitivity-timeliness curve , which measured the proportion of outbreaks that an algorithm detected within several days from the start of the outbreak.
Statistical inference and multiple linear regression
For each of the eight diseases in each of the ten counties on each of the four out- break magnitudes, we computed the sensitivity, timeliness and false alarm rate across all 320 analysis runs. We used ANOVA to test for significant difference in the algorithms’ sensitivities and timeliness. The Bonferroni correction was applied for multiple comparisons to control the family wise error rate. The significance level α was 0.05. Finally, multiple linear regression was run to understand the relationship of the algorithms sensitivities and timeliness with disease incubation, baseline counts and outbreak magnitude. All the analyses were performed using R software.
Summary the performance of outbreak detection algorithms
95% confidence interval
False alarm rate(%)
In the three incubation categories, the MPM had a higher probability of detecting the outbreak than CUSUM and EWMA within all days from the start of the outbreak; the sensitivity changes of CUSUM and EWMA over time were very close. In short incubation, the sensitivity of MPM reached 90% at the sixth day of a outbreaks start. The long incubation disease had a poorer sensitivity than short and medium incubation disease (Figure 5).
We used the 33.33, 66.67 percentiles with the cuts off 0.05, 0.2 to divide average baseline counts into low, medium and high levels. At the three levels, the MPM had a higher probability of detecting the outbreak than CUSUM and EWMA within all days from the start of the outbreak; the sensitivity of CUSUM and EWMA was very closely changed over time. The sensitivity of low baseline counts was better than medium and high levels, with the MPM reaching higher than 90% (Figure 6).
Multiple linear regression
Variable coding in multiple linear regression
Sensitivity or Timeliness
Summary of Multiple Regression Analysis for Variables Predicting Sensitivity
R 2 = 0.39
Summary of Multiple Regression Analysis for Variables Predicting Timeliness
R 2 = 0.28
As it was illustrated in Table 5, the algorithms, incubation period, baseline counts and outbreak magnitude had a statistically significant relationship with sensitivity. MPM had a higher probability of detecting outbreaks compared with EWMA (β* = 0.21, P < 0.001). Short(β* = −0.13, P < 0.001) and Medium(β* = −0.11, P = 0.002) incubation had a lower probability of detecting the outbreaks compared with long incubation. The lower the baseline counts, the higher probability (β* = −0.20, P < 0.001). The larger outbreak magnitude, the higher probability (β* = 0.55, P < 0.001).
The algorithms, incubation period, baseline counts and outbreak magnitude had a statistically significant relationship with timeliness. MPM needed shorter time to detect the outbreaks compared with EWMA(β* = −0.17, P < 0.001). Short(β* = −0.57, P < 0.001) and Medium(β* = −0.56, P < 0.001) incubation needed shorter time to detect the outbreaks compared with long incubation. The higher baseline counts, the longer time to detect the outbreaks (β* = 0.14, P < 0.001). The larger outbreak magnitude, the shorter time (β* = −0.23, P < 0.001) (Table 6).
Many determinants affect the performance of outbreak detection in automated surveillance, and knowing about how these factors influence the detection performance can help to improve automated surveillance system. In this study, we compared the performance of three outbreak detection methods by adding simulated outbreaks to actual daily counts of notifiable infectious diseases in CIDARS and examined the relationship of the detection performance with disease incubation, baseline counts and outbreak magnitude.
In algorithms’ detection performance, we found MPM had better performance than the EWMA, CUSUM methods. In theory, MPM method is simple, with fewer parameters and without the limit of the overall distribution of monitoring data. The results showed that the performance of MPM was stable under different test conditions, which indicated that the MPM method has a broad scope of application. These advantages prompted that this method should be first considered when designing an automatic disease surveillance system.
Consistent with previous evaluations of outbreak detection algorithms [5, 34–36], the multiple linear regression results found that the ability to detect outbreaks was better with lower baseline counts and larger magnitude. While previous studies inserted the actual counts with fixed outbreak case numbers, our study inserted the actual counts with outbreak case numbers based on the proportion of case distribution of simulated outbreaks.
Our study examined how different incubation periods affect the detection performance. There are three indicators(minimum, maximum and average of incubation period) to describe incubation, and these three indicators are closely related. To date there is still no definitely way of classifying the incubation period with these three indicators. So we tried the K-means clustering method to classify eight types of disease into three categories. The regression results showed that diseases of long incubation period had a higher sensitivity, but needed more time to detect than those of short and medium incubation period. Generally, the outbreaks of short incubation diseases occur ferociously and transiently, which are more easily to be missed by algorithms. The outbreaks of long incubation diseases, however, occur with longer duration, and can be detected by algorithms more accurately. As the early detection of outbreaks is important, additional work is still needed for timely detection long incubation diseases.
The biggest challenge for the evaluation of outbreak detection algorithms is to obtain a sufficiently large number of outbreak data with which to measure sensitivity and timeliness . Injecting geometric shaped spikes into real surveillance data is a feasible approach [5, 36–39]. In this study, in addition to inserting literatures-based simulated outbreak, we also used the public health emergency data. This method provided a solution on the issues where the simulated data may completely detach from real outbreaks and outbreaks in the real surveillance data may interfere with the performance evaluation.
We set a fixed false alarm rate of 0.05, using simulation method to obtain the optimized parameters of different daily average counts data, and focused on the evaluation based on sensitivity and timeliness, which could make the comparison more clear.
There are several limitations to our study that should be taken into consideration. First, eight types of diseases clustering into three categories may not have a good representation. Second, the simulated outbreaks of eight diseases based on literature have some limitations to reflect the complexity of real outbreaks. Third, as only using two-year test datasets, we inserted a limited number of simulated outbreaks, which, to some extent, may affect the stability of the evaluation. In addition, due to the large amounts of computation in this study, we only compared three detection algorithms, so the evaluation of other algorithms needed to be further carried out.
The results of this study show that the MPM has better detection performance than EWMA, CUSUM. It can be considered as a prior algorithm for automatic infectious disease outbreak detection. Infectious disease outbreak detection performance varies with incubation period, baseline counts and outbreak magnitude. This suggests that the actual automatic infectious disease surveillance practice should take epidemic features into consideration, and select the appropriate algorithm to improve detection performance.
This research is funded by the Ministry of Science and Technology of China (2006BAK01A13, 2008BAI56B02, 2009ZX10004-201).
- Wang LD, Wang Y, Jin SG, Wu ZY, Chin DP, Koplan JP, Wilson ME: Emergence and control of infectious diseases in China. Lancet. 2008, 372 (9649): 1598-1605. 10.1016/S0140-6736(08)61365-3.View ArticlePubMedGoogle Scholar
- Ministry of Health of the People’s Republic of China: Emergency infectious disease prevention and control strategies. 2007, Ministry of Health of the People’s Republic of China, BeijingGoogle Scholar
- Yang WZ, Li ZJ, Lan YJ, Wang JF, Ma JQ, Jin LM, Sun Q, Lv W, Lai SJ, Liao YL, Hu WB: A nationwide web-based automated system for outbreak early detection and rapid response in China. Western Pacific Surveillance and Response Journal. 2011, 2 (1): 1-6.Google Scholar
- Yang WZ, Lan YJ, Li ZJ, Ma JQ, Jin LM, Sun Q, Lv W, Lai SJ: The application of national outbreak automatic detection and response system, China. Chin J Epidemiol. 2010, 31 (11): 1240-1244.Google Scholar
- Wang L, Ramoni MF, Mandl KD, Sebastiani P: Factors affecting automated syndromic surveillance. Artificial Intelligence in Medicine. 2005, 34 (3): 269-278. 10.1016/j.artmed.2004.11.002.View ArticlePubMedGoogle Scholar
- Buckeridge DL: Outbreak detection through automated surveillance: A review of the determinants of detection. Journal of Biomedical Informatics. 2007, 40 (4): 370-379. 10.1016/j.jbi.2006.09.003.View ArticlePubMedGoogle Scholar
- Wang XL, Zeng D, Seale H, Cheng H, Luan RS, He X, Pang XH, Dou XF, Wang QY: Comparing early outbreak detection algorithms based on their optimized parameter values. Journal of Biomedical Informatics. 2010, 43 (1): 97-103. 10.1016/j.jbi.2009.08.003.View ArticlePubMedGoogle Scholar
- Pelecanos AM, Ryan PA, Gatton ML: Outbreak detection algorithms for seasonal disease data: a case study using ross river virus disease. BMC Med Inform Decis Mak. 2010, 10: 74-10.1186/1472-6947-10-74.View ArticlePubMedPubMed CentralGoogle Scholar
- Watkins RE, Eagleson S, Veenendaal B, Wright G, Plant AJ: Disease surveillance using a hidden Markov model. BMC Med Inform Decis Mak. 2009, 9: 39-10.1186/1472-6947-9-39.View ArticlePubMedPubMed CentralGoogle Scholar
- Jiang X, Cooper GF: A real-time temporal Bayesian architecture for event surveillance and its application to patient-specific multiple disease outbreak detection. Data Mining and Knowledge Discovery. 2010, 20 (3): 328-360. 10.1007/s10618-009-0151-4.View ArticleGoogle Scholar
- Lu HM, Zeng D, Chen HC: Prospective Infectious Disease Outbreak Detection Using Markov Switching Models. Ieee Transactions on Knowledge and Data Engineering. 2010, 22 (4): 565-577.View ArticleGoogle Scholar
- Shen Y, Cooper GF: A New Prior for Bayesian Anomaly Detection Application to Biosurveillance. Methods of Information in Medicine. 2010, 49 (1): 44-53.PubMedGoogle Scholar
- Alimadad A, Salibian-Barrera M: An Outlier-Robust Fit for Generalized Additive Models With Applications to Disease Outbreak Detection. Journal of the American Statistical Association. 2011, 106 (494): 719-731. 10.1198/jasa.2011.tm09654.View ArticleGoogle Scholar
- Jiang FJ: Evaluation research on the quality of reporting in Emergency Public Reporting System. Master thesis. Chinese Center for Disease Control and Prevention. 2006Google Scholar
- Stroup DF, Williamson GD, Herndon JL, Karon JM: Detection of aberrations in the occurrence of notifiable diseases surveillance data. Statistics in Medicine. 1989, 8: 323-329. 10.1002/sim.4780080312.View ArticlePubMedGoogle Scholar
- Farrington CP, Andrews NJ, Beale AD, Catchpole MA: A statistical algorithm for the early detection of outbreaks of infectious disease. Journal of the Royal Statistical Society, Series A. 1996, 159: 547-563.View ArticleGoogle Scholar
- Williamson GD, Hudson GW: A monitoring system for detecting aberrations in public health surveillance reports. Statistics in Medicine. 1999, 18: 3283-3298. 10.1002/(SICI)1097-0258(19991215)18:23<3283::AID-SIM316>3.0.CO;2-Z.View ArticlePubMedGoogle Scholar
- Reis BY, Mandl KD: Time series modeling for syndromic surveillance. BMC Med Inform Decis Mak. 2003, 3: 2-10.1186/1472-6947-3-2.View ArticlePubMedPubMed CentralGoogle Scholar
- Zhang J, Tsui FC, Wagner MM, Hogan WR: Detection of Outbreaks from Time Series Data Using Wavelet Transform. AMIA Annu Symp Proc. 2003, 748-752.Google Scholar
- Montgomery DC: Introduction to Statistical Quality Control. 2001, John Wiley and Sons, New YorkGoogle Scholar
- Hutwagner LC, Maloney EK, Bean NH, Slutsker L, Martin SM: Using laboratory-based surveillance data for prevention: an algorithm for detecting salmonella outbreaks. Emerg Infect Dis. 1997, 3: 395-400. 10.3201/eid0303.970322.View ArticlePubMedPubMed CentralGoogle Scholar
- Yang WZ, Xing HX, Wang HZ, Lan YJ, Sun Q, Hu SX, Lv W, Yuan ZA, Chen YX, Dong BQ: A study on early detection for seven infectious diseases. Chin J Epidemiol. 2004, 25 (12): 1039-1041.Google Scholar
- R Development Core Team: R: A language and environment for statistical computing. [http://www.R-project.org]
- Zhang RJ, Ji GH, Wang Y, Zai B: Investigation of a Shigella bacteria cause bacillary dysentery outbreak. Chin J Pest Control. 2009, 25 (6): 437-438.Google Scholar
- Li SX, Li LJ, Yu QF, Yu FB, Yu HX, Wu BW: Investigation of a scarlet fever outbreak in a school, Yuxi City. Chin J Sch Health. 2005, 26 (6): 510-511.Google Scholar
- Zhang TT, Chen HX: Analysis of a mumps outbreak in a junior middle school of Huaxi town, Huayin City. Henan Journal of Preventive Medicine. 2009, 20 (6): 449-450.Google Scholar
- Huo HC: Report on a measles outbreak investigation. Disease surveillance. 1997, 11: 430-Google Scholar
- Yao LJ, Chen HH: Analysis of a malaria outbreak investigation. Guangdong health and epidemic prevention. 1998, 24 (1): 43-44.Google Scholar
- Yin K, Li MQ, Liu JJ: Analysis of typhoid outbreak investigation in a school, Liuzhou. Modern preventive medicine. 2007, 34 (13): 2559-2560.Google Scholar
- Chen ZZ: A outbreak investigation of encephalitis B in 2000 year, Anxi County. Journal of Preventive Medicine Information. 2001, 17 (4): 278-279.Google Scholar
- Nong ZM, Meng LJ, Liu ZH, Jiang CH: Investigation of a hepatitis A outbreak in a school. Chin J Sch Health. 2009, 30 (3): 274-275.Google Scholar
- Ma YL: Infectious diseases. 2004, Shanghai scientific & Technical Publishers, ShanghaiGoogle Scholar
- Wagner MM, Wallstrom G: Methods For Algorithm Evaluation. Handbook of Biosurveillance. Edited by: Wagner MM, Moore AW, Aryel RM. 2006, Elsevier Press, Burlington, 301-310.View ArticleGoogle Scholar
- Nordin JD, Goodman MJ, Kulldorff M, Ritzwoller DP, Abrams AM, Kleinman K, Levitt MJ, Donahue J, Platt R: Simulated anthrax attacks and syndromic surveillance. Emerg Infect Dis. 2005, 11 (9): 1396-1400.View ArticleGoogle Scholar
- Ozonoff A, Forsberg L, Bonetti M, Pagano M: Bivariate method for spatio-temporal syndromic surveillance. MMWR Morb Mortal Wkly Rep. 2004, 53 (Suppl): 59-66.Google Scholar
- Jackson ML, Baer A, Painter I, Duchin J: A simulation study comparing aberration detection algorithms for syndromic surveillance. BMC Med Inform Decis Mak. 2000, 7: 6-View ArticleGoogle Scholar
- Hutwagner L, Browne T, Seeman GM, Fleischauer AT: Comparing aberration detection methods with simulated data. Emerg Infect Dis. 2005, 11: 314-316. 10.3201/eid1102.040587.View ArticlePubMedPubMed CentralGoogle Scholar
- Goldenberg A, Shmueli G, Caruana RA, Fienberg SE: Early statistical detection of anthrax outbreaks by tracking over-the-counter medication sales. Proc Natl Acad Sci USA. 2002, 99 (8): 5237-5240. 10.1073/pnas.042117499.View ArticlePubMedPubMed CentralGoogle Scholar
- Reis BY, Pagano M, Mandl KD: Using temporal context to improve biosurveillance. Proc Natl Acad Sci USA. 2003, 100 (4): 1961-1965. 10.1073/pnas.0335026100.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/12/418/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.