In the past decade chronic diseases have emerged as the most important health and health care problem in developed countries [1]. Accurate ascertainment of chronic disease is essential for sound research, clinical care and health care planning. While multiple data sources have been used to identify persons with chronic diseases—including population health surveys, disease registries, medical chart abstraction, and administrative databases—no method has emerged as the gold standard for diagnosis. Increasingly, health administrative data is being used for disease ascertainment and surveillance. While administrative data is not collected for the purposes of disease surveillance and there are concerns about the accuracy of the diagnostic information, it is a relatively inexpensive method that can provide information on large populations and can be explored overtime. In addition, administrative data can often be compared directly with other data sources, such as self-reported surveys, using individual record linkage techniques. Regardless of the data source used, there will be missed cases. The quality of the research using these data will depend on the identification, measurement of these errors, and correction or discussion of biased results.

Research comparing disease ascertainment using health administrative data with other data sources has found that agreement between sources relies on many factors, including individual and disease characteristics and the specific methods applied to the data [2–6]. Most published research has been case ascertainment validation studies of disease definitions applied to administrative data to identify disease cases [3, 4, 7, 8]. Only a few studies have explored the accuracy of administrative data compared with self-report of chronic conditions across chronic conditions [9, 10]. Agreement between these sources is higher for chronic diseases that are well-defined and require ongoing management, such as diabetes, and is lower for poorly defined diseases such as congestive heart failure [9, 10]. Worsening comorbidity, measured by number of chronic conditions, appears to be associated with lower agreement—particularly for those diseases where agreement is already poor [11, 12].

The objective of this study is to describe the validity of administrative data compared with self-reported disease status for the ascertainment of seven common chronic conditions in the general population and to explore the relationship with self-reported disease burden. This research takes advantage of individually linked health administrative and population survey data available for the population of Ontario, the largest province in Canada.

Our research expands on what is known from previous validation studies of administrative data by using the validated disease algorithms to identify cases in the Ontario provincial health administrative data and compares prevalence estimates with the self-report of chronic diseases in the Canadian Community Health Survey (CCHS). A limitation of previous studies was that they relied on typical measures of concordance such as Kappa scores or sensitivity and specificity to determine concordance; but, in the absence of a gold standard data source and when the prevalence of a condition is particularly high or low, these measures may not be as accurate and the underlying patterns in the data are obscured. This study will present a range of concordance measures to explore the validity of administrative data for disease ascertainment. A number of diseases are examined to explore whether concordance is similar across conditions. Finally, while previous research has used a count of chronic conditions to measure morbidity, we explore the relationship between morbidity and concordance of the data with the Health Utilities Index (HUI) [13]. To our knowledge, this is the first time that the HUI has been used to understand concordance in disease ascertainment.

This study was conducted using population based data from the province of Ontario, Canada—with a 2011 population of more than 13 million [14]. The Institute for Clinical Evaluative Sciences (ICES) houses the health administrative data on hospital and physician billings, provided by the provincial health ministry, as well as population survey data. These data have been individually linked using an anonymized identification number.

We evaluated agreement between health administrative data and self-report for ascertainment of chronic disease, in a population of community dwelling Ontarian residents, using linked population-based data. With the exception of acute myocardial infarction and stroke, prevalence estimates for diseases were higher based on health administrative data compared to self-report data. In general, we found that there was a good level of agreement between data sources only for diabetes and hypertension. For the remaining diseases that were examined, there was considerable discordance in ascertainment that could only be partially explained by individual and disease characteristics. There are likely multiple reasons for these discordances that include: disease specific biases, misclassification due to the disease definitions used and the prevalence of the disease.

Okura proposes that diseases which are less familiar to patients and have nonspecific and intermittent symptoms, such as heart failure or chronic lung disease, may be particularly prone to underreporting by patients [12]. Conversely, administrative data may be more likely to identify chronic diseases requiring ongoing contact with the health care system [10, 12]. This is in keeping with our results—where disease prevalence, by health administrative data, was higher for most diseases. Our finding that the self-reported prevalence of stroke or myocardial infarction was higher than the prevalence from administrative data is also consistent with other studies [11, 12, 22]. These two diseases are commonly known in the community and this may lead to patients falsely attributing their symptoms to them. False-positive rates of self-reported stroke ranging from 5% to 15% have even been reported from specialized stroke units, mostly from patients admitted with transient ischaemic attacks [23]. Rosamond et al. found a 40% false-positive self-report of myocardial infarction among patients in a coronary care unit, primarily due to hospitalization for unstable angina [24].

The particular question used in a survey and the case definition employed in administrative data can also affect ascertainment. In general, health administrative definitions restrict to patients with hospitalization or repeated health care contact for a disease and have a limited look-back period. This could lead to underreporting by administrative data particularly important for “event-based” diseases such as stroke and myocardial infarction where “silent” events not requiring hospitalization or events that occur outside the time period are not identified. The particular question used for ascertainment can impact ascertainment. For example, in this study the survey question for stroke was “do you suffer from the consequences of a stroke” and the health administrative data definition identified all persons admitted to the hospital with a diagnosis of stroke or transient ischemic attack (Additional file 1: Appendices B and C).

There is no agreement about which concordance measure is most valid when comparing ascertainment between data sources. Level of agreement in this study varied widely depending on the measure used particularly for the low prevalent diseases. For example, while stroke concordance was very high when comparing raw prevalence estimates (1.0% and 1.7% for administrative and self-report) it was only fair according to kappa (κ = 0.36). Some concordance measures have known limitations that are important in this context: sensitivity and specificity are less valid when no gold standard for diagnosis exists and the Kappa statistic is unreliable in the setting of a significant imbalance in the 2 × 2 table [21]. In a recent review evaluating the quality of health administrative data, Benchimol et al. proposed that a minimum of four statistical measures should be used to assess for accuracy and validity of administrative data source to help mitigate these limitations [25]. Others have similarly recommended that when measuring agreement in administrative data researchers should report kappa, the prevalence, positive agreement, negative agreement and relative frequency of each cell (a, b, c and d) [26]. While there are other measures of agreement, such as the prevalence adjusted Kappa, these may not be as accurate in the setting of low prevalent conditions. We agree with these general guidelines, and we found that looking at the raw counts in a 2 × 2 table often revealed most clearly the patterns of discordance in a particular disease. Until the patterns of concordance for specific diseases are more clearly understood, using summary concordance measures (including prevalence estimates) alone may obscure the underlying patterns and should be avoided. In addition the measures selected for concordance will need to consider the particularities of the disease and population sampled.

We were particularly interested in the relationship between morbidity and agreement. This study is the first to present and explore the relationship of HUI, a validated self-reported measure of overall disease burden, to disease ascertainment. As anticipated, we found for some conditions (myocardial infarction, stroke and heart failure) cases identified by administrative data had higher median HUI scores (thus lower reported morbidity) compared with self-report cases across all diseases. For these conditions health administrative data therefore tended to identify healthier patients than those found through self-report. While the HUI is an overall measure of morbidity, and not a disease-specific measure the severity, it is probable that the severity of underlying diseases relates strongly to overall morbidity. Our finding underscores the need for researchers to consider the clinical significance of cases identified by different data sources.

Stroke and congestive heart failure had the lowest HUI scores, the largest differences in median HUI scores, and poor concordance for the two disease ascertainment methods; while diabetes and hypertension had high HUI scores and high concordance in both median HUI and disease ascertainment. Previous research has, in general, found comorbidity is associated with poorer agreement in ascertainment [10–12]. Our findings confirm that care should be taken in the interpretation of disease estimates in population with high levels of disease burden.

Population based data is a powerful tool for chronic disease surveillance. This study explored agreement between administrative and survey data for ascertainment of seven key chronic diseases. We found that discordance was large for many chronic diseases, particularly disease with low prevalence and that these differences were not easily explained by individual and disease characteristics. In general health administrative data tended to identify patients who were healthier, although we were not able to comment on if their disease specific morbidity was also lower. We find that the accuracy, validity and generalizeability of chronic disease case ascertainment methods depends on the data source used. Researchers should be mindful as to the implication of data source on their results.

EM is a new investigator at the C.T. Lamont Primary Health Care Research Centre and Bruyere Research Institute, clinician scientist at the Department of Family Medicine University of Ottawa Canada and a research fellow at the Institute for Clinical Evaluative Sciences. EG was an analyst at the Institute for Clinical Evaluative Sciences at the time that this research was completed, CB is a Research Coordinator at Institute for Clinical Evaluative Sciences, DM is a Senior Scientist, Ottawa Hospital Research Institute, Adjunct Scientist, Institute for Clinical Evaluative Sciences, Chair in Applied Public Health Sciences, CIHR/PHAC, Associate Professor, University of Ottawa and University of Toronto, and Associate Scientist, C.T. Lamont Primary Health Care Research Centre and Bruyère Research Institute.