Validation of metabolic syndrome using medical records in the SUN cohort
© Barrio-Lopez et al; licensee BioMed Central Ltd. 2011
Received: 31 July 2011
Accepted: 15 November 2011
Published: 15 November 2011
The objective of this study was to evaluate the validity of self reported criteria of Metabolic Syndrome (MS) in the SUN (Seguimiento Universidad de Navarra) cohort using their medical records as the gold standard.
We selected 336 participants and we obtained MS related data according to Adult Treatment Panel III (ATP III) and International Diabetes Federation (IDF). Then we compared information on the self reported diagnosis of MS and MS diagnosed in their medical records. We calculated the proportion of confirmed MS, the proportion of confirmed non-MS and the intraclass correlation coefficients for each component of the MS.
From those 336 selected participants, we obtained sufficient data in 172 participants to confirm or reject MS using ATP III criteria. The proportion of confirmed MS was 91.2% (95% CI: 80.7- 97.1) and the proportion of confirmed non-MS was 92.2% (95% CI: 85.7-96.4) using ATP III criteria. The proportion of confirmed MS using IDF criteria was 100% (95% CI: 87.2-100) and the proportion of confirmed non-MS was 97.1% (95% CI: 85.1-99.9). Kappa Index was 0.82 in the group diagnosed by ATP III criteria and 0.97 in the group diagnosed by IDF criteria. Intraclass correlation coefficients for the different component of MS were: 0.93 (IC 95%:0.91- 0.95) for BMI; 0.96 (IC 95%: 0.93-0.98) for waist circumference; 0.75 (IC 95%: 0.66-0.82) for fasting glucose; 0.50 (IC 95%:0.35-0.639) for HDL cholesterol; 0.78 (IC 95%: 0.70-0.84) for triglycerides; 0.49 (IC 95%:0.34-0.61) for systolic blood pressure and 0.55 (IC 95%: 0.41-0.65) for diastolic blood pressure.
Self-reported MS based on self reported components of the SM in a Spanish cohort of university graduates was sufficiently valid as to be used in epidemiological studies.
Visceral obesity, insulin resistance, high blood pressure, and dyslipidemia are some of the characteristic abnormalities of the metabolic syndrome (MS). The association of these alterations confers a high risk of atherosclerosis and cardiovascular disease, more than the sum of the individual risks . It is estimated that MS, by its consequences, including type 2 diabetes and cardiovascular disease, is the first cause of death in women and the second cause of death in men in our society .
A unified definition of MS is needed to detect this entity in the general population [3–5]. The high prevalence (20 to 25% of the world population [6, 7]), the dramatic increase over last years  and socio-sanitary impact in developed countries is resulting in a global epidemic of MS . In Spain the prevalence of MS varies between 11% and 24%, depending on the different studies [10–14].
An epidemiological questionnaire is a useful tool to study MS in the population. The questionnaire is useful in assessing the characteristics and distribution of MS in the population and then analyzing epidemiological results and drawing conclusions. Sometimes, the only feasible way of studying these populations is to rely on self-reported data of the outcome. As a consequence, it is of the utmost importance to assess the validity of that information. Self-reported diagnosis of MS could not be useful in populations with low cultural level but could be enough valid in a cohort of university graduates.
Our objective was to assess the validity of self-reported diagnosis of MS in a sample of the participants in the Seguimiento Universidad de Navarra (SUN, University of Navarra Follow-up) study, a cohort study in Spain. In this sample we compared the self-reported data in the questionnaire with the data obtained from their medical records as the gold standard.
Material and methods
The SUN Project is a dynamic cohort of university graduates, recruited and followed up through mailed questionnaires. Beginning in December 1999 , the main objective of the study was to assess the association between a Mediterranean dietary pattern and the risk of cardiovascular disease, diabetes, obesity, MS, and other chronic disease. To select the volunteers, all graduates of the University of Navarra, registered nurses from some Spanish provinces, and university graduates from other colleges and associations received a mailed questionnaire and a letter of invitation to participate in the SUN study. A response to the initial questionnaire was considered as informed consent to participate in the study. Then, every 2 years, a follow-up questionnaire is mailed to each participant, gathering information about new medical diagnoses and changes in exposures of interest. Part of this information is self-reported and participants obtain it from their blood test and medical check-ups that they routinely undergo in Spain by their occupational physicians (blood pressure, levels of triglycerides, fasting glucose, high-density lipoprotein, cholesterol, etc.). The questionnaire includes questions about previous and new diagnoses and changes in medications. The diagnosis of MS was not present in the questionnaire. We preferred to ask separately about each of the MS components to avoid an underestimation and to be able to obtain more information (supplementary data file - Additional file 1).
The SUN Study was approved by the Institutional Review Board of the University of Navarra.
Validation of the study
All participants with MS self reported criteria in the SUN questionnaire and with medical records in the Clinica Universidad de Navarra (CUN) were selected (n = 132). Another random sample of 204 participants with no MS criteria in the questionnaire and medical record in the CUN was taken. We calculated this sample size for a single proportion  (for a confidence interval of 95%) with a desired precision of the confidence interval of 0.04. Among 336 selected participants, we obtained sufficient data from the medical records to confirm or reject MS in 172 participants. Medical record information and questionnaire information was obtained by a different investigator. Thus, the researcher who obtained information from the medical record (MTB-L) did not know if each participant had reported MS or not. Questionnaires were answered in 2006 and the information from the medical record was obtained between 2005 and 2006.
To conduct the validation we compared the information from the questionnaire with the information from the medical record. The ATP III  (Adult Treatment Panel III) criteria were applied for the MS classification in the questionnaire and diagnosis in the medical report. In the sensitivity analysis, IDF  (International Diabetes Federation) criteria were applied too.
For ethical reasons, the identity of the participants was masked. This was the process: a first investigator (C de la F) selected a list with only an identification number, the name and the surname of each participant (self-reported MS criteria were excluded), a second investigator (MTB-L) obtained the medical information about MS from the participants medical record, a third investigator (MB-R) did the analysis between the serf-reported data and the medical history data only with the participant identification number (anonymous).
In this way a) any investigator could not know both information (self-reported and measured information) and the participant identity at the same time b) the investigator who obtained the information from the medical record (MTB-L) was blind with respect to self-reported information; c) the investigator who performed the statistical analysis never knew the participants identity; d) in the moment of answer the questionnaire, participants did not know that we will obtain information from their medical records. Laboratory test results from the medical record and self-reported information were both determined after fasting. Blood pressure, weight and height were measured routinely in the clinic.
Definition of self-reported MS and confirmed MS
We considered that a participant reported MS if the criteria of MS according to ATP III criteria were present in the questionnaire. On the other hand, we classified a participant as confirmed MS if he or she presented criteria of MS according to ATP III criteria in his/her medical record.
In participants who had normal values of blood pressure, HDL-cholesterol, triglycerides or fasting glycaemia, the medical treatment was considered in order to classify correctly the participants and avoid effect of insulin, lipid-lowering or antihypertensive drugs. Then the same analyses were made using the IDF  criteria.
The proportion of subjects with confirmed MS (Positive predictive value, PPV) was calculated as the number of participants who reported MS and this diagnosis was confirmed by the medical history (true positives) divided by the total number of people who reported MS (total positives). In the same way, the proportion of confirmed non MS (Negative predictive value, NPV) was calculated as the number of participants who reported no MS and this diagnosis was confirmed by the medical history (true negatives) divided by the total number of people who reported no MS (total negatives).
The different characteristics (age, sex, body mass index, university career) between the proportion of cases with MS and without confirmed MS were studied using the Fisher's exact test .
The coincidence between the self-reported MS and confirmed MS was evaluated by the Kappa index. This index was calculated with ATP III and IDF criteria.
To calculate sensitivity and specificity of a self-reported diagnose of MS, the expected distribution of true and false positive and negative cases in the sample were estimated through sampling fractions and observed percentage of confirmed diagnoses to estimate the prevalence of MS in our population. We estimated the prevalence of MS in our population as 20%, which is similar to the prevalence reported by descriptive studies in Spain [10–13]. The Spearman and the Pearson correlation coefficients and intraclass correlation coefficients were calculated for continuous variables.
Characteristics of participants in the validation study of the MS according to the ATP-III criteria
Non self-reported MS
Doctorate/Master 11 (19.3%)
Validity of self-reported MS (Pearson's chi-square) according to relevant variables and ATP-III and IDF criteria
Presence of MS
Absence of MS
Presence of MS
Absence of MS
% confirmed (95% CI)
% confirmed (95% CI)
% confirmed (95% CI)
% confirmed (95% CI)
100 (n = 172)
91.2 (80.7- 97.1)
100 (n = 62)
100 (87.2- 100)
97.1 (85.1- 99.9)
84.2 (72.1- 92.5)
78.6 (49.2- 95.3)
97.4 (86.2- 99.9)
89.2 (85.6- 91.9)
96.2 (93.6- 96.4)
100 (71.5- 100)
100 (88.4- 100)
95.0 (93.3- 96.3)
100 (54.1- 100)
80.0 (28.4- 99.4)
90.9 (87.9- 93.2)
93.1 (90.8- 94.8)
100 (54.1- 100)
95.5 (93.9- 96.6)
86.8 (82.6- 90.1)
100 (66.4- 100)
100 (76.8- 100)
87.5 (83.4- 90.6)
95.8 (94.4- 96.9)
100 (73.5- 100)
95.2 (76.2- 99.8)
92.5 (79.6- 98.4)
87.8 (75.2- 95.4)
100 (83.9- 100)
75 (19.4- 99.4)
84.6 (54.5- 98.1)
94.0 (83.4- 98.7)
100 (54.1- 100)
100 (81.5- 100)
Based on the assumption that the cohort prevalence of MS was 20%, we estimated a sensitivity of 66% and a specificity of 98% to detect MS using self-reported data. The Kappa index coefficient between serf-reported MS and confirmed MS was 0.82 (95% CI 0.73-0.91%) using ATP III criteria and 0.97 (95% IC: 0.90-1) using IDF criteria.
Correlation between self-reported and measured MS variables
Intraclass Correlation Coefficient
Our findings showed an acceptable degree of confirmation of self-reported diagnoses of MS. Sensitivity was only moderate but specificity and predictive values were very high. In addition, the correlation between the component variables of MS was fair enough to be used in epidemiological studies.
In general, data recorded of MS like a binary expression (presence or absence of MS) has its limitations, and it is better to obtain information of the component elements of the diagnostic criteria of MS (HDL-cholesterol, LDL-cholesterol, fasted glycaemia, blood pressure and waist circumference) to avoid an underestimation. In the SHIELD  study (Study to Help Improve Early evaluation and management of risk factors Leading to Diabetes) prevalence of self-reported MS was studied in 211.097 participants. SHIELD respondents were asked if they had ever been told by a doctor, nurse or other health professional that they had MS or syndrome X. The proportion of respondents reporting a diagnosis of MS was estimated as the prevalence of MS in SHIELD. Only 0.6% of the total population reported a MS diagnosis. In contrast, NHANES  (n = 10,780) data using clinical and laboratory criteria indicated a MS prevalence of 25.9% in the adult population. The SHIELD respondents and NHANES participants belong to similar populations according to age, sex and ATP III criteria.
SHIELD study evidenced the lack of knowledge about MS in the general population and especially in high risk patients (obese, diabetic, hypertensive). In these risk populations the self-reported MS was similar to the general population. Therefore in our study we decided to ask for MS criteria to get more information and avoid an underestimation of the true prevalence.
In the SUN cohort we have reported a previous validation of the MS components . In the present study we wanted to validate not only the MS components, but also the MS according to the two more important criteria (IDF and ATP III). We collected continuous data of blood pressure, HDL-cholesterol, triglycerides, weight, height, and waist circumference. From this data MS was defined according with ATP III and IDF criteria.
Positive predictive value (91.2%) and negative predictive value (92.2%) are an evidence of the good validity of self-reported questionnaire to detect the presence or the absence of MS in this population of university graduates.
Correlation between components of MS was good too. As expected, blood pressure was the variable with worse correlation (r = 0.44 p < 0.001 for systolic blood pressure, r = 0.5 p < 0.001 for diastolic blood pressure). This phenomenon may be due to higher within individual day-to-day biological variability. The best correlation was observed for weight and height. In the other variables the correlation was good (r = 0.68 p < 0.001 for fasting glucose, r = 0.5 p < 0.001 for HDL-cholesterol, r = 0.69 p < 0.001 for triglyceride, r = 0.95, p < 0.001 for BMI).
There are several validation studies relating to the waist and hip circumferences [21, 22]. They show a sensitivity of 100% and a specificity of 95.7% for men and a sensitivity of 95.1% and a specificity of 97.2% for women to classify in two groups of abdominal obesity when it is compared to the gold standard (measurement made by an expert) .
Another study showed a trend towards underestimation of waist circumference in women especially in the group who have a higher obesity abdominal . In our study we found the limitation of poor measurement of waist and hip circumference in clinical practice. However we found a good correlation between self-reported waist circumference and waist circumference from medical records (r = 0.96 p < 0.001).
About blood pressure, we should take into account that there were occasional large fluctuations due to biological variability. In this context, the participants reported a single measurement and we collected only other measurement from medical history. In no case we had several measurements and in many cases these measurements could be influenced by white coat phenomenon or by medication. This last situation has been taken into account considering as hypertensive those patients diagnosed with hypertension and those taking antihypertensive drugs to avoid misclassification.
Isolated arterial hypertension has been previously validated in a subsample of the SUN cohort with enough validity .
We analyzed the differences between MS and no-MS and we observed more MS in men than in women and an increased prevalence with age. No differences were observed according to university degree. It is possible that in the SUN cohort there are less MS incidence than in the general Spanish population because it is a young cohort (mean age was 37.5 years).
We only could diagnose or rule out MS with information found in medical records in 172 of the 336 participants initially selected. This was a possible limitation of our study because of the existence of a potential selection bias. However, the sociodemographic characteristics and self-reported averaged values for the components of MS were similar in both groups with no statistically significant differences.
We accept that the sample was not representative of the general population because it is a cohort of university graduates and this could be considered as a limitation in our study. It is possible that university graduates have better knowledge about MS and about health in general that general population. However, this could be considered also as a strength of the study because our aim was to validate self-reported diagnosis of MS and its components among highly educated subjects (i.e. in this cohort). This would allow us to use the self-reported information from these highly educated participants to conduct studies of association between various risk factors and this entity. In this context, we should note that it is not necessary for the sample to be representative, in the statistical sense of random sampling, of the general population. Indeed, the sample of this 336 selected participants should be representative of all the SUN cohort participants. This situation contributes to increase the internal validity of our results because we are obtaining self-reported information of high quality. Moreover, uniform education of the cohort reduces the potential confounding effect by socioeconomic factors. On the other hand, a strong point of our design is that participants were unaware that we could get information from their medical records after answering their questionnaire. It thus prevents them to overstate or be artificially more precise in their reported data.
Self-reported MS based on self-reported components of the SM in a Spanish cohort of university graduates was sufficiently valid as to be used in epidemiological studies.
Seguimiento Universidad de Navarra, University of Navarra Follow-up Study
Clinica Universidad de Navarra
body mass index
Systolic blood pressure
Diastolic blood pressure.
The authors thank the collaboration of the participants in the SUN Study. The SUN Study has received funding from the Spanish Ministry of Health and Ministry of Science and Innovation (Grants PI01/0619, PI030678, PI040233, PI042241, PI050976, PI070240, PI070312, PI081943, PI080819, PI1002658, PI1002293, RD06/0045, and G03/140), the Navarra Regional Government (36/2001, 43/2002, 41/2005, 36/2008) and the University of Navarra.
- Gami AS, Witt BJ, Howard DE, Erwin PJ, Gami LA, Somers VK, Montori VM: Metabolic syndrome and risk of incident cardiovascular events and death: a systematic review and meta-analysis of longitudinal studies. J Am Coll Cardiol. 2007, 49: 403-414. 10.1016/j.jacc.2006.09.032.View ArticlePubMedGoogle Scholar
- Lakka HM, Laaksonen DE, Lakka TA, Niskanen LK, Kumpusalo E, Tuomilehto J: The metabolic syndrome and total and cardiovascular disease mortality in middle-aged men. JAMA. 2002, 288: 2709e16-View ArticleGoogle Scholar
- National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III): Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults. Circulation. 2002, 106 (25): 3143-421.Google Scholar
- Alberti KG, Zimmet P, Shaw J, IDF Epidemiology Task Force Consensus Group: The metabolic syndrome: a new worldwide definition. Lancet. 2005, 366: 1059-1062.01. 10.1016/S0140-6736(05)67402-8. 285: 2486-2497View ArticlePubMedGoogle Scholar
- Reinehr T, de Sousa G, Toschke AM, Andler W: Vestische Kinder- und Jugendklinik, University of Witten/Herdecke, Dr. F. Steiner Str. 5, 45711 Datteln, Germany. Comparison of metabolic syndrome prevalence using eight different definitions: a critical approach. Arch Dis Child. 2007, 92: 1067-1072. 10.1136/adc.2006.104588.View ArticlePubMedPubMed CentralGoogle Scholar
- Cameron AJ, Shaw JE, Zimmet PZ: The metabolic syndrome: prevalence in worldwide populations. Endocrinol Metab Clin North Am. 2004, 33: 351-375. 10.1016/j.ecl.2004.03.005.View ArticlePubMedGoogle Scholar
- Ford ES, Giles WH, Mokdad AH: Increasing prevalence of the metabolic syndrome among US adults. Diabetes Care. 2004, 27: 2444e9-Google Scholar
- Buckland GG, Salas-Salvadó J, Serra-Majem L, Castell C, Cabré J, Salleras-Sanmartí L: Increase in metabolic syndrome as defined by ATPIII from 1992-1993 to 2002-2003 in a Mediterranean population. Nutr Rev. 2009, 67 (Suppl 1): S117-S125.View ArticlePubMedGoogle Scholar
- Eckel RH, Grundy S, Zimmet P: The metabolic syndrome. Lancet. 2005, 365: 1415-1428. 10.1016/S0140-6736(05)66378-7.View ArticlePubMedGoogle Scholar
- Martínez-Larrad MT, Fernández-Pérez C, González-Sánchez JL, López A, Fernández-Alvarez J, Riviriego J, Serrano-Ríos M: Prevalencia del síndrome metabólico (criterios ATP III). Estudio de base poblacional realizado en áreas rural y urbana de la provincia de Segovia (España). Med Clin (Barc). 2005, 125: 481-486. 10.1157/13080210.View ArticleGoogle Scholar
- Boronat M, Chirino R, Varillas VF, Saavedra P, Marrero D, Fábregas M, Nóvoa FJ: Prevalence of the metabolic syndrome in the island of Gran Canaria: comparison of three major diagnostic proposals. Diabet Med. 2005, 22: 1751-1756. 10.1111/j.1464-5491.2005.01745.x.View ArticlePubMedGoogle Scholar
- Álvarez EE, Ribas L, Serra L: Prevalencia del síndrome metabólico en la población de la Comunidad Canaria. Med Clin (Barc). 2003, 120: 172-174. 10.1157/13043145.View ArticleGoogle Scholar
- Alegría E, Cordero A, Laclaustra M, Grima A, León M, Casasnovas JA, Luengo E, del Río A, Ferreira I: Investigadores del registro MESYAS. Prevalence of metabolic syndrome in the Spanish working population: MESYAS registry. Rev Esp Cardiol. 2005, 58: 797-806. 10.1157/13077231.View ArticlePubMedGoogle Scholar
- Palma Gámiz JL, Conget Donlo I, Bertomeu González V, Ascaso Gimilio JF, González Juanatey JR, Alegría Ezquerra E, Conthe Gutiérrez P, de la Calle Blasco H, Serrano Ríos M: Prevalence of the metabolic syndrome in Spanish patients with established cardiovascular disease: CLYDIA study. Med Clin (Barc). 2007, 128: 407-413. 10.1157/13100339.View ArticleGoogle Scholar
- Martínez-González MA, Sanchez-Villegas A, De Irala J, Marti A, Martínez JA: Mediterranean diet and stroke: objectives and design of the SUN project. Seguimiento Universidad de Navarra. Nutr Neurosci. 2002, 5: 65-73. 10.1080/10284150290007047.View ArticlePubMedGoogle Scholar
- Harvey M: Intuitive biostatistics. 1995, N.Y.: Oxford University Press, 363-366.Google Scholar
- Daniel WW: The chi-square distribution and the análisis of frequencies. Biostatistics: A foundation for analysis in the health sciences. 2005, Wiley, 629-631. 8Google Scholar
- Lewis SJ, Rodbard HW, Fox KM, Grandy S, SHIELD Study Group: Self-reported prevalence and awareness of metabolic syndrome: findings from SHIELD. Int J Clin Pract. 2008, 62 (8): 1168-1176. 10.1111/j.1742-1241.2008.01770.x. Epub 2008 Apr 29View ArticlePubMedPubMed CentralGoogle Scholar
- Mozumdar A, : Persistent increase of prevalence of metabolic syndrome among U.S. adults: NHANES III to NHANES 1999-2006. Diabetes Care. 2011, 34: 216-219. 10.2337/dc10-0879. [http://www.ncbi.nlm.nih.gov/pubmed?term=20889854]View ArticlePubMedGoogle Scholar
- Fernández-Montero A, Beunza JJ, Bes-Rastrollo M, Barrio MT, de la Fuente-Arrillaga C, Moreno-Galarraga L, Martínez-González MA: Validity of self-reported metabolic syndrome components in a cohort study. Gac Sanit. 2011, 25: 303-307. 10.1016/j.gaceta.2011.02.009.View ArticlePubMedGoogle Scholar
- Han TS, Lean ME: Self-reported waist circumference compared with the 'Waist Watcher' tape-measure to identify individuals at increased health risk through intra-abdominal fat accumulation. Br J Nutr. 1998, 80: 81-88. 10.1017/S0007114598001809.View ArticlePubMedGoogle Scholar
- Hall TR, Young TB: A validation study of body fat distribution as determined by self-measurement of waist and hip circumference. Int J Obes. 1989, 13: 801-807.PubMedGoogle Scholar
- Alonso A, Beunza JJ, Delgado-Rodríguez M, Martínez-González MA: Validation of self reported diagnosis of hypertension in a cohort of university graduates in Spain. BMC Public Health. 2005, 5: 94-10.1186/1471-2458-5-94.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/11/867/prepub