Research article
Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Clinical validity of a population database definition of remission in patients with major depression

  • Antoni Sicras-Mainar1Email author,
  • Milagrosa Blanca-Tamayo†2,
  • Laura Gutiérrez-Nicuesa†3,
  • Jordi Salvatella-Pasant†3 and
  • Ruth Navarro-Artieda†4
†Contributed equally
BMC Public Health201010:64

https://doi.org/10.1186/1471-2458-10-64

©  Sicras-Mainar et al; licensee BioMed Central Ltd. 2010

Received: 7 July 2009

Accepted: 11 February 2010

Published: 11 February 2010

Open Peer Review reports

Abstract

Background

Major depression (MD) is one of the most frequent diagnoses in Primary Care. It is a disabling illness that increases the use of health resources. Aim: To describe the concordance between remission according to clinical assessment and remission obtained from the computerized prescription databases of patients with MD in a Spanish population.

Methods

Design: multicenter cross-sectional. The population under study was comprised of people from six primary care facilities, who had a MD episode between January 2003 and March 2007. A specialist in psychiatry assessed a random sample of patient histories and determined whether a certain patient was in remission according to clinical criteria (ICPC-2). Regarding the databases, patients were considered in remission when they did not need further prescriptions of AD for at least 6 months after completing treatment for a new episode. Validity indicators (sensitivity [S], specificity [Sp]) and clinical utility (positive and negative probability ratio [PPR] and [NPR]) were calculated. The concordance index was established using Cohen's kappa coefficient. Significance level was p < 0.05.

Results

133 patient histories were reviewed. The kappa coefficient was 82.8% (confidence intervals [CI] were 95%: 73.1 - 92.6), PPR 9.8% and NPR 0.1%. Allocation discrepancies between both criteria were found in 11 patients. S was 92.5% (CI was 95%: 88.0 - 96.9%) and Sp was 90.6% (CI was 95%: 85.6 - 95.6%), p < 0.001. Reliability analysis: Cronbach's alpha: 90.6% (CI was 95%: 85.6 - 95.6%).

Conclusions

Results show an acceptable level of concordance between remission obtained from the computerized databases and clinical criteria. The major discrepancies were found in diagnostic accuracy.

Background

Major depression (MD) is one of the most frequent diagnoses in Primary Care (PC) and in the general population [1]. It is a disabling illness that alters patients' quality of life and causes an increase in the use of health resources [2–6]. It is important to define the therapeutic objectives for MD [7], and more experts are claiming that reaching remission should be the main objective since residual symptoms prolong psychosocial dysfunction and can cause higher rates of recurrence [8–10]. In this respect, antidepressants (AD) constitute the pillar of pharmacological treatment in order to achieve the sustained remission of symptoms [11, 12].

On the other hand, some prospective naturalistic studies are carried out in order to explore health problems using computerized databases; because key data in these databases are prescriptions or pharmacy claims, and because there is no specific clinical data recorded in these databases it is usually not possible to obtain quality data to measure remission, thus necessitating "approximate definitions". No consensus currently exists regarding a definition of an approximation for remission. However, definitions exist for depression episode: usually, a new episode for depression is defined as having a new antidepressant prescription with no antidepressant prescription in the previous 6 months [12]. From this, it can be derived that if a patient stops treatment and has no new prescription during the following 6 months, he/she has completed a depression episode. If he/she starts an antidepressant after that, it will be a new episode. This can be used in a model for remission.

Nevertheless, this "approximate definition" can cause several problems. On one hand, there may be other motives for discontinuing the prescription other than the patient being in remission (failure to observe treatment, adverse side effects, etc.). On the other hand, persistence of treatment with AD should not always be interpreted as absence of remission, as it is indicated in all patients for 6-9 months after achieving remission, and for a longer period in selected patients at risk of relapse [13].

Thus, it is pertinent to prove the validity of this construct, so as not to fall into classification bias [14], especially when there is no evidence at present in the revised literature. This study is an attempt to complement a previously started line of investigation on remission in patients with MD (results pending publication). The purpose of this study is to describe the validity (reliability, concordance) and the clinical utility of the remission measurement obtained from the direct review of patients' histories (reference criteria) and the one taken from the database registry (by approximation) in MD patients in a Spanish population.

Methods

Design and population under study

The design was cross-sectional and multi-centric. The population under study was comprised of people from six primary care facilities under the management of Badalona Serveis Assistencials SA, which provide health care for some 107,208 people, out of whom 15.9% are over 64. The diagnosis of MD was made according to International Classification of Primary Care (ICPC-2; code P76) [15]. The following scales were used for screening/diagnostics: Goldberg (co-existence of psychiatric morbidity), Hamilton Depression (severity of depression symptoms) and Yesavage Geriatric Depression [11]. Patients were selected according to the following criteria: a) over 17 years old, b) who have initiated a first MD episode with a treatment prescribed between January 2003 and March 2007 (determined for diagnosis by clinical interview of treating clinicians according to the ICPC-2), c) and demonstrate a period of at least six months without depression prior to the MD episode, d) prescription should follow the minimum required treatment criteria [11] (> 60 days of AD treatment after the first prescription) and e) a follow-up of at least 18 months (12-month study period and 6-month follow-up to assign study sub-group). Two sub-groups were considered: patients in remission and patients not in remission.

Definition of remission by approximation and according to reference criteria

Patients were considered to be in remission when they no longer required new prescriptions of AD for at least six months after completing treatment of the first episode; patients who could only interrupt treatment for a period less than six months were not considered in sustained remission [12]. As reference criteria, remission was determined based on the assessment of the database clinical course text by a psychiatry specialist. The sample was selected through simple random sampling stratified for age and gender. Sample size was calculated assuming an expected remission prevalence of 54%, a random error of 5% (bilateral) and an accuracy of 8.5%.

Validity and utility measure indicators

Validity indicators were Sensitivity [S], specificity [Sp], positive and negative predictive value [PPV] and [NPV]) and clinical utility (positive and negative probability ratio [PPR] and [NPR]. S is the probability of obtaining a positive result when the subject meets the approximation definition. Sp indicates the probability of obtaining a negative result when the subject does not meet the approximation definition. PPV is the probability of meeting the reference criteria when the approximate remission variable is positive. NPV is the probability of not meeting the reference criteria when remission by approximation is negative. The probability ratios of positive or negative likelihood-ratios indicate up to what point a certain result in a diagnostic test will increase or diminish pre-test probability [16].

Confidentiality of the information and statistical analysis

Data confidentiality was respected at all times, according to the Personal Data Protection Act (Ley de protección de Datos de Carácter Personal [LOPD]); this study was approved by the Ethics Committee of Gol i Gurina Foundation. Concordance between both remission measurements was evaluated, itemized by systematic error and random error. Concordance index was calculated using Cohen's kappa coefficient, correlation coefficient was calculated for random error and McNemar test for systematic error [17]. Reliability analysis was established using Cronbach's alpha. Confidence intervals [CI] of 95% were calculated. SPSSWIN version 12 was used, establishing a statistic significance of p < 0.05.

Results

4,572 subjects, according to the database criteria, complied with the selection (new episode of depression), of whom 54.6% (CI of 95%: 53.2 - 56.0%) were considered to be in remission. 133 patient histories were reviewed according to clinical criteria. Concordance between remission by approximation and remission according to reference criteria is detailed in table 1. The weighted kappa concordance index obtained was 82.8% (CI of 95%: 73.1 - 92.6), PPR 9.8% and NPR 0.1%. Eleven patients showed discrepancies between remission by approximation and remission according to reference criteria. In five cases (false positives), review of the clinical course revealed they did not meet the clinical criteria for remission, the cause of this discrepancy was either incorrect initial diagnostic classification or on later occurrence of stress factors (dysthymia: 2; recurrent depression: 1; adaptive disorder: 1; and vital occurrence: 1). Six patients (false negatives) were in remission by clinical course assessment but not approximation criteria. These patients were wrongly classified (dysthymia: 4; adaptive disorder: 1; recurrent depression: 1) and required treatment to prevent recurrence for a longer period of time.
Table 1

Validity of remission by approximation in major depression

Validation Method

Reference Criteria

Remission by Approximation

Clinical Histories

N = 133

Positive

Negative

Positive

74

5

Negative

6

48

Statistics

Value (%)

95% CI

Validity of the measurement

  

   Sensitivity

92.5

88.0 - 96.9

   Specificity

90.6

85.6 - 95.6

   Positive predictive value

93.7

89.6 - 97.8

   Negative predictive value

88.9

83.6 - 94.2

   False positives

7.5

3.0 - 11.9

   False negatives

9.4

4.4 - 14.4

Reliability

  

   Cronbach's alpha

90.6

85.6 - 95.6

   Area under the curve

91.2

86.5 - 96.1

Concordance

  

   McNemar test

58.1

49.7 - 66.5

   Pearson correlation

82.8

73.1 - 92.5

   Weighted kappa (Cohen)

82.8

73.1 - 92.6

Clinical utility

  

   Positive Probability Ratio

9.8

---

   Negative Probability Ratio

0.1

---

Reference criteria: Review of patients' histories. Significance: p < 0.001 in all cases. CI: confidence intervals

Discussion

Our study deals with the clinical validity of the measurement of remission in patients with major depression, as obtained from population databases. It is worth pointing out that the organization of care, through territorial-based assignment of teams and the increasing computerization of its centres provides a suitable environment to carry out these types of studies in a clinical practice environment [18, 19]. Thus, the quality of the measurements is a basic aspect in achieving an efficient health system, and we must be sure in any kind of biomedical research that measurement error is reasonably small. In regulated environments, such as clinical testing in the development of pharmaceuticals, data quality and especially measurement procedures require attention for reasons related to both ethics and efficiency [17].

The results of our study reveal an acceptable degree of concordance between the remission by approximation and the variable obtained from patient history reviews (reference criteria). The main cause of discrepancy between them is diagnostic accuracy by the PC team. In this respect, several authors have affirmed the existing variability between the PC diagnosis and the one by mental health specialists [20–22]. There seems to be certain difficulty in identifying affective and adaptive disorders, which has an impact on remission measurement. There are several possible motives affecting MD diagnosis. On one hand, it may be affected by shortcomings in the training of the professionals, health care pressure in their consults and the brief amount of time they can devote to each patient.

Furthermore, it can also affect the classification system used (ICPC-2), which can condition the diagnosis of these patients. In this regard, it must be taken into consideration that these differences between both disorders make differential diagnoses difficult, as PC doctors usually tend to view disorders from a dimensional rather than a categorical perspective, probably because it best fits the needs of their consultation. Nevertheless, there are studies that reveal a certain tendency of over-diagnosis of depression disorders, while others point out an under-diagnosis of MD and anxiety in PC [23, 24]. The interpretation of the measurement of remission in our study may be in agreement with the interpretation of probability coefficients (PPR and NPR) as the measurement of clinical utility. Thus, the measurement of remission by approximation can generate moderate changes from pre-test to post-test probability (reference criteria).

Possible limitations in the study demand caution in generalization of results. The study presented is framed within the studies known as technique validation in clinical effectiveness conditions; therefore, the bias assumed in research is typical of a study with a cross-sectional model, oriented towards validation of diagnostic criteria. Moreover, sample size can have a high random component for precision in measurement, and the study was carried out based on a specialist's diagnostic criteria; therefore, the results obtained may be influenced by the professional's characteristics.

Conclusions

Results show acceptable concordance in the remission obtained from the computerized databases. The largest discrepancies are in diagnostic accuracy.

Notes

Abbreviations

MD: 

Major Depression

PC: 

Primary Care

AD: 

Antidepressants

ICPC-2: 

International Classification of Primary Care

S: 

Sensitivity

Sp: 

Specificity

PPV: 

Predictive Positive Value

NPV: 

Negative Predictive Value

PPR: 

Positive Probability Ratio

NPR: 

Negative Probability Ratio.

Declarations

Acknowledgements

Thanks to the different professionals of the centres for their constant input of data on a daily basis. Without their contribution, this study could not have been conducted

Authors’ Affiliations

(1)
Directorate of Planning, Badalona Serveis Assistencials SA, Badalona, Barcelona. Spain
(2)
Psychiatric Services, Badalona Serveis Assistencials SA, Badalona
(3)
Medical Department, Lundbeck España SA
(4)
Medical Documentation, Germans Trias i Pujol Hospital, Badalona

References

  1. ESEMeD, Haro JM, Palacin C, Vilagut G, Martinez M, Bernal M, Luque I, Codony M, Dolz M, Alonso J y el Grupo ESEMeD-España: Prevalencia de los trastornos mentales y factores asociados: resultados del estudio ESEMeD-España. Med Clin (Barc). 2006, 126: 445-51. 10.1157/13086324.View ArticleGoogle Scholar
  2. Sobocki P, Lekander I, Borgstrom F, Strom O, Runeson B: The economic burden of depression in Sweden from 1997 to 2005. Eur Psychiatry. 2007, 22: 146-52. 10.1016/j.eurpsy.2006.10.006.View ArticlePubMedGoogle Scholar
  3. Shvartzman P, Weiner Z, Vardy D, Friger M, Sherf M, Biderman A: Health services utilization by depressive patients identified by the MINI questionnaire in a primary care setting. Scand J Prim Health Care. 2005, 23: 18-25. 10.1080/02813430510018383.View ArticlePubMedGoogle Scholar
  4. Sobocki P, Ekman M, Agren H, Krakau I, Runeson B, Martensson B, Jonsson B: Health Related Quality of Life measured with EQ-5D in patients treated for depression in primary care. Value Heath. 2007, 10: 154-60.Google Scholar
  5. Goetzel RZ, Hawkins K, Ozminkowski RJ, Wang S: The health and productivity cost burden of the "top 10" physical and mental health conditions affecting six large U.S. employers in 1999. J Occup Environ Med. 2003, 45: 5-14. 10.1097/00043764-200301000-00007.View ArticlePubMedGoogle Scholar
  6. Sicras Mainar A, Navarro Artieda R, Rejas Gutiérrez J, Blanca Tamayo M, Serrat Tarrés J, Llopart López JR: Comorbilidad y coste ambulatorio asociado a los pacientes con trastornos depresivos en un ámbito poblacional. Farm Hosp. 2007, 31: 101-5. 10.1016/S1130-6343(07)75720-3.View ArticlePubMedGoogle Scholar
  7. Keller MB: Past, present and future directions for defining optimal treatment outcome in depression: remission and beyond. JAMA. 2003, 289: 3152-60. 10.1001/jama.289.23.3152.View ArticlePubMedGoogle Scholar
  8. Ansseau M, Demyttenaere K, Heyrman J, Migeotte A, Leyman S, Mignon A: Objective: remission of depression in primary care The Oreon Study. Eur Neuropsychopharmacol. 2009, 19: 169-76. 10.1016/j.euroneuro.2008.10.003.View ArticlePubMedGoogle Scholar
  9. Sobocki P, Ekman M, Agren H, Runeson B, Jönsson B: The mission is remission: health economic consequences of achieving full remission with antidepressant treatment for depression. Int J Clin Pract. 2006, 60: 791-8. 10.1111/j.1742-1241.2006.00997.x.View ArticlePubMedGoogle Scholar
  10. Zimmernan M, McGlinchey JB, Posternak MA, Friedman M, Attiullah N, Boerescu D: How should remission from depression be defined?. The depressed patient's perspective. Am J Psychiatry. 2006, 163: 148-50. 10.1176/appi.ajp.163.1.148.View ArticleGoogle Scholar
  11. National Institute for Clinical Excellence: Depression: management of depression in primary and secondary care. Clinical Guideline 23. 2004. Amended. 2006, London: NICEGoogle Scholar
  12. Wade AG, Despiégel N, Danchenko N, Saragoussi D, Verpillat P: How does remission in major depressive disorder impact resource use in UK does: A General Practitioner Research Database (GPRD) based study. 23rd International Conference on Pharmacoepidemiology & Therapeutic Risk Management (ISPE), Québec City Canada, 19-22. 2007, AugustGoogle Scholar
  13. Kupfer DJ: Long-term treatment of depression. Clin Psychiatry. 1991, 52 (Suppl): 28-34.Google Scholar
  14. White E: Design and interpretation of studies of differential exposure measurement error. Am J Epidemiol. 2003, 157: 380-7. 10.1093/aje/kwf203.View ArticlePubMedGoogle Scholar
  15. Lamberts H, Wood M, Hofmans-Okkes IM, eds: The International Classification of Primary Care in the European Community. With a multi-language layer. 1993, Oxford: Oxford University PressGoogle Scholar
  16. MacMahon B, Trichopoulos D: Epidemiology: Principles and Methods. 1996, Boston: Lippincott Williams & Wilkins, 2Google Scholar
  17. Carrasco JL, Jover L: Métodos estadísticos para evaluar la concordancia. Med Clin (Barc). 2004, 122 (Supl 1): 28-34. 10.1157/13057543.View ArticleGoogle Scholar
  18. Campillo-Artero C: Fracturas en la informatización de los servicios de salud. Gac Sanit. 2008, 22: 371-7. 10.1157/13125361.View ArticlePubMedGoogle Scholar
  19. Ludwick DA, Doucette J: Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. Int J Med Inform. 2009, 78: 22-31. 10.1016/j.ijmedinf.2008.06.005.View ArticlePubMedGoogle Scholar
  20. Landa-González N, Goñi-Sarriés A, García-de-Jalón-Aramayo E, Lizasoain-Urra E: Concordancia en el diagnóstico entre atención primaria y salud mental. Aten Primaria. 2008, 40: 285-9. 10.1157/13123680.View ArticlePubMedGoogle Scholar
  21. Herrán A, López-Lanza JR, Ganzo H, Cadiñanos A, Díez-Manrique JE, Vázquez-Barquero JL: Derivación de los pacientes con enfermedad mental desde Atención Primaria a Salud Mental. Actas Esp Psiquiatr. 2000, 28: 13-21.PubMedGoogle Scholar
  22. Klinkman MS, Coyne JC, Gallo S, Schwenk TL: False positives, falses negative, and the validity of the diagnosis of major depression in primary care. Arch Fam Med. 1998, 7: 451-61. 10.1001/archfami.7.5.451.View ArticlePubMedGoogle Scholar
  23. Thompson C, Ostler K, Baker N, Kinmonth A: Dimensional perspective on the recognition of depressive symptoms in primary care. Br J Psychiatry. 2001, 179: 317-23. 10.1192/bjp.179.4.317.View ArticlePubMedGoogle Scholar
  24. Gilbody S, Richards D, Barkham M: Diagnosing depression in primary care using self-completed instruments: UK validation of PHQ-9 and CORE-OM. Br J Gen Pract. 2007, 57: 650-2.PubMedPubMed CentralGoogle Scholar

    25. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/10/64/prepub

Copyright

© Sicras-Mainar et al; licensee BioMed Central Ltd. 2010

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Advertisement