Skip to main content

Are risk estimates biased in follow-up studies of psychosocial factors with low base-line participation?



Low participation in population-based follow-up studies addressing psychosocial risk factors may cause biased estimation of health risk but the issue has seldom been examined. We compared risk estimates for selected health outcomes among respondents and the entire source population.


In a Danish cohort study of associations between psychosocial characteristics of the work environment and mental health, the source population of public service workers comprised 10,036 employees in 502 work units of which 4,489 participated (participation rate 45%). Data on the psychosocial work environment were obtained for each work unit by calculating the average of the employee self-reports. The average values were assigned all employees and non-respondent at the work unit. Outcome data on sick leave and prescription of antidepressant medication during the follow-up period (1.4.2007-31.12.2008) was obtained by linkage to national registries.


Respondents differed at baseline from non-respondents by gender, age, employment status, sick leave and hospitalization for affective disorders. However, risk estimates for sick leave and prescription of antidepressant medication, during follow-up, based on the subset of participants, did only differ marginally from risk estimates based upon the entire population.


We found no indications that low participation at baseline distorts the estimates of associations between the work unit level of psychosocial work environment and mental health outcomes during follow-up. These results may not be valid for other exposures or outcomes.

Peer Review reports


Participation in population-based studies addressing determinants of health outcomes have declined in past decades in several countries [13]. This is also the case for studies of psychosocial work factors. Low participation in cross-sectional studies obviously may cause selection bias if participation is related to rating of the psychosocial work environment and the outcome. In prospective follow-up studies, however, the risk of selection bias has been considered low because participation cannot depend on future outcomes. Although direct links between participation, psychosocial work environment and health outcomes are not an issue, participation might be related to gender, age, personality, occupational and social factors that may be related to exposure as well as later health outcomes [4]. Therefore the possibility of biased risk estimates in prospective follow-up studies with low participation rate cannot be ignored. So far only few studies have examined this issue [58]. Although reported findings in general are reassuring indicating that risk estimates among respondents do not differ substantially from risk estimates based upon the entire population including non-respondents, such findings can hardly be generalized. Low participation may have causes that differ from one setting to another. The objective of this paper is to examine bias by non-participation in a prospective cohort study addressing occupational stressors and mental health. We compared risk ratios for sick leave and prescription of anti-depressive medication redeemed at pharmacies according to established and potential determinants among respondents and the entire source population including non-respondents.


The PRISME study

The Danish PRISME cohort study was designed for purposes of prospective studies of job related psychosocial determinants of major depression and other common mental disorders [9]. The cohort was established in 2007 by recruiting 502 work units with 10,036 employees within large public service workplaces in a Danish county. A total of 4,489 employees returned a completed questionnaire on the psychosocial work environment and health (participation rate 45%). The workplaces included hospitals, schools, day care centres, social, technical and environmental services and administration. For 490 work units, we identified the work-unit leaders. These work units employed 10,009 workers that comprised the source population. After exclusion of 41 participants with incomplete information, 468 work units and 4,448 participants (including 263 leaders) remained. The number of employees in a work unit ranged from 1 to 124. Characteristics of the cohort are given in Table 1 and a detailed description is given in [9].

Table 1 Number of employees, work-units and questionnaire respondents at baseline

Measures of demographic characteristics

Information on gender, age, civic status and socio-economic position at study baseline in 2007 was obtained by linkage to public registries in Statistics Denmark. Civic status was defined as married or registered partnership versus all others. Socio-economic position was defined by level of education and current occupational position and classified into three categories: high, medium and low.

Measures of the psychosocial work environment

The psychosocial working environment was characterized according to the demand-control [10], the effort reward imbalance [11] and the organizational justice [12] models and measured by established questionnaires.

Job demands, decision authority and skill discretion were each assessed by the mean score of four 5-level items, derived from the Copenhagen Psychosocial Questionnaire [13]. The mean score of the latter two scales was used to obtain a score for decision latitude. The combination of high job demands and low decision latitude using medians as level of dichotomies was defined as job strain. Effort-reward imbalance (ERI) was assessed as the ratio of the mean scores of three items addressing effort and the mean scores of seven items addressing reward [11]. Overcommitment at work, as measured by the intrinsic-effort dimension of the ERI model, was ignored in this study. Organizational justice was measured by the mean of item scores in the scales developed by Moorman [12]. Four items were used to assess procedural injustice and four items to assess relational injustice. In both scales responses were given on a five point Likert scale that ranged from strongly disagree to strongly agree.

The arithmetic mean values of each of the five scales of the psychosocial work environment were computed among employees by work-unit and this work-unit mean was subsequently assigned all non-respondents employed in the same work-unit. In the exposure-outcome analysis each of the scales was dichotomized into low and high (assigned) individual values by the median value.

Measures of outcome

Using the unique personal identification code that all Danish citizens are assigned, we obtained complete information about sick leave exceeding two weeks and redeemed antidepressant prescription by linkage to the DREAM registry [14] and the Danish Medicinal Product Registry, respectively.

Data on sick leave were obtained from The National Register on Public Transfer Payments (DREAM), which contains weekly information on all public transfer payments for all residents in Denmark, and has been shown to be feasible for register-based follow-up of social and economic consequences of disease.

The Danish Medicinal Product Registry covers all pharmacies in Denmark. Antidepressants are only available by prescription in Denmark. The Medicinal Product Registry classifies prescribed Pharmaceuticals according to the Anatomical Therapeutic Chemical classification system (ATC) at the level of the generic pharmaceutical. We used prescription of one or more of the following drugs to define the endpoints for the present study: tricyclic antidepressants (TCA, ATC code N06AA), selective serotonin reuptake inhibitors (SSRI, ATC code N06AB), noradrenaline reuptake inhibitors (NARI, ATC code N06AX) and monoamin oxidase inhibitors (MAO-inibitors, ATC codes N06AF and N06AG). Lithium salts are mostly prescribed for bipolar affective disorders and were not included.

The study protocol was approved by The Danish National Committee on Biomedical Research Ethics, Region Central Denmark (RRS 2006-1028) and the Danish Data Protection Agency (2009-41-3215).


First, we compared respondents with non-respondents with respect to demographic characteristics. Second, associations between the average work-unit response rate and the average work-unit scores of the psychosocial work characteristics were examined by linear regression of the former on the latter, and in a multiple linear regression model adjusting for effects of all other psychosocial work factors. Third, we compared differences in risk estimates among respondents as compared with the entire study population: we analysed sick leave of more than 2 weeks (yes/no) and prescription of antidepressive medication redeemed at pharmacies at least once during the follow-up period from 1.5.2007 through 31.12.2008 by proportional hazard regression in the subsets of the study population that had no sick leave lasting 2 weeks or more from 1.1.2007-31.4.2007 (analyses of sick leave) or that had no prescription of antidepressive medication in the first four months of 2007 (analyses of antidepressive medication). For the comparison of differences in risk estimates among respondents as compared with the entire study population, we only included work-units with more than 5 employees, to avoid very unstable average measures of exposure. Time during follow-up was counted in weeks. Observations were censured if the participant died or emigrated during follow-up (n = 84). In analyses of sick leave, observations were also censured in case of retirement, rehabilitation or leave for other reasons except sick leave (e.g., education, pregnancy, child care). Hazard ratios (HR) were computed with respect to gender, age, civic status, social position and psychosocial work environment dimensions among questionnaire respondents and in the entire study population with adjustment for age and gender. Finally, we calculated relative risk ratios: defined as the risk estimate in the participant population divided by the risk estimate in the entire source population. Approximate confidence intervals were computed by the formula given in [8]. If the relative hazard ratio (RHR) is 1.0 there is no indication of bias.


Compared to non-responders, participants at baseline in 2007 were more often women [OR 1.84 (95% CI 1.8-2.2)], older than 45 years of age [OR 1.42 (95% CI 1.3-1.5)], more often of high social class [OR 1.53 (95% CI 1.4-1.7)] or medium social class [(1.93 (95% CI 1.7-2.2)], had less often sick leaves of more than 2 weeks duration [OR 0.84 (95% CI 0.8-0.9)], and were less often prescribed antidepressant medication in 2007, when the cohort was recruited.

The association between the work-unit specific response rate and the average work unit scale score among questionnaire respondents are shown in Figure 1 for job demands. While the response rate increased with increasing job demands in a multiple linear regression model adjusting for effects of all other psychosocial work factors (p < 0.01), we observed no association for decision latitude. Likewise there was no association between work-unit response rate and the dimensions of the effort-reward imbalance model and the organisational justice models (data not shown).

Figure 1

The association between work-unit specific participation rate (%) and average scale scores of self-reported job demands. The PRISME study, Denmark, 2007.

The risk of sick leave and prescription of antidepressants during follow-up according to age, gender, civic status, socio-economic position and psychosocial work characteristics among responders and entire study population is provided in Table 2 and 3. When we calculated hazard ratios (HR), sixteen of 22 risk estimates differed by less than 10%, 19 by less than 15% and only the risk estimates for gender and decision latitude differed by more than 20%. None of the p-values for the relative HRs were less than 0.05. Risk estimates based on the subset of participants did only differ marginally from risk estimates based upon the entire population.

Table 2 Hazard ratio (HR) for use of antidepressants among respondents and for the entire study population
Table 3 Hazard ratio (HR) for sick leave above 2 weeks among respondents and for the entire study population


The participation rate in the PRISME study was 45% and respondents differed from non-respondents at baseline by gender, age, social class and sick leave. The analyzed variables pointed to better health among participants. In spite of differential participation the study does not indicate that the association between the work place level of psychosocial work environment and mental health outcomes were systematically biased since risk estimates based on the subset of participants did only differ marginally from risk estimates based upon the entire population. These findings are in line with the results of few similar studies [48, 15, 16], but need cautious interpretation.

Limitations of self-reported measures of the psychosocial work environment have been acknowledged for many years [17, 18]. As an alternative to crude objective indicators as overcrowding in hospital wards [19] and costly observational methods [20], the PRISME study relies on the average reporting of psychosocial work characteristics in managerial work-units. This approach assumes that psychosocial work characteristics are more homogeneous within than between work-units [9].

In studies of psychosocial work factors and health or sick leave, a slight overrepresentation of healthy, older, middle and high class women is expected, and may represent jobs with slightly different working conditions than the average. Furthermore, men and women may have different working conditions [21]. We have therefore adjusted all risk estimates between psychosocial factors and health for possible confounders: Age, gender, civic status, and social status.

Changing working conditions during a 20 months follow-up period may have played a role for the results. However, we expect that this may have been the case for both the risk estimates for the entire source population and the risk estimates for the subgroup of responders.

Our measure of work-unit specific psychosocial work characteristics were based upon the answers from those 45% that filled-in the questionnaire. When the work-unit average values were assigned to the non-respondents a bias could be introduced if the non-respondents perceived the working environment differently. In order to address this issue we analyzed the associations between the work-unit response rate and the average scale values for work characteristics among respondents. With one exception we found no such association indicating that biased estimation of exposure related to low response rate is unlikely. However, the perception of job demands was higher in work-units with high response rate. This might indicate that employees in work units with high job demands were more interested in participating in the study. If so, no reporting bias is expected when the rating of the respondents are assigned to the non-respondents within the same work-units. However, higher average job-demands in work units with high participation could also reflect a higher participation of employees that - everything equal - perceive the work as more demanding than other employees in the same department. Since the prescription of anti-depressive drugs was also higher among non-participants, bias of the exposure - outcome relation toward the null cannot be ruled out.

Although reassuring, findings for the limited number of outcomes that could be analyzed in both the subset of questionnaire respondents and the source population do not exclude biased risk estimates for other outcomes. On the other hand it is a strength that comprehensive measures of the psychosocial environment were available for the entire source population and that complete and independent information on the chosen outcomes was available.


In conclusion, we found no indications that low participation distorts the estimates of associations between the work place level of psychosocial work environment and mental health outcomes.


  1. 1.

    Tjonneland A, Olsen A, Boll K, Stripp C, Christensen J, Engholm G, Overvad K: Study design, exposure variables, and socioeconomic determinants of participation in Diet, Cancer and Health: a population-based prospective cohort study of 57,053 men and women in Denmark. Scand J Public Health. 2007, 35: 432-41. 10.1080/14034940601047986.

    Article  PubMed  Google Scholar 

  2. 2.

    Brown WJ, Bryson L, Byles JE, Dobson AJ, Lee C, Mishra G, Schofield M: Women's Health Australia: recruitment for a national longitudinal cohort study. Women Health. 1998, 28: 23-40.

    CAS  Article  PubMed  Google Scholar 

  3. 3.

    Andersen LB, Vestbo J, Juel K, Bjerg AM, Keiding N, Jensen G, Hein HO, Sørensen TI: A comparison of mortality rates in three prospective studies from Copenhagen with mortality rates in the central part of the city, and the entire country. Copenhagen Center for Prospective Population Studies. Eur J Epidemiol. 1998, 14: 579-85. 10.1023/A:1007485116788.

    CAS  Article  PubMed  Google Scholar 

  4. 4.

    Nygard JF, Klungsoyr O, Sandanger I, Svensson E: Selection effects in psychiatric epidemiology: A 14-year prospective study of the Hopkins Symptom Check List-25 as a predictor of mortality in the Norwegian general population. Soc Psychiatry Psychiatr Epidemiol. 2009, 44: 881-6. 10.1007/s00127-009-0010-z.

    Article  PubMed  Google Scholar 

  5. 5.

    der Wiel AB, van EE, de Craen AJ, Gussekloo J, Lagaay AM, Knook DL, Westendorp RG: A high response is not essential to prevent selection bias: results from the Leiden 85-plus study. J Clin Epidemiol. 2002, 55: 1119-25. 10.1016/S0895-4356(02)00505-X.

    Article  PubMed  Google Scholar 

  6. 6.

    Fejer R, Hartvigsen J, Kyvik KO, Jordan A, Christensen HW, Høilund-Carlsen PF: The Funen Neck and Chest Pain study: analysing non-response bias by using national vital statistic data. Eur J Epidemiol. 2006, 21: 171-80. 10.1007/s10654-006-0006-x.

    Article  PubMed  Google Scholar 

  7. 7.

    Osler M, Kriegbaum M, Christensen U, Holstein B, Nybo Andersen AM: Rapid report on methodology: does loss to follow-up in a cohort study bias associations between early life factors and lifestyle-related health outcomes?. Ann Epidemiol. 2008, 18: 422-4. 10.1016/j.annepidem.2007.12.008.

    Article  PubMed  Google Scholar 

  8. 8.

    Nohr EA, Frydenberg M, Henriksen TB, Olsen J: Does low participation in cohort studies induce bias?. Epidemiology. 2006, 17: 413-8. 10.1097/01.ede.0000220549.14177.60.

    Article  PubMed  Google Scholar 

  9. 9.

    Kolstad HA, Hansen AM, Kærgaard A, Thomsen JF, Kaerlev L, Mikkelsen S, Grynderup MB, Mors O, Rugulies R, Kristensen AS, Andersen JH, Bonde JP: Job strain and the risk of depression: Is reporting biased?. Am J Epidemiol. 2011, 173: 94-102. 10.1093/aje/kwq318.

    Article  PubMed  Google Scholar 

  10. 10.

    Karasek R, Theorell T: Healthy Work. Stress productivity and the reconstruction of working life. 1990, New York: Basic Books

    Google Scholar 

  11. 11.

    Siegrist J, Starke D, Chandola T, Godin I, Marmot M, Niedhammer I, Peter R: The measurement of effort-reward imbalance at work: European comparisons. Soc Sci Med. 2004, 58: 1483-99. 10.1016/S0277-9536(03)00351-4.

    Article  PubMed  Google Scholar 

  12. 12.

    Moorman R: Relationship between organisational justice and organisational citizenship behaviours: Do fairness perceptions influence employee citizenship?. Journal of Applied Psychology. 1991, 76: 845-55.

    Article  Google Scholar 

  13. 13.

    Kristensen TS, Hannerz H, Høgh A, Borg V: The Copenhagen Psychosocial Questionnaire--a tool for assessment and improvement of the psychosocial work environment. Scand J Work Environ Health. 2005, 31: 438-49.

    Article  PubMed  Google Scholar 

  14. 14.

    Hjollund NH, Larsen FB, Andersen JH: Register-based follow-up of social benefits and other transfer payments: accuracy and degree of completeness in a Danish interdepartmental administrative database compared with a population-based survey. Scand J Public Health. 2007, 35: 497-497. 10.1080/14034940701271882.

    Article  PubMed  Google Scholar 

  15. 15.

    Reijneveld SA, Stronks K: The impact of response bias on estimates of health care utilization in a metropolitan area: the use of administrative data. Int J Epidemiol. 1999, 28: 1134-40. 10.1093/ije/28.6.1134.

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Heilbrun LK, Nomura A, Stemmermann GN: The effects of non-response in a prospective study of cancer: 15-year follow-up. Int J Epidemiol. 1991, 20: 328-328. 10.1093/ije/20.2.328.

    CAS  Article  PubMed  Google Scholar 

  17. 17.

    Kasl SV: Measuring job stressors and studying the health impact of the work environment: an epidemiologic commentary. J Occup Health Psych. 1998, 3: 390-401.

    CAS  Article  Google Scholar 

  18. 18.

    Freese M, Zapf D: Methodological issues in the study of Work Stress: Objective versus subjective measurement of Work Stress and the question of Longitudinal studies. Causes, Coping and Consequences of Stress at Work. Edited by: Cooper CL, Pain P. 1988, John Wiley and Sons Ltd

    Google Scholar 

  19. 19.

    Virtanen M, Pentti J, Vahtera J, Ferrie JE, Stansfeld SA, Helenius H, Elovainio M, Honkonen T, Terho K, Oksanen T, Kivimäki M: Overcrowding in hospital wards as a predictor of antidepressant treatment among hospital staff. Am J Psychiatry. 2008, 165: 1482-6. 10.1176/appi.ajp.2008.07121929.

    Article  PubMed  Google Scholar 

  20. 20.

    Griffin JM, Greiner BA, Stansfeld SA, Marmot M: The effect of self-reported and observed job conditions on depression and anxiety symptoms: a comparison of theoretical models. J Occup Health Psych. 2007, 12: 334-349.

    Article  Google Scholar 

  21. 21.

    Waldenström K, Härenstam A: Does the job demand-control model correspond to externally assessed demands and control for both women and men?. Scand J Public Health. 2008, 36: 242-9. 10.1177/1403494807085079.

    Article  PubMed  Google Scholar 

Pre-publication history

  1. The pre-publication history for this paper can be accessed here:

Download references


The study was supported by a grant from the Danish Working Environment Research Fund ( 5-2005-09). We acknowledge biostatician Jan Nielsen for preparing the layout to Figure 1.

Author information



Corresponding author

Correspondence to Linda Kaerlev.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

LK, HAK, ÅMH, JFT, AK, JPB coordinated the data collection in 2007. LK and JPB performed the statistical analysis for the present paper and drafted the manuscript. HAK, ÅMH, JFT, AK, RR, SM, JHA, OM, MBG have helped with interpretation of the analyses and with revising the manuscript critically. All authors participated in the design and coordination of the PRISME study, development of the questionnaire and have made substantial contributions to interpretation of data; and read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Kaerlev, L., Kolstad, H.A., Hansen, Å.M. et al. Are risk estimates biased in follow-up studies of psychosocial factors with low base-line participation?. BMC Public Health 11, 539 (2011).

Download citation


  • cohort study
  • health survey
  • non-response
  • psychosocial distress
  • affective disorders