External validation of two prediction models identifying employees at risk of high sickness absence: cohort study with 1-year follow-up

Study population and setting

In November 2006, 1,137 office workers of an insurance company were invited to participate in an occupational health check-up. The health check-up questionnaire assessed general health, mental health, work conditions, and the working environment. General health was investigated with the question: In general, how would you rate your health? This question has been used as a health measure in surveys worldwide and was found to be associated with various morbidity measures and the use of health services [23]. For SRH, office workers rated their health in categories 4 = “excellent”, 3 = “good”, 2 = “fair” and 1 = “poor”. The Medical Ethics Committee of the University Medical Center Groningen granted ethical clearance for linking the health check-up data to the SA data.

Sickness absence data

Sickness absence (SA) was defined as absence from work due to work-related and non work-related injuries or illnesses. SA data were retrieved from an occupational health service register that records SA from the day of reporting sick to the day an employee resumed work at equal earnings as before SA. The calendar days between the first and last SA day were accumulated. For example, if a worker is off work 1 day on one occasion and 5 days on another, this was counted as 6 SA days and 2 SA episodes, unless the episodes were less than 28 days apart. In line with Dutch SA insurance policies, SA episodes with less than 28 days worked between them were regarded as one SA episode. The total number of SA days in 2005 and 2006 was tallied for each employee as a measure for prior SA in the SA days model. Likewise, the total number of SA episodes in 2005 and 2006 was accumulated for each employee as a measure for prior SA in the SA episodes model.

The number of SA days and episodes were also recorded for each employee during 1-year follow-up in 2007. At the development of the prediction models, high SA days was defined as ≥30 accumulated (not necessarily consecutive) SA days and high SA episodes as ≥3 episodes during 1-year follow-up [17]. The same definitions for high SA were adopted in the current external validation study.

External validation of prediction models

The original prediction models were applied in the external dataset with fixed regression coefficients i.e., by transporting the regression coefficients from the development setting to the validation setting. The external validity of predictions was quantified by performance measures related to discrimination and calibration [24–26]. Discrimination was evaluated by the area under the receiver operating characteristic curve (AUC). An AUC of 0.5 indicates no discrimination above chance and an AUC of 1.0 indicates perfect discrimination. Generally, an AUC = 0.9-1.0 represents excellent, AUC = 0.8-0.9 good, AUC = 0.7-0.8 fair, and AUC = 0.6-0.7 poor discriminative ability. Discrimination is assumed to be useful if AUC ≥0.75 [27]. The prognostic validity of the prediction models was investigated in more detail by calculating the sensitivity and specificity at different cut-off points of predicted high SA probabilities. Calibration was assessed by plotting predicted probabilities with fixed regression coefficients obtained from the development setting of health care workers against the observed probabilities in the population of office workers [24–26]. Calibration was expressed in the calibration slope with a calibration slope = 1 indicating perfect calibration.

Updating the prediction models

The aforementioned external validation kept the regression coefficients fixed at their original value obtained from the development setting. Updating was performed by model revision, which is the re-estimation of the regression coefficients of predictor variables and/or considering more covariates for inclusion in the model [25, 26, 28]. First, the regression coefficients of the prediction models were re-estimated for the population of office workers. After re-estimation of the regression coefficients, gender was added as a predictor to the prediction models. The effect of including gender was assessed by using the Likelihood Ratio (LR) test and significance was concluded for LR p < 0.05. Furthermore, we evaluated the effect of excluding SRH from the prediction models, since SRH is not usually recorded in SA registers. The effect of excluding SRH was also assessed by LR testing.

Software

External validation was performed in R (R Development Core Team, 2009) using Harrell’s Regression Modeling Strategies (rms) package, version 3.2-0 [29].

External validation of prediction models

A total of 66 (11%) office workers had high SA days and 67 (11%) office workers had high SA episodes during follow-up; 29 office workers had both high SA days and episodes. Figure 1 shows the receiver operating characteristic curves, which reflected a poor discriminative ability of the SA days model (AUC = 0.65; 95% CI = 0.58 – 0.71) and a fair discriminative ability of the SA episodes model (AUC = 0.76; 95% CI = 0.70 – 0.82). The sensitivity and specificity at different cut off points for the probability of high SA episodes are shown in Table 2.

Calibration was acceptable for both prediction models, as is shown in the calibration plot (Figure 2) with calibration slopes of 0.89 for the SA days model and 0.96 for the SA episodes model.

Updating the prediction models

With regard to the SA episodes model, the re-estimated regression coefficient of SRH was reduced from −0.715 at the development setting to −0.190 at the current validation setting. The SA episodes model neither improved after adding gender (LR-test p = 0.11), nor degraded after excluding SRH (LR-test p = 0.31).

Prognostic performance

The discriminative ability of both prediction models degraded in the population of office workers, although the SA episodes model still showed fair performance. Furthermore, the cut-off probabilities of the SA episodes model confirm those of the development setting. At a cut-off risk of high SA of 10%, the sensitivity was acceptable, but the specificity was low due to high false-positive rates. A sensitive cut-off point can be used to identify as much office workers at risk of high SA as possible. For example, workers with high SA episodes may suffer chronic recurrent conditions that are not yet diagnosed or treated. From a societal perspective, it may be desirable to select workers with a ≥10% probability of high SA episodes for further diagnosis and treatment to prevent worsening of chronic conditions, long-term SA and subsequent disability pensioning. Alternatively, more specific cut-off points can be used to reduce false-positive rates, for instance to select high-risk office workers for costly interventions.

Why did the prognostic performance degrade?

The purpose of a prediction model is to provide valid predictions for new subjects [24–26]. External validation refers to the transportability of a prediction model to other settings than where the model was developed [18, 30]. Prediction models tend to perform better in the subjects used to develop the model than in other subjects, a phenomenon known as over-optimism [19]. For internal model validation, bootstrapping methods are recommended to provide bias-corrected estimates of model performance. In the development sample of health care workers, internal validation by bootstrapping revealed an over-optimism of 0.06 for the SA days model and 0.03 for the SA episodes model. Subsequently, the performance parameters were shrunken to adjust for this over-optimism [24–26, 28]. Although adjustment for over-optimism by bootstrap techniques may not be sufficient in relatively small data sets [31], this low over-optimism made it unlikely that the poorer performance of the prediction models in the sample of office workers was due to overfitting to the development sample.

Alternatively, underfitting occurs when important predictors are missing from the prediction models. Internal validation by bootstrapping techniques will not detect underfitting because the bootstrap samples are drawn from the same population. The poorer performance of the prediction models in the present study may well be explained by underfitting, in particular because the Nagelkerke pseudo R² values were lower than in the development sample of health care workers. The Nagelkerke’s pseudo R² reflects the variance in high SA between office workers that is explained by the covariates fitted in the prediction models [32]. Low Nagelkerke’s pseudo R² values indicate that other factors than those included in the model may be important for predicting high SA among office workers. Hence, future studies should further update the prediction models with other predictors, e.g. work variables and personal characteristics, provided that these variables are readily available or easy to obtain by physicians.

Another explanation for the lower performance may be the different case-mix in the population of office workers. Case-mix refers to the distribution of known and unknown predictors of SA in the studied populations. The population in which the prediction models were developed consisted of 535 health care workers, predominantly female nurses who were younger than the office workers in the present study. One-third of the development population of health care workers reported excellent health as compared to a quarter of the present population of office workers. Furthermore, 8% of healthcare workers reported less than good health as compared to 18% of office workers. The distribution of prior SA did not differ between the development and the validation populations.

Finally, the regression coefficients may really differ between the two working populations i.e., the working populations were not plausibly related. The prediction models were developed in health care workers, predominantly working in physically and emotionally demanding nursing care. Possibly, this development sample differed too much from the current validation sample of office workers performing mentally demanding work at an insurance company. Furthermore, the ‘healthy worker effect’, which selects the healthiest employees to work until older age, may be greater in nursing care which is more physically demanding than office work. This may explain why the inverse association between age and high SA was stronger in the development sample of health care workers than in the validation sample of office workers. The ‘healthy worker effect’ may also explain why SRH was a stronger predictor of high SA in health care workers than in office workers, particularly since SRH was found to reflect physical functioning rather than mental health [33].

Practical implications and future directions

Prediction models have practical perspectives if they accurately predict outcomes for different populations [18, 30]. This study showed that the SA episodes model accurately predicted the risk of high SA episodes in both health care workers and office workers. Therefore, this prediction model may be a promising tool to select employees at risk of high SA episodes for preventive occupational health consultations. Such consultations were found to reduce SA duration [7, 8], but not SA frequency [7]. Duijts et al. reported that in employees who received preventive coaching the mean SA duration was 11.7 days during 8 – 12 months follow-up as compared to 13.1 days in the control group. The mean SA frequencies were 1.07 and 1.40 respectively, though none of the differences in SA measures was statistically significant [34]. In the current study, the SA episodes model identified employees at risk of a high SA frequency, but the model may also indirectly identify employees at risk of future long SA duration, because frequent SA has been recognized as a risk factor for long-term SA [35–37]. Further research is needed to clarify which frequent absentees develop long-term SA in the future.

It is also important to further validate the SA episodes model, for example in large heterogeneous populations and in multiple settings [18, 20]. The more numerous and diverse the settings in which the SA episodes model accurately predicts high SA, the more likely it will generalize to untested working populations [18]. Furthermore, the SA episodes model should be developed into a nomogram or score chart that is easier to understand and use in daily practice than the regression formula. Simpler presentation formats provide approximate predictions, but this will not be problematic for identifying employees at risk of high SA.