Use of the Pediatric Symptom Checklist for the detection of psychosocial problems in preventive child healthcare

Population

The sample was obtained using a two-stage selection procedure. In the first stage, a national sample from 9 of the 41 Dutch Preventive Child Health Services was taken. In the second stage, each Service provided a sample of children aged 7–12 who were invited for routine well-child examinations. We aimed at a sample size of 700 respondents for evaluation, as earlier studies [15, 16] demonstrated that short questionnaires used in PCH settings allow for an area under the ROC curves (AUC) of about 0.90 with a clinical CBCL Total Problems score as criterion. A sample of 700 suffices to estimate this AUC with a 95% confidence interval of +/- 0.02.

Of the total sample of 815 eligible children, 687 participated and 674 provided complete data on both questionnaires (84.3% and 82.7% of the original sample, respectively). Both groups were representative of the total sample regarding age and gender, but non-response was higher for children of immigrant/minority origin (27.4% vs. 12.2%). Analyses were restricted to children with complete data for both questionnaires to make interpretation easier.

Data collection

The data were collected according to a standardised procedure during routine well-child examinations, from September 2004 to July 2005. The study was performed in compliance with the Helsinki Declaration [17]. The design of the study was approved by the local TNO Medical-Ethical Committee and includes verbal informed consent by parents.

The PSC [10] and the CBCL [18, 19] were mailed to children, along with the standard invitation for the preventive health assessment. Before attending the assessments, parents completed the questionnaires, placed them in sealed envelopes and gave them to the CHPs, who in turn passed them on to the researchers without opening them (in contrast with routine use, where the CHP would partially base the interview on the information from the PSC). The CHP interviewed each child and its parents regarding mental health and background, and examined each child. After each assessment, the CHP answered the following question: 'Does the child have a psychosocial problem, at this moment?' (yes, no) and scored its severity (mild, moderate or severe) and the type of problems identified using a pre-coded list. Children who only had risk indicators for the development of psychosocial problems, such as having parents with psychiatric problems or other family problems, had to be coded as having no problems.

The PSC was translated following the procedure proposed by Guillemin et al. [20]. Firstly, the original US English version of the questionnaire was translated into Dutch by three certified translators working independently of each other. Secondly, three further certified translators each translated one Dutch translation back into US English. The resulting US English versions were compared to the originals and all discrepancies were discussed by three researchers (SAR, MRC and AGCV) who spoke both Dutch and English. Discrepancies were also discussed with the developers of the PSC, Dr J.M. Murphy and Dr M.S. Jellinek, especially where items raised questions as to their intended meaning. The PSC consists of 35 items that are rated as never, sometimes or often present (0, 1 and 2, respectively). Item scores are summed; we dichotomised at 0–27 vs. 28–70, following the US cut-off [10].

The CBCL was used to assess parents' reports of the behavioural and emotional problems of their children over the preceding six months. Its (good) reliability and validity has been established [18, 19]. We used only the 120 problem items from the CBCL and computed scores for two broad-band groups of syndromes designated as Internalising and Externalising, and a Total Problems score. Children were also allocated to a normal range or a clinical range, using the 90th percentile of the Dutch normative sample as the cut-off [19].

Analysis

In the analysis we assessed the psychometric properties of the PSC and its added value in identifying psychosocial problems. Regarding psychometric properties, we first computed its internal consistency and examined the fit between the scale structure and the observed data using confirmatory factor analysis (CFA) with structural equation modelling. Next, we assessed the validity of the PSC using dichotomised CBCL scores (Total Problems score and Internalising/Externalising scales) and referral by the CHP due to psychosocial problems as criteria. Finally, we assessed whether mean PSC scores differed with the children's background.

Regarding the added value of the PSC in identifying psychosocial problems, we assessed the odds of identification of mental health problems (i.e. a clinical CBCL Total Problems score) using an elevated score on the PSC. This was repeated with adjustment for social and demographic risk indicators known to the CHP that might have helped in the identification of psychosocial problems [3, 4]. Regarding social and demographic risk indicators, we retained children with missing data in the logistic regression models by creating separate dummies for the missing category of each variable.

All analyses were done with SPSS 12.0 for Windows [21], except the CFA, which was done with Amos 5 [22]. All analyses were repeated for boys and girls separately. Results for these subgroups are provided only if they differed in a statistically significant way (p < 0.05).

Demographics

Scores on PSC and CBCL

Validity

Subsequently, the degree to which the score on the PSC is truly elevated in the case of psychosocial problems as measured by these four criteria (i.e. sensitivity) and the degree to which it is 'normal' in the case of the absence of these problems (i.e. specificity) were assessed. For the recommended cut-off of the PSC at 28 and above, scores were 0.33 and 0.98 respectively, using a clinical CBCL Total Problems score as the criterion, and 0.19 and 0.97 respectively, using being under treatment for mental health problems as the criterion. Figure 1 shows the Receiver Operating Characteristic (ROC) curve for all possible cut-off points. The curve is close to the upper-left corner of the figure, particularly when the CBCL is used as the criterion, indicating a high validity of the PSC if this gold standard is used. Curves for CBCL Internalising and Externalising Problems are largely similar but slightly more off the upper-left corner (i.e. less favourable; not shown). The same holds for problems detected by the CHP when compared with the curve for 'under treatment' (not shown). Table 3 shows the resulting areas under the ROC curves (AUC) and positive and negative predictive values for both cut-offs. Results regarding AUCs did not differ by gender (not shown).

Differences in scores by background characteristics

Mean PSC scores were higher for boys, for children from minority backgrounds, single-parent families and unemployed families (Table 1, final columns).

Added value

Finally, we examined the degree to which information from the PSC contributed to the diagnosis of psychosocial problems as measured by the CBCL over and above the clinical opinion of the child health physician without knowledge of the PSC. This yielded an odds ratio of 21.3 (95% confidence interval 8.7 to 52.2), with the only predictive background characteristic being family composition. Using the alternative PSC cut-off of 22+ yielded slightly higher odds ratios.

Parent opinion of the PSC

A large majority of parents completed the PSC fully (91.1%) and no parent missed more than 3 items. However, 20% of parents made critical remarks about the PSC, mainly concerning lack of fit between questions and answer categories (7%) and unclear questions (5%).

Limitations

Methodological factors are unlikely to have affected these results. In general, the response rate was high (84%). Moreover, we used the CBCL as a criterion, which has been proven to be a valid measure for psychosocial problems. Because of complexity and high costs, structured clinical interviews such as the Diagnostic Interview Schedule for Children were not used as criteria [23]. Doing so may have provided additional information but differences with questionnaire-based information have been shown to be small [24].

Fit with previous research on PSC and on other questionnaires used in PCH

This first study of the Dutch version of the PSC yielded results on reliability and on validity regarding the CBCL that are very similar to those found in comparable US samples. Jellinek et al. reported a sensitivity of 51.5% and a specificity of 95.4% at a cut-off of 0–27/28+, using the CBCL as a criterion in a sample of 206 children from the USA [9, 25]. These values are very similar to those for a cut-off of 0–24/25+ for the Dutch version, i.e. 53.3% and 97.3% respectively (compare figure). We found the internal consistency of the PSC similar to that found by Jellinek et al. [25] but the results of our confirmatory factor analyses cast some doubt as to whether it measures a single latent, as did a previous study of Gardner et al. [26].

We found much lower mean scores on the PSC than have been found for comparable US samples. Mean scores on other symptom checklists such as the CBCL are also lower for Dutch children than for children from the USA [27]. Therefore, the Dutch children's lower mean PSC scores probably reflect real differences between these countries in the levels of symptoms reported by parents. This also implies that the cut-off for an elevated score on the PSC should be set lower for Dutch children than for children from the US. At this lower cut-off, the sensitivity and specificity of the Dutch version is similar to that of the US version. Moreover, the test characteristics of the PSC are comparable with or slightly better than those of most questionnaires currently used in Dutch PCH [15, 16, 28].

Finally, we found higher mean PSC scores for boys and for children from single-parent families, similar to those found by Jellinek et al. for children in the USA [10]. We did not find differences in terms of parental education level, in contrast to the findings of Jellinek et al. [10], but we did find elevated scores among children with unemployed parents, an indicator of familial socioeconomic status that was not studied by Jellinek et al. [10].

Implications

The results of our study imply that the PSC is useful for the early detection of psychosocial problems by PCH, especially if an adjusted cut-off is used. The PSC mostly detects behavioural and emotional problems, which are common in this age group. However, questions on more extreme behaviours such as the abuse of alcohol and drugs are not asked. Screening using the PSC is best carried out as a first step in a two-step process on the way to referral. A relatively low-cut-off can then be used to avoid missing too many cases. In a second step, cases flagged by the PSC should then be assessed by a CHP before making a final decision about referral. Parental responses show that some questions may require revision. In any event, the PSC is a useful aid for the early detection of psychosocial problems that could be considered for use in other countries as well.