Skip to main content

Table 1 Important sources of bias independent of study design

From: Challenges in evaluating Welfare to Work policy interventions: would an RCT design have been the answer to all our problems?

Intervention characteristic Potential bias & other influences on effect estimates
Only a small proportion of the eligible IB population access the intervention Selection
- it is very difficult to get a comparison group with the same characteristics as the intervention group
Intervention is to promote uptake of a service already available elsewhere (Jobcentre Plus) Dilution - variation in exposure to the intervention across the study sample
- risk of contamination among comparison group
Intervention is targeted at socio-economically deprived population Selection & Attrition
- less likely to participate in the study
- little incentive for comparison group to take part in research, withdrawals likely
GP practices volunteer to participate in the intervention Selection
- primary care service that is already motivated to promote service use among vulnerable group
Generalisability
- GP practices who volunteer to take part may differ from those who do not
Referral is opportunistic, referral criteria not well defined Selection
- referral decisions may vary within and between GPs
Number of potential sample within an intervention practice is unknown Selection
- unknown number of eligible IB recipients will not be referred by GP
Identification and referral of eligible IB recipients initiates the intervention, before recruitment to the evaluation study Recall
- Pre-intervention data reliant on medical notes and retrospective recall from IB recipient
Expected short term health effects likely to be small Study powered to detect small differences in health requires unfeasibly large population. Underpowered study may produce false result.