Data
SHARE data are longitudinal data collected at five time points 2004 (wave 1), 2006 (wave2), 2008 (wave 3), 2011 (wave 4) and 2013 (wave 5). We exclude wave 3 and wave 2. Wave 3 is focused on participants’ lifestyle and does not provide information relevant for this study. Wave 2 provides information about loneliness using one-single item with two level answers: yes and no, while the other waves use a different measurement with multi-level answers. Data for all included waves are available for 10 different European countries, namely: Austria, Germany, Sweden, the Netherlands, Spain, Italy, France, Denmark, Switzerland and Belgium. Since the aim of this study is to explore the effects of a policy change on loneliness among older adults in the Netherlands after 2007, countries that in a given period did not experience the same policy changes as the treated unit can be used as a “donor pool” to create the control group for both SCM and DiD. The first wave is collected in the 2004 and it represents the pre-treatment period, while the other two waves are collected after the policy intervention and they are used for the post-treatment estimation.
The SHARE data include older individuals and their partners, who live either in their own house or in a nursing home. As household help is available only for people who live in their own home, we exclude individuals who live in a nursing home.
Loneliness is measured by a one item question which is asked in all three waves: “How often did you feel lonely during the last 12 months?”. The answers are on a three level Likert scale and include the following categories: “hardly ever or never”, “sometimes” and “often”. We have constructed a binary indicator variable where categories “often” and “sometimes” are coded 1 (lonely) while category “hardly ever or never” is coded as 0. For the DiD, this indicator will be our outcome variable. We also use this indicator to estimate the prevalence of loneliness for each of the countries in the four waves. This will be our outcome variable for the SCM.
As predictors we use the same set of covariates in both the SCM and the DiD: gender, marital status, being a migrant, household size, age, number of children, type of settlement, being depressed (measured by the standardized EURO-D multi-level scale), number of chronic diseases, level of mobility (measured by The Global Activity Limitation Index) and using help from others. There was no statistical significant difference in age between waves (67.36 ± 13.7 before 2011 to 68.55 ± 9.24 after 2011, p = 0.01). For the depression scale we use a binary indicator (depressed – not depressed) provided by SHARE data. The indicator is calculated based on cut-off points for the EURO-D scale. For the number of chronic diseases we have created a binary indicator coded as yes if number of chronic diseases is one or more; otherwise this variable is 0. Regarding the association between the number of chronic diseases and loneliness - we did not find a statistically significant correlation. The level of mobility is also provided as a binary indicator (limited – not limited) within SHARE data. It is calculated by using The Global Activity Limitation Index that is a one question instrument. Recent studies show that older women report higher levels of loneliness than older men, while also older migrants have a higher probability to experience loneliness than older people living in their country of birth [19]. Older people who live with their partner or with other family members less often report loneliness [16]. Also, older adults who use help of others including their social network report loneliness less often. We also wanted to include education as a potential predictor. However, this variable is measured using different classifications in different waves of the SHARE data and we were therefore unable to use this variable. In both analyses, we use the Netherlands as the treated unit while the other 9 countries are used to construct the control group.
Synthetic control method and differences in differences
DiD has been widely applied in the evaluation of health policy measures and health interventions [20]. This approach uses observational longitudinal data to simulate an experimental design. It calculates the difference in outcome measures for the treatment group before and after the treatment. The same difference is calculated for the control group. After that the difference within two differences is estimated. In our case, DiD will first estimate the difference in loneliness among older individuals in the Netherlands and those from 9 control countries for the period until 2007. After this, the same difference is calculated for the period 2011–2013. The difference between two differences gives us the average treatment on the treated (ATT) effect and its statistical significance. DiD also allows us to estimate the ATT with a binary outcome variable.
The main disadvantage of DiD is that this approach is based on the very strict “parallel trend assumption” which assume that the average outcomes for control and treatment groups on the outcome measure would follow the same parallel trend over time in the absence of the policy intervention [18]. In other words, it assumes the unobserved confounders that affect the outcome measure do not change over time. However, this assumption is not always plausible when it comes to the evaluation of health policy interventions [22].
Recently Abadie et al., 2015 has suggested SCM to evaluate policy interventions. The SCM compares results on the outcome variable between the treated unit (one country or region) with its counterfactual outcome. The counterfactual outcome is calculated by using the weighted average of the outcomes from several control units (synthetic control group) that were not exposed to the policy measure. The policy effect (treatment effect) is calculated as the difference in the outcome variable between the treated unit and the synthetic control group (control group) after the policy implementation [15]. In this way SCM incorporates advantages from DiD (comparing the control and treated groups before and after the intervention) and propensity scores matching (the synthetic control group is constructed as an average of several control units that are matched on a set of covariates in order to be the most similar with the treated unit) [21]. In other words, by SCM we compare the loneliness among older adults in The Netherlands and the synthetic control group for the Netherlands (control group constructed in a way to be most similar to the Netherlands) before and after policy change. The SCM also does not require a “parallel trend assumption”- the effects of unobserved cofounding factors can vary within time. The main disadvantage of this method is that it is applied using only one treated unit [15]. In order to provide more robust results, we will use both approaches.
To obtain the level of statistical significance of the treatment effect, Abadie et al. (2015) suggested the use of a placebo – test. The placebo - test represents a permutation in which each control group is used as if it was exposed to the treatment and the treatment unit is excluded. In this study we choose two countries with similar trends in loneliness as the Netherlands and use them as potential treatment units in the placebo tests.