Skip to main content

Table 1 Studies included in review: summary of data extracted

From: The use of Qualitative Comparative Analysis (QCA) to address causality in complex systems: a systematic review of research on public health interventions

Publication details Study details extracted Indicators of reporting quality
Lead Author Publication date Public health domain Rationale for using QCA Crisp or Fuzzy Set? Country/region setting Population Intervention evaluated Data used for analysis - source Type of data (qual/qant) Case level of analysis Number of cases/potential cases Outcome (s) (phenomena explained) Evidence of familiarity of cases Justification for selection of cases Calibration of set membership scores discussed in detail and justified Raw data matrix available Truth tables reported Solution formula reported and justified Consistency and coverage measures reported
Bianchi 2018a Nutrition /obesity “to identify configurations of intervention characteristics associated with, and those not found to be associated with, statistically significant reductions in the demand for meat” Crisp Any (included studies all high income countries) Any Micro-environment interventions to reduce meat consumption Published analyses (systematic review) Quant Intervention component 21 out of total of 22 interventions identified Reduction in consumption, purchase, or selection of meat Y Y N N N Y Y
Bianchi 2018b Nutrition/obesity “to identify combinations of intervention characteristics associated with significant reductions in the demand for meat” Crisp Any Any Interventions targeting conscious determinants of human behaviour to reduce the demand for meat. Published analyses (systematic review) Quant Intervention component 31 out of a total of 59 interventions identified Reductions in actual or intended meat consumption Y Y N N N N Y
Blackman 2011 Health inequalities “to find configurations of these conditions with either the narrowing or not narrowing outcomes.”; “enables causal arguments to be made rigorously by creating a very close correspondence between theory and data analysis, analysing evidence in ways that directly address theoretical arguments about what matters to achieve some outcome: how it happens. This makes QCA especially appropriate for policy research; the process of defining conditions is then also a process of identifying conditions necessary for a policy outcome.” Crisp UK Those living in ‘deprived’ areas/regions, and at-risk of premature death from cancer and cardiovascular disease (CVD) Target setting to reduce inequalities (Local policies and service delivery to identified disadvantaged areas) Primary data (generated for study)
Secondary data
Quant Region/ subnational area - England’s Spearhead areas 27 of possible 70 (27 complete questionairres returned for both cancer and CVD) Not/narrowing cancer gap; or not/narrowing CVD gap Y Y N N Y Y N
Blackman 2010 Health inequalities “QCA […] assumes that outcomes are produced by variables acting together in combination, including the possibility of different combinations leading to the same outcome”; “using fsQCA to explore how the attributes configured against each outcome.” Crisp UK - North-West England Those living in ‘deprived’ areas/regions, and at-risk of premature death from cancer and circulatory diseases Local policies and service delivery to identified disadvantaged areas Primary data (generated for study) Secondary data Qual and Quant Region/ subnational area - local strategic partnerships (LSP) 15 local authority areas in North West England of 21 LSP areas in the North West region that were among the most deprived 88 local authority areas in England (14 and 15 complete questionairries were returned for circulatory diseases and cancers, respectively) Narrowing OR widening cancer gap; or narrowing OR widening circulatory diseases gap Y Y N N Y Y N
Blackman 2013 Health inequalities “enabling causal pathways to be discerned from how sets of conditions combine with particular outcomes: in this instance, whether inequalities in conception rates do or do not narrow, compared with the England average.” Crisp England, deprived local authority areas Women under 18 Local policies and service delivery addressing pregnancy in those aged under-18 Primary data (generated for study) Secondary data Quant Region/ subnational area – local authority 27 from 70 local authority areas A narrowing teenage pregnancy rate gap compared to the national average OR a gap that had not been narrowing. Y Y Y N Y Y Y
Bruer 2018 Mental health Goal to “to evaluate outcomes and processes simultaneously … [QCA] holds promise as a way to do this” Fuzzy Nepal, West Chitwan Primary care patients Programme for Improving Mental Healthcare’ (PRIME) Mental Health Care Plans (MHCPs) Secondary data Quant Organisation - health facilities 10 of the 10 health facilities in Chitwan where the PRIME MHCPs were implemented Effectiveness of the intervention:increased service utilisation Y Y Y Y Y Y Y
Brunton 2014 Community engagement “Given that interventions targeting social determinants of health are necessarily complex (Medical Research Council 2008), this method [QCA] is well-suited to examine the components of effective interventions in maternity and early years interventions.” Fuzzy OECD countries (USA, UK, RoI) Disadvantaged pregnant women and new mothers Community engagement – orientated to antenatal care, breastfeeding, child illness prevention and parenting Published analysis (systematic review) Quant Intervention 24 of 29 studies (a sub-set of the 131 trials included in the braoder review of studies) Effectiveness of the intervention. N Y Y Y Y Y Y
Burchett 2018 Nutrition/ obesity QCA method used to attend to “the inherent variance in intervention content, context and outcomes […] to explore the critical features of WMPs [weight management programs] for children to understand the mechanisms through which interventions have the impact that they do.” Crisp UK Children and Young People Lifestyle weight management interventions Published analysis Quant Intervention 15 (most/least effective) of 30 interventions Intervention effectiveness N Y N N N Y N
Chippone 2018 Nutrition/ obesity & PA “to determine necessary and sufficient technical assistance conditions supporting NAP SACC [Nutrition and Physical Activity Self-Assessment in Child Care] outcomes.”; “to account for equifinality” Fuzzy USA Children and Young People Technical assistance Primary data (generated for study) Quant Intervention (‘ECE programs’) 15 highest performing programs and 15 lowest performing programs (from 10 collaboratives, comprising 84 early care and education programs) Effectiveness of technical assistance Y Y N N Y N N
Fernald 2018 Chronic condition management “to identify conditions/sets important for successful implementation. Qualitative Comparative Analysis (QCA) maintains complexity in the analytical process and allows for multiple pathways to achieve the outcome” Fuzzy USA Primary care patients Community-Created Self-Management Support (SMS) Tools in Primary Care Practices Primary data (generated for study) Qual Organisation (primary care practice) 16 practices of 16 enrolled in study Routine SMS implementation in primary care setting Y Y Y N Y Y Y
Ford 2005 General population health status “allows for the exploration of the necessity, sufficiency and interactions among the three core public health functions and relates them to the outcome of interest—health impact assessments.” Crisp USA Whole population Assessment, assurance and policy development functions (Adherence to IoM’s recommendations) Secondary data Quant Region/ subnational area: State 41 of 50 states Population health improvement 1990–2000 (composite measure based on United Health Group’s ratings) above or below average N Y N N Y Y Y
Glatman-Freedman 2010 Vaccine adoption and implementation “to examine the alternative combinations of factors that are conducive to the success or failure of new vaccine introduction” Crisp African countries eligible for GAVI funding Organisations GAVI funding Secondary data Quant Country 35 (all eligible countries) Country is in one of 3 groups: with both Hep B and Hib vaccines introduced; just hep b introduced; or neither introduced N Y N N N Y Y
Harris 2019 Chronic condition management “to identify the combination of intervention components and processes that are aligned with successful intervention implementation” Fuzzy Any (“Most studies were conducted in North America in socially disadvantaged populations.”) Children and Young People Asthma management interventions (“if their purpose was to help children improve management of their asthma by increasing knowledge, enhancing skills, or changing behaviour.”) Published analysis (systematic review) Qual and Quant Intervention 33 of 55 studies Implementation of intervention successful Y Y Y Y Y Y Y
Harting 2019 Programme implementation “Our theoretical framework indicates that it is the combination of conditions that is important for network performance, rather than the influence of conditions separately. Therefore, we performed a fuzzy set qualitative comparative analysis (fsQCA)” Fuzzy Netherlands, municipalities Policy actors Multi-sectoral network
Gezonde Slagkracht (Decisive Action for Health) program which provided resources for multisectoral networks for action on overweight, alcohol and drug abuse and/or smoking
Primary data (generated for study) Quant Public Health Policy Networks (in municipalities or collaborations of municipalities) 29 of the 34 Dutch public health policy networks in the programme Network performance - indicated by implementation of more than 3 non-educational PH strategies N Y Y Y Y Y Y
Hartman-Boyce 2018 Nutrition /obesity “to identify combinations of intervention components associatedwith statistically significant changes (P < 0.05) in the desired direction for at least one of the foods targeted by the intervention […] to augment the narrative review.” Crisp Any Food store consumers Food purchasing interventions Published analyses (systematic review) Quant Intervention 89 interventions, reported in 35 studies Sucessful implementation of intervention (“statistically significant changes in the desired direction for ≥1 of the foods targeted by the intervention”) N Y N N N Y Y
Kahwati 2011 Nutrition /obesity & PA “to identify facility conditions or combinations of conditions associated with larger 6-month patient weight-loss outcomes. QCA is a method that allows for systematic cross-case comparison to better understand causal complexity.” Crisp USA Adults (Veterans exposed to the MOVE! Weight Management Program) MOVE! Weight Management Program for Veterans Primay data and secondary data Qual and Quant Organisation - health facilities Twenty-two facilities of 239 facilities eligible for selection (“Eleven sites with larger outcomes and 11 sites with smaller outcomes”) Intervention effectiveness (“larger patient weight-loss outcomes”) Y Y Y N Y Y Y
Kane 2017 Programme implementation “To determine which combination of elements of capacity is most prevalent among the awardees that successfully implemented objectives”; “Programs such as CCPW [Communities Putting Prevention to Work] invove multiple components that may work together and different program models can lead to success. This method accommodates the complexity associated with evaluating such programs.” Crisp USA Organisations Communities Putting Prevention to Work (CPPW) program Primary data (generated for study) Qual Organisation - awardee (“In most cases, the lead organization for an awardee was a city or county health department, although community-based organizations served as leads for a few awardees”) 22 of 50 community awardees Intervention implementation effectivesness (“completion of approximately 60% of work plan objectives”) Y Y Y N Y Y Y
Kien 2018 PA “We chose QCA as the method of analysis, as we investigated a small number of cases and aimed to answer a question related to the combinations of conditions and not related to the identification of the independent influence of a variable. Furthermore, we were interested, in whether different combinations of causal conditions could lead to the same outcome.” Fuzzy Austria Children and Young People School based health promotion programme: “Classes in Motion” Primary data (generated for study) Qual and Quant Classes 24 classes out of 26 classes that received the intervention Percent of children showing an improvement in emotional and social school experience. Y Y Y Y Y Y Y
Lubold 2017 Breastfeeding “to examine the combinations of conditions leading to both high and low national breastfeeding initiation” Fuzzy OECD countries Mothers Baby-Friendly Hospital Initiative (BFHI) Secondary data Quant Country 18 countries of 34 OECD countries Percent of breastfeeding initiation N Y N Y Y Y Y
Lucidarme 2016 Programme implementation QCA as most appropriate method because “we have a combination of a relatively large number of determinants and a small number of cases.”; “a large amount of in-depth qualitative data was collected and CCMs [Configurational comparative methods] are able to deal with such large amounts of qualitative data.” Crisp Flanders, Belgium Organisations Local health promotion networks (LHP) Primary data (generated for study)
Secondary data
Qual and Quant Network 13 of 13 LHPs Composite score of effectiveness of network from 5 measures at Community level (Awareness and change in awareness of ‘10,000 steps’ programme) and Network level (measures of participation, and actions at municipal and regional level) Y Y Y Y Y Y Y
McGowan 2019 Mental health & Community Engagement “QCA is therefore of potential use in the evaluation of the effectiveness of complex public health interventions as applied to small populations” Crisp UK, ‘Big Local’ (BL) areas (disadvantaged areas in UK) Adult participants in BL partnerships (residents, workers, volunteers in locality) A community-led empowerent initiative Primary data (generated for study) Quant Person /individual 48 participants of 65 participating in a Wave 2 survey Improvement in mental health N Y Y N Y N N
Melendez-Torres 2018 Nutrition /obesity & PA “we aimed to understand why some interventions appeared to work better than others, or, whether specific combinations of WMP [weight management program] features were associated with greater intervention effectiveness. [...] QCA was particularly suitable [...] as it facilitates the identification of configurations of various intervention and other contextual features that are (or are not) present when the intervention has been evaluated and found successful (or not) in obtaining a desired outcome” Fuzzy Any (but notes in original review that over half of included studies were from US) Overweight or obese adults Weight management programs Published analysis Quant Intervention 20 (10 most effective; 10 least effective) from 40 intervention arms within 30 trials Pathways to high (and low) intervention effectivess N Y Y Y Y Y Y
Parrott 2018 Nutrition/obesity “QCA is much more suited to answering the question: For whom and under what conditions does the intervention make a difference” Crisp Any Children and Young People Pediatric weight management (PWM) interventions Published analysis (systematic review) Quant Intervention component 209 separate treatment arms from 99 controlled trials, were included in this analysis. Positive weight status outcomes N Y Y N Y Y Y
Roberts 2018 Vaccine adoption and implementation “Qualitative comparative analysis (QCA) is a formalized qualitative analytic approach that can be leveraged to determine which sets of state policies may be necessary or sufficient for high state-level HPV vaccination uptake. States have enacted multiple, often overlapping policies that may influence HPV vaccine uptake; QCA is well suited for characterizing these complex relationships.” Fuzzy USA Children and Young People State policies Secondary data Quant Region/subnational area 51 (of 50 States + Washington DC) Higher levels of HPV vaccine uptake in adolescent boys and girls N Y Y N Y Y Y
Thomas 2014 Community engagement & Breastfeeding “the context for this paper: a need to identify important components of interventions when making commissioning decisions, but a lack of established methods of synthesis which enable such investigations. We therefore examine an analytical technique, [QCA] which has been designed to overcome some of the limitations outlined above.” Fuzzy OECD countries Expectant and new mothers Community engagement programmes directed toward expectant and new mothers to promote breastfeeding. Published analysis (systematic review) Quant Intervention 12 studies as a subset of a review that included 319 studies Intervention effectiveness (“membership in the set of highly effective interventions”) N Y Y Y Y Y Y
Warren 2013, 2014 Chronic condition management To capture complexity (particularly on how policy interventions work across heterogeneous contexts) (2013); “QCA addresses multiple causation” (2014) Crisp UK - North-East England Adult working age recipients of incapacity benefit (IB) Case management to help benefit recipients return to work Primary data Quant Individual 131 participants receiving the intervention Self-reported health (EQ. 5-S score) improves/does not improve relative to UK population Y Y N Y Y Y Y