This study was completed in three phases (Fig. 1). Methods and results are reported below by study phase. Phase 1 included conceptualization and survey development assessed for content validity. Phase 2 included a pre-post assessment and reliability testing (test-retest) with stakeholders from two completed whole-of-community interventions using retrospective reporting [23]. Phase 3 included survey modifications and a second prospective reliability assessment.
Phase 1: Survey development
The COMPACT Stakeholder-driven Community Diffusion Survey included three modules to (a) assess the network structure of stakeholders’ professional relationships related to childhood obesity prevention efforts, (b) knowledge about childhood obesity prevention, and (c) engagement with the issue.
Part A: Social networks
In Stakeholder-driven Community Diffusion, social networks represent pathways for knowledge and engagement diffusion. The survey was designed to allow respondents to name up to 20 people with whom they had “discussed issues related to childhood obesity” during an intervention [24]. Due to the retrospective nature of the initial survey and to diminish likelihood of inaccurate recall, we used two name generation methods (free recall and a roster of stakeholders who had already provided informed consent for the survey) in a three-stage procedure (free recall, roster identification, final free recall) [25]. This approach was used to capture the complete network of bounded stakeholder groups (e.g., steering committees) and stakeholders’ broader networks when exploring community-wide connections [25].
Part B: Knowledge
We conceptualized knowledge as stakeholders’ understanding of community-wide efforts to prevent childhood obesity. We identified five domains from completed intervention trials that reduced unnecessary weight gain among children [6, 7, 26]:
-
1.
The problem of childhood obesity (“Problem”)
-
2.
Modifiable determinants of childhood obesity and level of social ecology to address them, e.g., individual behavior change versus environment and policy change (“Intervention factors”)
-
3.
Stakeholders’ roles in the whole intervention, what others are doing, and knowledge of multi-setting components (“Roles”)
-
4.
How to intervene to achieve sustainability (“Sustainability”)
-
5.
Available resources (“Resources”)
We conducted comprehensive literature reviews (peer-reviewed and grey) to source relevant instruments and survey items measuring aspects of community readiness, group dynamics, coalitions, and community-based participatory research (CBPR) to adapt and apply to the identified domains. For “resources”, we adapted four items from the Community Capacity Index [27] and the Community Readiness Handbook [21]. For the remaining domains, we identified eight items from the CBPR Conceptual Model matrix of variables and instruments [12, 17, 28] and the coalition literature. Five research team members with experience in community-based interventions scored items to assess content validity. Scoring resulted in disagreement on items to include. Through iterative critique and feedback, the team developed new fact-based, multiple-choice items for domains 1–4 (four items per domain). The knowledge domain included 20 total items.
Part C: Engagement
We conceptualized engagement as a latent construct representing stakeholders’ enthusiasm and agency for preventing childhood obesity in their community. Our Stakeholder-driven Community Diffusion theory suggests that engagement motivates stakeholders to share their knowledge with others, and represents stakeholders’ desires and ability to translate their knowledge into effective action for whole-of-community interventions.
We used the CBPR Model to identify domains describing stakeholder engagement [12, 28]:
-
1.
Exchange of skills and understanding (“Dialogue and mutual learning”)
-
2.
Willingness to compromise and adapt (“Flexibility”)
-
3.
Ability or capacity to have an effect on course of events, others’ thinking, and behavior (“Influence and power”)
-
4.
Action of directing and being responsible for a group of people or course of events (“Leadership and stewardship”)
-
5.
Belief and confidence in others (“Trust”)
We used 46 items from existing instruments cited in the CBPR Model to construct an engagement scale [17]. We also conducted a secondary search in Scopus, PubMed, and the National Cancer Institute’s Team Science Toolkit [29] for community and group partnership tools, yielding 104 total items from 20 instruments.
Six research team members evaluated the 104 items for content validity. They scored items from 0 to 2 points (0 = no; 1 = maybe; 2 = yes), with a maximum per-item score of 12 points. Item scores ranged from 3 to 11 points (mean = 7.2; SD = 1.8). We eliminated items with low scores (≤ 6 points; n = 37) and/or if an item scored lower than a similar item. We retained 50 items from 17 instruments: dialogue and mutual learning (11), flexibility (8), influence and power (4), leadership and stewardship (22), and trust (5) (Additional file 1: Table S1A). We set response options to a 5-point agree/disagree Likert scale and adapted wording to fit the context of whole-of-community childhood obesity prevention interventions.
Phase 2: Retrospective study
Methods
Participants
Respondents were members of stakeholder groups involved in two completed whole-of-community childhood obesity interventions: Shape Up Somerville (SUS) [6] and Romp & Chomp (R&C) [7]. Both interventions demonstrated measured reductions in childhood obesity prevalence. SUS was a community-based environmental change intervention from 2003 to 2005 targeting early elementary school children in Somerville, Massachusetts, USA. The SUS Community Advisory Council included stakeholders from academia, public schools, foodservice, local health department, community-based organizations, and met every 2–4 months throughout the intervention. R&C was a capacity-building and environmental intervention from 2004 to 2008 targeting children from birth to five years in Geelong, Victoria, Australia. The R&C Management Committee [30] included stakeholders from academia, local health department, government, department of human services, and the local kindergarten association, and met every 1–2 months.
Procedures
We identified potential participants’ names from historical SUS and R&C records and meeting minutes, and then acquired current contact information (email and/or telephone) via records, existing contacts, and the internet. We first contacted participants for informed consent. Upon providing consent, we invited participants to complete the web-based (Qualtrics) survey in May–June 2015 for SUS and August–October 2015 for R&C.
To aid participants’ memories in what life was like during the interventions, the surveys started by listing historical milestones at the local, state, and national level (e.g., elected government officials, major sports victories). This was followed by an optional, open-ended question that asked participants to “write any names, phrases, or keywords that describe what was going on in your life” during the intervention period. We informed participants that this response would not be retained and that the purpose was to help them provide more accurate recalled responses [24].
We then asked participants to identify social relationships and to report their own levels of knowledge and engagement related to childhood obesity prevention at the start (T1) and end (T2) of their involvement in SUS or R&C. Time was based on intervention involvement due to varying participation and attrition in stakeholder meetings. Participants reported their gender, current age, education, and affiliated organizational sector (e.g., school administration) at the start of the intervention.
To assess the test-retest reliability of the knowledge and engagement survey components, we asked participants to complete a second web-based survey, one week after the first survey.
In the SUS study, we offered participants up to $49 (electronic Amazon gift card) for completing both test-retest surveys. Consistent with usual practices in Australian studies of this type, no monetary incentive was offered to R&C participants. Procedures for individuals participating in research were approved by the Tufts University Institutional Review Board and the Deakin University Human Ethics Advisory Group for the SUS and R&C studies, respectively.
Data analysis
Demographics
We calculated frequencies for categorical variables (gender, education, organizational sector affiliation) and means and standard deviations (SD) for participant age.
Social networks
We extracted online data from the three-stage name generator of childhood obesity ‘discussion’ networks and imported to the [sna], [igraph], and [network] packages in the R programming language to conduct descriptive analyses and produce sociograms [31,32,33]. In the sociograms, participants are represented as nodes and are connected by a directed tie to represent a discussion relationship. Visualizations demonstrate structural attributes of networks and are useful in generating hypotheses about pathways for knowledge and engagement diffusion. Calculated descriptive connectivity statistics included number of nodes and ties, density (the proportion of ties to the number of possible ties between node pairs), and in-degree centralization (an indicator of node connectivity, or the extent to which one or few nodes in the network receive a high number of ties).
Knowledge and engagement
We calculated composite and domain-specific scores (mean, SD) at both time points. Knowledge domains 1–4 each had four multiple-choice questions with a maximum score of four points per domain (− 1 = incorrect response; 0 = not sure; 1 = correct response). Knowledge domain 5 had four 4-point agree/disagree Likert-scale items (− 1 = strongly disagree; − 0.5 = disagree; 0.5 = agree; 1 = strongly agree) with a maximum score of 4 points. The maximum composite score was 20 points. There were 50 5-point agree/disagree Likert-scale engagement items. We weighted scores based on number of items per domain to ease domain-domain comparisons, with a maximum composite score of 25 points (1 = strongly disagree to 5 = strongly agree). We used paired t-tests and corresponding 95% CIs to assess change in mean knowledge and engagement scores from T1 to T2 within interventions (test survey data used).
Knowledge and engagement reliability
We analyzed reliability data from T1. We assessed item-specific test-retest reliability using Cohen’s weighted Kappa statistic (κw) [34]. We calculated intraclass correlation coefficients (ICCs) and within-subject coefficients of variation (WSCV), each with 95% confidence intervals (CIs), to inform composite and domain-specific reliability. We used Cronbach’s alpha (α) to assess composite and domain-specific engagement internal scale consistency. We did not calculate scale consistency for the retrospective knowledge measure, as items in domains 1–4 assessed fact-based knowledge and were not expected to relate. Data were analyzed using SAS 9.3 (Cary, NC) and StataSE 14 (College Station, TX).