Skip to main content
Download PDF

Implementation fidelity to a behavioral diabetes prevention intervention in two New York City safety net primary care practices

BMC Public Health volume 23, Article number: 575 (2023) Cite this article

Abstract

Background

It is critical to assess implementation fidelity of evidence-based interventions and factors moderating fidelity, to understand the reasons for their success or failure. However, fidelity and fidelity moderators are seldom systematically reported. The study objective was to conduct a concurrent implementation fidelity evaluation and examine fidelity moderators of CHORD (Community Health Outreach to Reduce Diabetes), a pragmatic, cluster-randomized, controlled trial to test the impact of a Community Health Workers (CHW)-led health coaching intervention to prevent incident type 2 Diabetes Mellitus in New York (NY).

Methods

We applied the Conceptual Framework for Implementation Fidelity to assess implementation fidelity and factors moderating it across the four core intervention components: patient goal setting, education topic coaching, primary care (PC) visits, and referrals to address social determinants of health (SDH), using descriptive statistics and regression models. PC patients with prediabetes receiving care from safety-net patient-centered medical homes (PCMHs) at either, VA NY Harbor or at Bellevue Hospital (BH) were eligible to be randomized into the CHW-led CHORD intervention or usual care. Among 559 patients randomized and enrolled in the intervention group, 79.4% completed the intake survey and were included in the analytic sample for fidelity assessment. Fidelity was measured as coverage, content adherence and frequency of each core component, and the moderators assessed were implementation site and patient activation measure.

Results

Content adherence was high for three components with nearly 80.0% of patients setting ≥ 1 goal, having ≥ 1 PC visit and receiving ≥ 1 education session. Only 45.0% patients received ≥ 1 SDH referral. After adjusting for patient gender, language, race, ethnicity, and age, the implementation site moderated adherence to goal setting (77.4% BH vs. 87.7% VA), educational coaching (78.9% BH vs. 88.3% VA), number of successful CHW-patient encounters (6 BH vs 4 VA) and percent of patients receiving all four components (41.1% BH vs. 25.7% VA).

Conclusions

The fidelity to the four CHORD intervention components differed between the two implementation sites, demonstrating the challenges in implementing complex evidence-based interventions in different settings. Our findings underscore the importance of measuring implementation fidelity in contextualizing the outcomes of randomized trials of complex multi-site behavioral interventions.

Trial registration

The trial was registered with ClinicalTrials.gov on 30/12/2016 and the registration number is NCT03006666.

Peer Review reports

Background

The NationalInstitutes for Health recommends fidelity measurement in health behavior studies because without the knowledge of implementation fidelity, it may be impossible to draw correct inferences about the effectiveness of or to replicate an intervention [1]. Erroneous inferences may lead to invalid conclusions about the relationships between the intervention components and outcomes, especially in pragmatic studies of complex behavioral interventions [2, 3]. However, most pragmatic trials do not report systematic assessment of implementation fidelity and very few assess how participant and contextual factors may influence fidelity. For example, a review found that only 3.5% of complex behavioral interventions for drug abuse adequately addressed intervention fidelity to planned core intervention components [4].

CHORD (Community Health Outreach to Reduce Diabetes), launched in 2017, is a cluster-randomized, pragmatic trial that tests a Community Health Worker (CHW) driven intervention to promote healthy lifestyle changes to reduce the incidence of Type II diabetes mellitus (DM) among pre-diabetic patients cared for in two safety-net primary care sites that serve veterans, or uninsured and Medicaid populations [5]. We sought to improve the understanding of subsequent CHORD trial results, as well as to learn about how contextual factors affected implementation of the CHORD intervention, by assessing fidelity to its four core components across the study’s two participating sites.

By reporting on the fidelity of the CHORD study implementation, our study attempts to fill a gap in systematic documentation and reporting of implementation processes of complex behavioral interventions [4, 6]. In addition to clarifying causal mechanisms, assessment of implementation fidelity can provide critical information to guide future implementation of the intervention [4, 7,8,9]. Examining implementation fidelity is particularly necessary for multicomponent, complex behavioral interventions because of their higher likelihood of deviation from the protocol [9]. Few studies conduct concurrent fidelity assessment as opposed to retrospective. Concurrent process evaluations are important as they can capture implementation experiences in real time [9]. By building in the collection of fidelity measures and moderating factors in CHORD implementation, this study seeks to address this gap in the literature. Moreover, the study responds to the general calls for conducting quantitative evaluations of fidelity in intervention studies [9].

Figure 1 shows the logic model for the CHORD intervention. Evidence supports engagement of CHWs by safety net primary care practices for improving health, knowledge, behaviors and outcomes for underserved communities [10, 11]. The CHORD CHWs deliver evidence-based lifestyle interventions known to prevent DM among people with prediabetes [12,13,14]. CHWs are trusted members of the community who share lived experience and are familiar with community resources and norms [5, 10, 15]. Culturally congruent CHWs can bridge gaps between communities and health care systems to facilitate positive behavioral and lifestyle changes [12,13,14]. The evidence-based lifestyle interventions implemented in CHORD were based on guidelines from the National Diabetes Prevention Program (DPP) with a focus on healthy eating and physical activity [16]. People with prediabetes who participated in such a structured lifestyle change program reduced their risk of DM by 58% [16].

Fig. 1
figure 1

Logic Model and the Theory of Change of the CHORD Intervention

Full size image

CHORD is a complex behavioral intervention, where CHWs facilitate patients’ adoption of guideline-based lifestyle changes by using four behavioral intervention components: patient goal setting; education topic coaching; facilitating primary care (PC) visits; and referrals to address social determinants of health (SDH) [17]. We assessed fidelity to these four behavioral intervention components and examined factors affecting implementation fidelity using the Conceptual Framework for Implementation Fidelity (CFIF) [6, 7]. The overarching purpose of this analysis is to inform researchers and program implementors about the level of fidelity achievable of an intervention such as CHORD in the context it was implemented.

Methods

Trial participants and description

The CHORD trial’s priority population was patients with prediabetes receiving care at two PC clinics, the Manhattan campus of the VA NY Harbor Healthcare System (VA), and Bellevue Hospital Center (BH) of New York City’s municipal hospital system [5]. Protocol details have been described previously [5]. This study was approved by the New York University Langone Health and the Veterans Affairs Institutional Review Boards and was registered with clinicaltrials.gov (NCT03006666) on 30/12/2016. Informed consent was obtained from all the participants in the study. All the procedures were followed in accordance with the relevant guidelines (eg. Declaration of Helsinki) along with the rest of the ethical declarations. Findings were reported in accordance with the StaRI checklist for implementation studies [18].

Briefly, all PC clinicians within each site were randomized to intervention or usual care. Eligible patients were seen by one of the study clinicians (at least 1 PC visit in the past 2 years at the VA or at least 3 PC visits in the past 2 years at BH), were between ages 18–75 years, had ≥ 1 glycosylated hemoglobin (HbA1c) between 5.7–6.4% at baseline, no prior DM diagnosis, and ability to communicate in English or Spanish. Two CHWs, members of the study team assigned to each site, consented and enrolled eligible intervention patients, who then completed an intake survey, and began a 12-month intervention during which at least monthly encounters were planned between the CHW and patient. The two CHWs at each site made up to ten attempts to call and enroll each intervention group patient. A successful encounter was defined as when the CHW was able to speak with the patient in-person or by phone, or when a letter or a text message was delivered (no evidence of failure to deliver was apparent). During these encounters, CHWs delivered one or more of four core intervention components.

Implementation of CHORD behavioral components

In the first component, CHWs established individualized goals with each patient and completed a 6-item, Patient Activation Measure (PAM) [19, 20]. These goals were then translated into a health action plan (HAP) tailored to each patients’ goals, PAM score, and preferences. Second, using the HAP, CHWs chose among 22 educational topics organized into 5 modules (prediabetes, healthful eating, MyPlate [21] and plate portions, physical activity, and stages of change) to conduct education sessions with patients, and provide information packets. Third, CHWs called or met patients before and/or after PC visits to encourage them to discuss diabetes prevention with their clinician or to review their after-visit summaries regarding diabetes. Finally, CHWs facilitated referrals as needed to hospital or community-based programs to help support behavioral change or address identified social needs.

CHW training and fidelity monitoring

To facilitate and standardize the implementation of the intervention, CHWs received comprehensive training and then ongoing feedback during weekly team meetings and case review sessions. To address behavioral components, CHWs completed training on coaching competencies, motivational interviewing, mental health and nutrition needs, weight management and healthy lifestyle programs, multicultural competence, diabetes and diabetes prevention, elderly and loneliness, and technical trainings on using Excel, Outlook, and REDCap (Research Electronic Data Capture).

Data collection

Figure 2 shows the modified version [6] of the Conceptual Framework for Implementation Fidelity (CFIF) [7] which guided our data collection and analyses. The key indicator of implementation fidelity according to CFIF is adherence, defined by Consolidated Framework of Implementation Research (CFIR) as “whether a program service or intervention is being delivered as it was designed or written.” [7] We measured adherence as coverage(“what proportion of target group participated in the intervention”) [6], content(“was each of the intervention components implemented as planned”) [6], and dosage, which includes duration and frequency(“amount of an intervention received by participants” [7], or “were the intervention components implemented as often and for as long as planned”) [6]. We also measured two factors that could moderate fidelity: Context(“what factors at the political, economic, organization, and work group levels affected the implementation”) [6], and participant responsiveness (“how were the participants engaged with the intervention”).

Fig. 2
figure 2

The Modified Version of the Conceptual Framework for Implementation Fidelity that guided Fidelity Assessment of the CHORD intervention (adapted from Carroll et al. and Hasson et al.)

Full size image

Data on the CFIR implementation fidelity elements identified above were collected from the start of the CHORD trial (December 2017) until October 2019, using standardized Research Electronic Data Capture (REDCap) [22] forms completed by CHWs for each patient to record their demographics, information on outreach, enrollment, and intake, and weekly logs on encounters. Electronic Health Records (EHR) from VA and BH provided descriptive data on study participants and their healthcare utilization.

Measures of fidelity and fidelity moderators

Table 1 shows the data sources for measures of fidelity and fidelity moderators. Coverage was the percent of patients who completed an intake and hence were able to receive the intervention. Content adherence was the percent of patients who received each of the four intervention components. Dose frequency was how much of the four components was received measured as an average or median across all intervention patients and dose duration measured for how long the intervention was delivered.

Table 1 Study measures and their data sources mapped to the framework constructs
Full size table

Measures of hypothesized moderators:

  1. 1)

    Participant responsiveness, measured as the baseline Patient ActivationMeasure (PAM) score, which measures the extent to which patients are activated for participating in managing their health and healthcare including seeking health information and readiness to change [23,24,25]. According to Hasson et al., “the uptake of the intervention depends on the responsiveness of those receiving it” (page 2). [6] We hypothesized that people with higher PAM score (indicating higher activation to participate in health) will be more responsive and motivated to participate and receive the intervention.

  2. 2)

    Context, measured as the two clinical sites. As the intervention components were designed to be delivered by CHWs to patients, the clinical site was selected to capture systematic differences in the sociodemographic and economic context of patient populations at the two sites. It should be noted that, organizational differences between the two sites were not directly hypothesized to moderate fidelity measures of CHORD intervention which relied on individual encounters between CHWs and patients. Therefore, clinical site reflects differences in patients’ social profiles at the two sites. We did not assess the role of other specific patient characteristics that are not explained by their chosen site of healthcare.

Analysis

Fidelity measures of coverage, content and dose were reported using applicable descriptive statistics including percentage for categorical variables or median with inter-quartile ranges (IQRs) for continuous or count variables. To evaluate fidelity moderation, we computed unadjusted p-values from Chi-square or Mann–Whitney U test and adjusted p-values from regression models that controlled for patient gender, language, race, ethnicity and age. To assess moderation by PAM score, we dichotomized the score as above or below the median score. As a sensitivity analysis, we also compared patients with a total maximal score of 24 vs. < 24. The first regression models treated PAM score as the primary independent variable, and the second set treated clinical site as the primary independent variable. Each set of models included separate models for each fidelity measure as the dependent variable. Logistic regression models were used for binary measures of coverage and content adherence (received or not received the intervention component). Linear regression models were used for fidelity measures of dosage that were continuous, and negative binomial regression models were used to model the fidelity measure of dosage that was a count variable.

Among patients who completed intake (n= 444), the denominator for our fidelity assessment, 32 patients (7.2%) with missing PAM scores were excluded from fidelity moderation analysis by PAM score. Chi-square test or Mann–Whitney U test, as appropriate, were used to compare two population groups—how eligible/outreached patients differed by their enrollment status and how enrolled patients differed by their intake completion status – on their gender, language, race, ethnicity, age at outreach and PAM score. All quantitative analysis was conducted in the R statistical software environment [26], and statistical significance required an alpha < 0.05.

Results

Fidelity measures

Coverage

Among 1449 eligible patients assigned to CHWs for outreach, we excluded 471 patients with incorrect/no contact information or who could not be reached after ten phone calls, 416 who declined to participate, and another three found to be ineligible. Of the remaining 559 patients who enrolled in the intervention arm, 444 (79.4%) completed an intake survey and were eligible to receive the intervention, hence comprised the analytic sample for fidelity assessment (Fig. 3). The 559 enrolled and the 890 unenrolled patients differed in their primary language, race, ethnicity, implementation site and median age at outreach. Those who enrolled were more likely to be Spanish speaking, Black and Hispanic (Supplemental Table 1). However, among those enrolled, patients completing intake vs. those not completing intake were similar (Supplemental Table 2). Among intake patients, 362 (81.5%) established a HAP, the first core component.

Fig. 3
figure 3

CONSORT Flow Diagram for the Intervention Arm

Full size image
Table 2 Implementation fidelity (Content Adherence) for each intervention component
Full size table

Content adherence

Eighty percent of patients had ≥ 1 successful encounter with the CHW (Table 2). About 80% of the patients completing an intake established ≥ 1 goal for their HAP, received coaching on ≥ 1 education topic or had ≥ 1 PC visit, indicating high content adherence to these three core components. Forty percent received coaching on all five education modules. Forty-five percent received a SDH referral, most of which were for a healthcare visit, with other referrals to facilitate healthy lifestyle, employment/workforce training, social security benefits, or mental health services. Overall, 34.9% received all four core components in some capacity.

Dose – frequency and duration

 Among the 362 patients who established a HAP, more than half established three goals (median 3; IQR: 2, 3), more than half completed ≥ 1 goal, and 25% completed three goals (median:1; IQR: 0, 3). The median number of education sessions delivered was 18 (IQR: 8, 33) and the median number of education modules covered was four of five (IQR: 2, 4). Median number of referrals was one (IQR: 1, 3), with 25% receiving ≥ 3. The median number of PC visits was three (IQR: 2,5). A total of 4,072 encounters with patients were documented by CHWs, of which 55.6% were successful. The median number of successful encounters per patient was five (IQR: 3, 8) (Table 3). Among them, 9.9% had 12 successful encounters. The median duration of follow-up was 411 days [IQR: 341, 446], as we extended the overall intervention period to > 12 months for some patients.

Table 3 Implementation fidelity (frequency and duration) for each intervention component
Full size table

Impact of moderators on fidelity measures

Patient activation

The median PAM score was 18 of a maximal score of 24. None of the fidelity measures were moderated by PAM score when dichotomized at the median (Supplemental Table 3) or at the highest score of 24 versus < 24 (results not presented).

Context

The implementation of the CHORD intervention was moderated by clinical site, with 60% of patients from BH and 40% from VA. VA patients had higher coverage and overall content adherence than BH. But a greater percentage of BH patients received coaching on all education modules and received all four core components. Three content adherence measures, including percent of patients who received ≥ 1 referral, ≥ 1 successful encounter and ≥ 1 PC visit, were similar at the two sites. Three dose-frequency measures, including median number of PC visits (4.0 VA vs. 4.0 BH), number of education modules covered (4.0 VA vs. 4.0 BH), and the number of successful encounters (4.0 VA vs. 6.0 BH), differed between the two sites (Supplemental Table 4).

Discussion

In the CHORD study, we hypothesized that trained CHWs, through individualized goal setting, educational coaching, and facilitated referrals, would support positive lifestyle changes and prevent the onset of diabetes among patients with prediabetes. However, an intervention may not affect lifestyle change if it deviates from its protocol during implementation. In this concurrent process evaluation of the CHORD intervention, we examined implementation fidelity and fidelity moderators. Our analysis demonstrated moderate to high (depending on intervention component) rates of implementation fidelity of CHORD, and moderation by implementation site. The level of implementation fidelity achieved in this intervention is likely to be sufficient to justify inferences drawn about the impact of the intervention on the outcomes. However, with less than 100% adherence and the variability observed in these components, as is likely common in complex behavioral interventions, the analysis of CHORD outcomes will provide an opportunity to explore the relative importance and impact of each of these components, accounting for their degree of implementation fidelity. Our use of quantitative methods for fidelity assessment will allow us to use these measures in outcome data analyses to determine the role of fidelity in observed outcomes [1, 27].

We found that CHWs completed an intake with nearly 80% of the patients enrolled in the intervention arm, and three of the four core components (goal setting, education and PC visits) were delivered to nearly 80% of the patients. Even though coverage and content adherence were moderate to high, there was high variability in the dosage. While the fourth core component in our intervention, referrals for social determinants of health and promoting access to healthy lifestyle choices, was delivered to only 45% of the patients, a quarter received 3 or more referrals. As referrals were designed to be tailored to each patient’s circumstance, the dosage of referrals cannot be directly associated with outcomes as more referrals are not necessarily better. Further, we do not know if the remaining 55% patients who did not receive a referral needed one.. We strived to deliver all core components for each patient, but we had not established, a priori, thresholds for adherence for core components among study participants. Our analysis explores the extent to which we attained implementation success as measured by percent of patients for which the core components were delivered. An implementation fidelity analyses of a randomized trial delivering a complex care continuum intervention for frail elderly people in Sweden also reported high variability in the adherence to intervention components. On a 4-point scale (never, seldom, sometimes, often, always), while most components were reported to be ‘always’ delivered, several were only ‘seldom’ delivered [28]. In another study implementing a program to promote care aide involvement in formal team communications about resident care, the adherence was comparable to our study. Nearly 63% of the units participated in all three workshops and nearly 80% of the units participating were delivered the inter-team activities [9]. However, some measures were higher than our study – 93% of the units in this trial completed goal setting [9]. In another trial implementing a complex behavior change intervention to enable participants living with long-term musculoskeletal pain to improve their quality of lives, although the median score on adherence to components ranged from 1.67 to 2.00 on a scale of 0 to 2.00, overall course score of 2.00 showed 100% adherence to these components [29]. Higher fidelity in interventions such as those targeting pain can be explained by the nature of the intervention. According to CFIR, the relative advantage of the intervention plays a key role in implementation success: [30] “If users perceive a clear, unambiguous advantage in effectiveness or efficiency of the innovation, it is more likely the implementation will be successful….benefits of the innovation must be clearly visible (observable) to assess relative advantage” [31]. The benefits of our intended outcome, diabetes prevention in patients with pre-diabetes, are more intangible and hence could have reduced patients’ motivation to participate fully in the intervention components.

We assessed the role of patient motivation in our study by examining moderation of fidelity measures by PAM score. Null findings on moderation by PAM score was unexpected because patient activation measured by PAM scores has been found to be positively associated with engaging in healthy lifestyle behaviors and exhibiting readiness to change [25]. PAM score at intake might be limited in its representation of the concept of ‘participant responsiveness’ as defined in the CFIF [6, 7]. Patient activation might not be directly associated with the perception of patients about the effectiveness of an intervention in improving health, even though they value health. Finally, using an adapted, shortened version of the PAM [19, 20], may not have fully measured this construct of activation.

The pragmatic nature of the CHORD trial permitted and promoted continuous adaptations. While this improved the meaningfulness of our intervention, it resulted in deviations from the original intervention protocol, decreasing fidelity. We adapted implementation in response to ongoing developments and situations. For example, the recruitment and follow-up process changed significantly after BH adopted a new EHR system in the second year of the CHORD trial. Provider communication and follow-up was adapted to align with their work schedule. In five cases, CHWs delivered education sessions before establishing goals as an engagement strategy to demonstrate how the intervention could be useful and encourage patients to set goals. In addition, the number of patients with a successful encounter was less than the number of patients who completed at least one goal or the number of patients who received coaching on at least one education topic. This occurred because for some patients, CHWs completed intake, established goals and provided some education in the same intake visit, but were unable to follow-up with the patient after repeated attempts, resulting in patients with goals and education but few successful encounters. The median number of successful encounters was 5 and about 10% patients had 12 or more successful encounters. As the CHORD protocol planned at least monthly contacts between a CHW and patient, these findings suggest that CHWs divided their time between meeting motivated patients less frequently and checking-in more frequently with patients who perhaps had more complex social needs. All these adaptations could have impacted our fidelity. Future analysis will focus on sharing these adaptations or natural deviations in implementation of a pragmatic trial.

Our finding of moderation by the implementation site underscores the need to account for site-based differences in patient characteristics such as their social risk profiles, when implementing evidence-based interventions. Although BH and VA are both safety-net settings, they differ in terms of their patient populations.BH is the flagship of New York City’s large, municipal hospital system, and serves a diverse, multicultural and multilingual population with high numbers of poor and racial/ethnic minorities, many of whom are uninsured or on Medicaid. VA NY Harbor is smaller, and provides care to veterans (mostly older, White, male, and English speakers) and is funded federally. Variation in the complexity of patient populations have historically necessitated innovation, and the differences in fidelity at the two sites could be reflecting such adaptations by CHWs to meet the needs of their patient populations. Earlier studies have found different implementation fidelity across different organizations [32, 33], without any impact on outcome differences at patient-level [27]. These findings, including ours, suggest that an effective implementation fidelity might be based on the local organization’s conditions and therefore might require implementors to consider local adaptations when scaling-up evidence-based behavioral interventions. It is also important to note that, while our use of implementation site to measure ‘context’ reflects different healthcare ecosystems in terms of the patients served, neither of these clinical sites nor the scope of our analysis fully measures ‘context’ as conceptualized in the CFIF. A comprehensive assessment of patients’ social environments is an important consideration in a CHW-led intervention such as CHORD, which relies on patients’ achieving behavior change in the conditions of their everyday lives.

These differences in patient populations at the two sites could also explain the different participation rates at the two sites. Utilizing CHWs as a healthcare workforce is an equity-oriented approach to improving healthcare access and health outcomes among underserved populations [34]. Given their purpose, training and demographic composition (belonging to racial-ethnic minority groups), it is not surprising that the patients who enrolled in our CHW-led intervention were more likely to be Spanish speaking, Black, Hispanic, and seeking care at Bellevue hospital – these population subgroups have historically been excluded or marginalized from healthcare systems and face challenges in accessing resources to live a healthy life. Our CHWs were also either black or Hispanic, and those at the VA were veterans.

Study limitations

First, as a pragmatic trial, the CHORD implementation did not start or end on fixed days, because CHWs maintained continued contacts with their patients. As a result, some interventions, such as referrals, were delivered outside of the intervention period. We included them in this analysis if they were recorded by CHWs. Second, fidelity measures can be intervention specific. Therefore, the measures used in this study might not directly translate to other complex interventions. Third, one important aspect of implementation is how well the participants engage with the intervention. In our study, while we measured the delivery of core components from the perspective of CHWs, we could not assess the extent to which delivered interventions were received by patients. Fourth, with two CHWs per site, differences in approach and skill by the CHWs, despite their uniform training and monitoring, may have contributed to the differences by site. Fifth, with four CHWs, we did not have enough variation in race/ethnicity to track patient enrollment by the concordance of their race/ethnicity with that of CHWs’. Sixth, we report fidelity measures across the three waves of CHORD implementation, but do not examine changes in fidelity over time. Finally, while we report on fidelity delivery, our study did not measure fidelity receipt or fidelity enactment [9], or assess qualitative aspects of fidelity, such as the quality of goal setting or the comprehensiveness of education sessions beyond the coverage of the required education modules.

Study strengths and contributions

Our study adds to the limited literature with systematically reported concurrent evaluation of implementation processes of multicomponent complex behavioral interventions [4, 6]. Moreover, the study responds to the general calls for conducting quantitative evaluations of fidelity in intervention studies [9]. Use of real time data reported by the key implementors, the CHWs, adds to the validity of our analysis. Lastly, our study empirically tested the CFIF and found that the framework is a useful tool for conceptualizing and organizing measures of fidelity and their moderators. However, our process evaluation suggests that to standardize quantitative fidelity assessments, the field will benefit from further guidance on “how-to” quantitatively measure fidelity moderators.

Conclusion

Our concurrent quantitative, implementation evaluation of a complex pragmatic trial to prevent diabetes in safety-net settings, found moderate-to-high adherence to the core components of the intervention, as well as moderation of several fidelity measures by implementation site, with no impact of the baseline patient-activation measure on fidelity measures. Analyses of implementation fidelity of complex interventions such as this trial, advances the field of implementation science. This implementation evaluation will inform our analyses of the study outcomes and may be useful for other researchers conducting complex behavioral interventions.

Availability of data and materials

The datasets used and/or analyzed during the current study are not publicly available due to patient identifiers but is available from the corresponding author on reasonable request.

Abbreviations

BH:

Bellevue Hospital

CFIF:

Consolidated Framework for Implementation Fidelity

CHORD:

Community Health Outreach to Reduce Diabetes

CHW:

Community Health Worker

DM:

Type 2 Diabetes Mellitus

HAP:

Health Action Plan

IQR:

Inter-quartile range

NY:

New York

PAM:

Patient Activation Measure

PC:

Primary care

REDCap:

Research Electronic Data Capture

VA:

Veterans Administration

References

  1. Bellg AJ, Borrelli B, Resnick B, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23(5):443–51. https://doi.org/10.1037/0278-6133.23.5.443.

    Article  PubMed  Google Scholar 

  2. Dobson D, Cook TJ. Avoiding type III error in program evaluation: Results from a field experiment. Eval Program Plan. 1980;3(4):269–76. https://doi.org/10.1016/0149-7189(80)90042-7.

    Article  Google Scholar 

  3. Scanlon JW, Horst P, Nay JN, Schmidt RE, Waller JD. Evaluability assessment: Avoiding type III and IV errors. In: Evaluation Management: A Sourcebook of Readings. 1977. p. 264–82.

    Google Scholar 

  4. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res Nurs Health. 2010;33(2):164–73.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Islam N, Gepts T, Lief I, et al. Protocol for the CHORD project (community health outreach to reduce diabetes): a cluster-randomized community health worker trial to prevent diabetes. BMC Public Health. 2018;18(1):1–11.

    Article  Google Scholar 

  6. Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Sci. 2010;5(1):67. https://doi.org/10.1186/1748-5908-5-67.

    Article  Google Scholar 

  7. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ Br Med J. 2015;350:h1258. https://doi.org/10.1136/bmj.h1258.

    Article  Google Scholar 

  9. Ginsburg LR, Hoben M, Easterbrook A, et al. Examining fidelity in the INFORM trial: a complex team-based behavioral intervention. Implementation Sci. 2020;15(1):78. https://doi.org/10.1186/s13012-020-01039-2.

    Article  Google Scholar 

  10. Agency for Healthcare Research and Quality. Outcomes of Community Health Worker Interventions. 2009. (https://www.ahrq.gov/downloads/pub/evidence/pdf/comhealthwork/comhwork.pdf).

    Google Scholar 

  11. Spencer MS, Rosland A-M, Kieffer EC, et al. Effectiveness of a community health worker intervention among African American and Latino adults with type 2 diabetes: a randomized controlled trial. Am J Public Health. 2011;101(12):2253–60. https://doi.org/10.2105/AJPH.2010.300106.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Absetz P, Valve R, Oldenburg B, et al. Type 2 diabetes prevention in the “real world”: one-year results of the GOAL Implementation Trial. Diabetes Care. 2007;30(10):2465–70.

    Article  PubMed  Google Scholar 

  13. Tseng E, Greer RC, O’Rourke P, et al. Survey of primary care providers’ knowledge of screening for, diagnosing and managing prediabetes. J Gen Intern Med. 2017;32(11):1172–8.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Yarnall KS, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health. 2003;93(4):635–41.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Gore R, Brown A, Wong G, Sherman S, Schwartz M, Islam N. Integrating Community Health Workers into Safety-Net Primary Care for Diabetes Prevention: Qualitative Analysis of Clinicians’ Perspectives. J Gen Intern Med. 2020;35(4):1199–210. https://doi.org/10.1007/s11606-019-05581-8.

    Article  PubMed  Google Scholar 

  16. Centers for Disease Control and Prevention. National Diabetes Prevention Program. https://www.cdc.gov/diabetes/prevention/about.htm

  17. Mihalic SF, Fagan AA, Argamaso S. Implementing the LifeSkills Training drug prevention program: factors related to implementation fidelity. Implementation Sci. 2008;3(1):5. https://doi.org/10.1186/1748-5908-3-5.

    Article  Google Scholar 

  18. Pinnock H, Barwick M, Carpenter CR, et al. Standards for Reporting Implementation Studies (StaRI): explanation and elaboration document. BMJ Open. 2017;7(4):e013318–e013318. https://doi.org/10.1136/bmjopen-2016-013318.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Erskine NA, Gandek B, Waring ME, et al. Survivors of an Acute Coronary Syndrome With Lower Patient Activation Are More Likely to Experience Declines in Health-Related Quality of Life. J Cardiovasc Nurs Mar/Apr. 2018;33(2):168–78. https://doi.org/10.1097/JCN.0000000000000429.

    Article  Google Scholar 

  20. Schwartz MD, Jensen A, Wang B, et al. Panel Management to Improve Smoking and Hypertension Outcomes by VA Primary Care Teams: A Cluster-Randomized Controlled Trial. J Gen Intern Med. 2015;30(7):916–23. https://doi.org/10.1007/s11606-015-3204-y.

    Article  PubMed  PubMed Central  Google Scholar 

  21. US Department of Agriculture. MyPlate. 2021. (https://www.myplate.gov/).

    Google Scholar 

  22. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81. https://doi.org/10.1016/j.jbi.2008.08.010.

    Article  PubMed  Google Scholar 

  23. Hibbard JH, Mahoney ER, Stockard J, Tusler M. Development and Testing of a Short Form of the Patient Activation Measure. Health Serv Res. 2005;40(6p1):1918–30. https://doi.org/10.1111/j.1475-6773.2005.00438.x.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004;39(4p1):1005–26.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Fowles JB, Terry P, Xi M, Hibbard J, Bloom CT, Harvey L. Measuring self-management of patients’ and employees’ health: Further validation of the Patient Activation Measure (PAM) based on its relation to employee characteristics. Patient Educ Couns. 2009;77(1):116–22. https://doi.org/10.1016/j.pec.2009.02.018.

    Article  PubMed  Google Scholar 

  26. Team RC. R: A language and environment for statistical computing. 2013.

    Google Scholar 

  27. Agarwal R, Connolly J, Gupta S, Navathe AS. Comparing Medicare Advantage And Traditional Medicare: A Systematic Review: A systematic review compares Medicare Advantage and traditional Medicare on key metrics including preventive care visits, hospital admissions, and emergency room visits. Health Aff. 2021;40(6):937–44.

    Article  Google Scholar 

  28. Hasson H, Blomberg S, Dunér A. Fidelity and moderating factors in complex interventions: a case study of a continuum of care program for frail elderly people in health and social care. Implementation Sci. 2012;7(1):23. https://doi.org/10.1186/1748-5908-7-23.

    Article  Google Scholar 

  29. Mars T, Ellard D, Carnes D, Homer K, Underwood M, Taylor SJC. Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity. BMJ Open. 2013;3(11):e003555. https://doi.org/10.1136/bmjopen-2013-003555.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Sci. 2009;4(1):50. https://doi.org/10.1186/1748-5908-4-50.

    Article  Google Scholar 

  31. Research CRT-CfCM. CFIR Constructs. 2022. https://cfirguide.org/constructs/ Accessed 23 Nov 2022

  32. Augustsson H, von Thiele SU, Stenfors-Hayes T, Hasson H. Investigating variations in implementation fidelity of an organizational-level occupational health intervention. Int J Behav Med. 2015;22(3):345–55. https://doi.org/10.1007/s12529-014-9420-8.

    Article  PubMed  Google Scholar 

  33. Hoekstra F, van Offenbeek MAG, Dekker R, et al. Implementation fidelity trajectories of a health promotion program in multidisciplinary settings: managing tensions in rehabilitation care. Implementation Sci. 2017;12(1):143. https://doi.org/10.1186/s13012-017-0667-8.

    Article  Google Scholar 

  34. Ibe CA, Hickman D, Cooper LA. To Advance Health Equity During COVID-19 and Beyond, Elevate and Support Community Health Workers. JAMA Health Forum. 2021;2(7):e212724–e212724. https://doi.org/10.1001/jamahealthforum.2021.2724.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Presented in part at Society of General Internal Medicine (SGIM) 2021 conference and Academy Health ARM 2021 conference. We thank Dr. Erin Rogers, Assistant Professor at the Department of Population Health, NYU Grossman School of Medicine, for her valuable review of the manuscript.

Funding

This study has undergone peer-review by and is supported by the National Institutes of Health—National Institute of Diabetes and Digestive and Kidney Diseases under award number R18DK110740. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funding agency has not influenced the design of this study or the writing of this manuscript and will not influence the study’s implementation, data collection and analysis, and the writing of future manuscripts.

Author information

Authors and Affiliations

  1. School of Global Public Health, New York University, 708 Broadway, New York, NY, 10003, USA

    Avni Gupta

  2. Department of Population Health, NYU Grossman School of Medicine, 180 Madison Ave 2F Rm 222, New York, NY, 10016, USA

    Jiyuan Hu

  3. Department of Population Health, NYU Grossman School of Medicine, 180 Madison Ave, 2Nd Floor, New York, NY, 10016, USA

    Shengnan Huang & Michael Bergman

  4. Department of Population Health, NYU Grossman School of Medicine, 180 Madison Ave, 9-43A, New York, NY, 10016, USA

    Laura Diaz

  5. Department of Population Health, NYU Grossman School of Medicine, 180 Madison Ave, New York, NY, 10016, USA

    Radhika Gore

  6. Department of Medicine, NYU Grossman School of Medicine, 462 First Avenue, Area 2d, New York, NY, 10016, USA

    Natalie Levy

  7. Department of Medicine, NYU Grossman School of Medicine, 423 East 23Rd Street, Room 16049C, New York, NY, 10010, USA

    Michael Bergman

  8. VA New York Harbor Healthcare System, 423 East 23Rd Street, Room 16049C, New York, NY, 10010, USA

    Michael Bergman

  9. Department of Medicine, NYU Grossman School of Medicine, 462 1St Ave, New York, NY, 10016, USA

    Michael Tanner

  10. Department of Population Health, NYU Grossman School of Medicine, 180 Madison Avenue, New York, NY, 10016, USA

    Scott E. Sherman & Nadia Islam

  11. VA New York Harbor Healthcare System, 180 Madison Avenue, New York, NY, 10016, USA

    Scott E. Sherman

  12. Department of Population Health, NYU Grossman School of Medicine, 180 Madison Avenue, Suite 955, New York, NY, 10016, USA

    Mark D. Schwartz

  13. VA New York Harbor Healthcare System, 180 Madison Avenue, Suite 955, New York, NY, 10016, USA

    Mark D. Schwartz

Authors
  1. Avni Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiyuan Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shengnan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laura Diaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Radhika Gore
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Natalie Levy
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael Bergman
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Michael Tanner
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Scott E. Sherman
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Nadia Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Mark D. Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AG conceptualized the paper, wrote the first draft and finalized the subsequent revisions along with data interpretation and data presentation. JH led data management, data analyses and data presentation along with reviewing multiple versions of the manuscript to offer critical edits and comments. SH performed data management, data analyses and reviewed multiple versions of the manuscript to offer critical edits and comments. LD managed project implementation performed data management and reviewed multiple versions of the manuscript to offer critical edits and comments. RG was involved with data interpretation, data presentation and also offered critical edits and comments to the manuscript. NI and MS conceptualized the trial and the manuscript along with providing key inputs to the content and data for the paper by reviewing multiple versions of the manuscript to offer critical edits and comments. All other authors contributed to the manuscript by critically reviewing and revising the manuscript.

Authors’ information

Not applicable.

Corresponding author

Correspondence to Avni Gupta.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the New York University Langone Health (Protocol #:16–00690) and the Veterans Affairs (Protocol #:1609) Institutional Review Boards. Informed consent was obtained from all the participants in the study. All the procedures were followed in accordance with the relevant guidelines (eg. Declaration of Helsinki) along with the rest of the ethical declarations.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Supplement Table 1.

Characteristics of Patients Who Were Determined Eligible for the Intervention Arm (and were Outreached by CHWs), by Their Enrollment Status. Supplement Table 2. Characteristics of Patients Who Were Enrolled in the Intervention Arm, by their Intake Completion Status. Supplement Table 3. Moderation of Fidelity Measures by PAM Score Among Intervention Patients Completing the Intake Survey. Supplement Table 4. Moderation of Fidelity Measures by Clinical Site* Among Intervention Patients Completing the Intake Survey.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gupta, A., Hu, J., Huang, S. et al. Implementation fidelity to a behavioral diabetes prevention intervention in two New York City safety net primary care practices. BMC Public Health 23, 575 (2023). https://doi.org/10.1186/s12889-023-15477-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12889-023-15477-2

Keywords

BMC Public Health

ISSN: 1471-2458

Contact us