Navigating barriers: two-year follow up on recommendations to improve the use of maternal health guidelines in Kosovo

Abstract

Background

Although there are a growing number of initiatives aimed at supporting guideline implementation in resource-constrained settings, few studies assess progress on achieving next steps and goals after the initial activities are completed and the initial funding period has ended. The aim of the current study was to conduct a qualitative process evaluation of progress, barriers, facilitators, and proposed solutions to operationalize nine recommendations to prepare Kosovo to implement the 2012 World Health Organization (WHO) prevention and treatment of postpartum haemorrhage guideline.

Methods/Design

In 2012, we co-created nine recommendations designed to support implementing the WHO’s guideline on the prevention and treatment of postpartum haemorrhage in Kosovo. The current study uses a realist evaluation approach to assess activities and progress two years after the recommendations were developed. The study involved conducting qualitative focus groups and one-on-one interviews with participants from the first meeting to evaluate the activities and progress on the nine recommendations.

Results

Forty-three participants provided insights into the barriers and opportunities experienced to date and proposed future directions. Although progress has been made towards implementation of a number of the recommendations, scaling up has been limited by barriers, such as lack of awareness, limited resources, and evaluation challenges. Participants proposed addressing these barriers by building within- and between-country partnerships to facilitate guideline implementation. In addition, participants reported less progress on implementing recommendations related to broader cultural changes, which indicates a need for specific and actionable recommendations to operationalize implementation efforts.

Conclusions

In the two years since the initial meeting, there has been mixed progress on the recommendations. Based on participant feedback, we refined the recommendations so that they can be operationalized by health care system stakeholders in Kosovo to further support implementation efforts. It is beneficial to share these lessons learned throughout the implementation process to inform next steps in Kosovo and offer ideas for use in other settings.

Background

Despite the large number of high-quality, evidence-based guidelines available, clinicians do not routinely use guideline recommendations in practice [1]. The goal of knowledge translation (KT) is to address these research-to-practice gaps by using evidence on the best ways to support systemic, organizational, and individual change [2]. There are a growing number of initiatives aimed at supporting guideline implementation in low and middle-income countries, (LMICs) [3, 4], yet very few clearly defined implementation strategies are currently used [5], and even fewer studies follow up on what has been done after the initial funding has expired. Challenges to applying research evidence, and to implementing guidelines in particular, are compounded when working in LMICs, where resources are constrained [6] and there is a history of non-governmental organizations (NGOs) allocating short-term funding without monitoring and evaluating progress or planning for long-term sustainability [7].

Globally, many LMICs struggle with high rates of maternal and perinatal mortality. Although there have been improvements in maternal and infant outcomes, maternal mortality rates (MMR) have declined by 45 % from 1990 to 2013, nevertheless, for every 100 000 live births in 2013, an estimated 210 women died due to complications in pregnancy and childbirth. Moreover, 2013 data showed that the MMR in LMICs was 14 times higher than that in high-income countries [8]. Kosovo, where the current study was conducted, has a perinatal mortality rate (18.7 % in 2011) that is higher than that of other countries in Europe [9]. Furthermore, Kosovo’s perinatal mortality rate increased from 2005 to 2008, even though 95 % of women reportedly gave birth in a health care facility. During the years following this period (2008–2010), the second most common cause of maternal mortality in Kosovo^{1 1} was postpartum haemorrhage (PPH; 5 out of 24 deaths or 21 %) [10]. Based on these trends, we focused our implementation initiative in Kosovo on the 2012 World Health Organization (WHO) guidelines for the prevention and treatment of PPH [10].

The current study is a partnership between health care system stakeholders working in maternal health in Kosovo, the UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction (HRP), and the KT Program at St. Michael’s Hospital (SMH) in Toronto, Canada. In 2012, we conducted a survey, two focus groups, a prioritization activity, and a workshop to plan for the implementation of maternal and perinatal health guidelines in Kosovo [11]. Primary outcomes of the workshop included a list of prioritized guideline recommendations, barriers and facilitators to guideline adherence, and nine key recommendations, which served as potential actions to move guideline implementation efforts forward. Although some of the barriers pertained to larger resource and system-level issues (e.g., lack of a standardized data collection system; post-conflict fragmentation of the health care system; and human resource challenges, including high turnover of health care workers), several barriers were more easily turned into actionable recommendations (e.g., develop a guideline implementation working group).

Researchers have found that implementation efforts are not always effective and typically produce small effect sizes [12]. This suggests that it is important to not only ensure that implementation strategies are carefully prioritized, selected, and matched to address known barriers and facilitators, but also that implementation activities are continuously monitored and evaluated. Because implementation is an iterative process, the findings of a process evaluation are critical to the implementation pathway and can inform how an implementation strategy should be revised or discontinued [2]. Process evaluations provide an opportunity to see what progress has been made, change course if necessary, and refocus implementation efforts. They also give program developers and implementers an opportunity to infuse evidence and theory into the implementation process, which further strengthens the quality of the initiative [13, 14].

Study objectives

The aim of the current study was to conduct a qualitative process evaluation, using a realist evaluation approach [15], of activities and progress on the nine recommendations made to prepare Kosovo to implement the WHO prevention and treatment of PPH guideline. The period of interest for this evaluation was from October 2012 until November 2014. The specific objectives were to: 1) understand progress to date on each 2012 recommendation; 2) identify barriers and facilitators to progress on each recommendation; 3) capture stakeholders’ proposed future directions; and 4) develop refined recommendations to facilitate implementation.

Methods

Using a realist evaluation approach [15], we conducted qualitative focus groups and one-on-one interviews to evaluate the activities and progress on the nine recommendations that emerged from a 2012 prioritization meeting. The purpose of this process evaluation was to understand progress to date and to support a renewed interest in developing, adapting, and implementing guidelines to change provider behaviour and patient outcomes, focused in maternal health. As a result of these goals, we used a realist evaluation approach [15, 16] to conduct the study and develop actionable recommendations to move forward. Using a realist evaluation approach assumes that implementation is inextricably linked with the context, meaning that the outputs and outcomes of an intervention cannot be separated from the setting in which the intervention was implemented. Since contextual factors change over time, it is imperative to revisit goals and progress to date; this process of refining goals can also serve to reinvigorate stakeholder buy-in, enhance motivation, and identify next steps that are feasible and appropriate at a given juncture in the implementation process.

Setting

Focus groups and interviews were conducted in Kosovo with support from the WHO Country Office in Pristina, Kosovo; the United Nations Development Programme (UNDP)/United Nations Population Fund (UNFPA)/United Nations Children’s Fund (UNICEF)/WHO/Special Programme of Research, Development and Research Training in Human Reproduction (HRP), Department of Reproductive Health and Research, World Health Organization Headquarters in Geneva, Switzerland; and the KT Program at St. Michael’s Hospital in Toronto, Canada. Focus groups were conducted during an in-person meeting held in Pristina, Kosovo in 2014.

Participants

Local stakeholders (e.g., policymakers, researchers/academics, frontline health care providers, and representatives from professional associations and non-governmental organizations) who took part in the two-day workshop in 2012 were invited to participate in a focus group. They were told that they would be asked to discuss progress, challenges, and opportunities related to the nine key recommendations made to improve the uptake of the WHO guideline recommendations for the prevention and treatment of PPH. Additionally, five local stakeholders were identified as key informants on maternal health in Kosovo and were invited to participate in one-on-one phone interviews. Participants were identified because they play a key role in guideline development and/or guideline implementation; two of the four participants were also present at the in-person meeting.

Data collection

Facilitators and interviewers used a semi-structured guide during both the focus group (FG) session (which lasted approximately 60 min) and interviews (which lasted approximately 45–60 min; Additional file 1). Participants received a summary of the 2012 workshop findings and recommendations to review before the FG/interview (Additional file 2). The summary document included a column where participants had the option of anonymously commenting on the progress made on each recommendation. The FG session and one-on-one interviews were audio recorded. The FG session was conducted in Albanian and recordings were simultaneously translated into English and transcribed by a translator. Interviews were conducted in both English and Albanian and were transcribed verbatim or simultaneously translated and transcribed by an independent translator, respectively.

Data analysis

Two qualitative analysts (not involved with data collection) independently performed a qualitative analysis of the transcripts by using a thematic analysis approach [17]. First, the analysts familiarized themselves with the data to develop coding themes. Second, these themes were refined into categories, which were then compared, sorted, and grouped to develop a framework of findings. Third, all transcripts were imported into qualitative analysis software (NVivo 10) and the analysts independently coded using the developed framework. Inter-rater reliability (i.e., the degree of agreement between two coders) was compared by calculating percentage agreement within NVivo 10; any discrepancies (i.e., < 80 % agreement) between the analysts were reconciled through discussion. Using the technique of integration, FG and interview data were considered together to draw meaningful and pertinent recommendations that are feasible and relevant for the Kosovo context. Revised recommendations were developed collaboratively with the study authors, including end-users (Kosovo partners), guideline development experts (WHO partners), and KT experts (SMH partners).

Results

Thirty-nine stakeholders, (e.g., policymakers, researchers/academics, health care providers, professional associations, non-governmental organizations) took part in the FG, and 17 of the 39 stakeholders completed the summary worksheets. Of the 39 participants, 14 participated in the original 2012 meeting (there were 28 participants at the 2012 meeting); the remaining participants have subsequently been involved with activities related to the recommendations. Table 1 presents information on the professional roles of the FG participants. Four of the five invited stakeholders participated in one-on-one interviews.

Table 1 Focus group participant professional role information

During the process evaluation conducted in 2014, participants described factors (i.e., barriers and facilitators) they perceived to have influenced progress made regarding the recommendations identified in the original workshop in 2012. For recommendation-specific barriers and facilitators refer to Table 2.

Table 2 Summary of progress, barriers, facilitators, and future directions for nine recommendations to support the implementation of WHO guideline on the prevention and treatment of PPH as identified by participants

Perceived barriers to applying 2012 guideline implementation workshop recommendations

Lack of guidelines and protocols

The majority of participants cited the absence of guidelines and protocols as one of the key barriers to the progress of most of the recommended guideline implementation activities (e.g., the inclusion of a monitoring system for guideline adherence, and the creation of motivational strategies to encourage adherence). Participants revealed that guidelines have remained in the development stages and as a result, activities proposed have been negatively impacted. Stakeholders’ lack of awareness and/or knowledge of the clinical guidelines have contributed to the lack of progress made towards the availability of guidelines. In addition, some have suggested that the lack of understanding between the differentiations of guidelines versus protocols have attributed to this situation as well.

“…we had problems identifying and clarifying what a clinical guideline and what a clinical protocol means.”

Presence of official procedures, formalities, and bureaucracy

Other participants perceived factors such as official procedures, formalities, and bureaucracy as impediments. For example, although a National Council for guideline and protocol development had been established in 2011, it is no longer active, “awaiting administrative instruction to renew membership of the committee” and what is more, changes to key positions (i.e., officials) in the Ministry have triggered other issues and delays to the re- instatement of this committee.

“We have so many frequent changes at the key positions within the Ministry so, one goes, the other comes, and they don’t see it as a priority issue. So most of our time we are just wasting by advocating and informing newcomers about the importance, either for guidelines or for other issues which are within our mandate.”

As a result, without this committee, participants expressed that steps to carry forth recommended activities such as the creation of working groups with local representative stakeholders (e.g., clinician groups, researchers and WHO) are futile.

“If the committee on guideline development is not reactivated, then what is the aim of the group to work, then there is nobody to endorse the guidelines, so it [the process] should parallel somehow, the committee to be reactivated, and the working groups to be endorsed so they could start working their job.”

Lack of decision-making capacity

Aside from the availability of guidelines and protocols in place, participants also expressed frustration with their lack of capacity to make any significant official decisions to support guideline implementation activities. One participant recounted that “we are not in a position that as outsiders from the Ministry of Health to take lead on the process,” a statement which is similarly echoed by others:

“The only problem with coordination meetings is that we don’t have a kind of decision-making role, we can only advocate and we can only suggest and recommend, actually, and we can of course as stakeholders or even clinicians, share our concerns and our achievements or whatever, but we are not a decision-making body.”

Lack of available resources

Other factors that participants perceived to have influenced the progress of guideline implementation activities included the availability of monetary funds, although the perceptions of its impact varied from participant to participant. Lack of monetary funds was perceived by one participant as the barrier amid recommendations to supply future guideline development workshops and engage local clinicians in the process, stating that their budget was insufficient to financially support another workshop and those interested in attending. Whereas, another participant explained that while there are stakeholders who are interested in contributing to working groups with or without financial compensation, it may be more difficult to engage those less interested but more knowledgeable in this area because “without financial means they are not willing to dedicate their time.”

Lack of motivation

Lastly, lack of motivation was cited by a few participants as a hindrance in some cases. For instance, currently MOH holds no authority over private practices and there seems to be no clear distinction between private and public practice, therefore, as a result clinicians that associate themselves as practicing in the private sectors are less motivated to engage in processes developed in a public health institution. Furthermore, one participant stated that “motivation of health workers is not a priority in Ministry of Health approach toward improving health, health system performance” and as such there is a lack of interest amongst health workers to become engaged in initiatives that foster guideline development and implementation.

Perceived facilitators to applying 2012 guideline implementation workshop recommendations

Accessibility of existing guidelines

Efforts to encourage the use of existing international guidelines (i.e., reliable guidelines of other countries and associations) to develop local protocols have indirectly contributed to the availability of guidelines and thus the progress of recommended activities. By doing so, this has in turn helped to mitigate some of the barriers to the progress of the recommendations associated with lack of guidelines and protocols mentioned above.

Incentives

The provision of incentives was suggested by some as a means to motivate and encourage engagement. Several participants believed that by offering monetary incentives, certifications and/or credits local clinicians would become more involved and participate in activities such as guideline development working groups/committees.

Recommendation-specific barriers and facilitators

Table 2 presents the findings of the process evaluation conducted in 2014 organized by the nine recommendations identified in the original workshop in 2012. For each recommendation, we have presented progress to date, barriers, facilitators, and participants’ proposed future directions. The information provided in Table 2 represents the aggregate perspectives of participants in the FGs and interviews.

Participants reported that substantial progress had been made on three recommendations. First, significant progress has been made in developing the centralized system for data collection across clinical settings (recommendation 1). Although the system has not yet been established, a number of steps have been completed and the initiative has gained leadership buy-in. Specifically, it has been added as a priority to the Ministry’s Sectorial Health Strategy 2014–2020. The Electronic-Based National Health Information System is being piloted in three Regions of Kosovo (Pristina, Prizren and Ferizaj) in all three levels of care. This pilot is a necessary pre-requisite to implementation of the Health Insurance System, planned for 2016. They are currently determining the most important and priority SMART (Smart, Measurable, Attainable, Relevant, Trackable) performance indicators to initiate payment for performance, as next step toward improved quality of care. Pay for performance was meant as motivating factor for improved individual and institutional performance and creating competitive health care environment in terms of higher level and/or better quality of clinical guideline implementation. Second, progress has been made in incorporating standards into clinical practice using a monitoring system for guideline adherence (recommendation 2). These efforts have, however, been made primarily at a local rather than national level; the current goal is to scale up these efforts. Barriers to scaling up these monitoring standards nationally include lack of awareness, resource issues (human and financial), and evaluation challenges. Participants proposed several future directions to address these barriers and facilitators. Third, Kosovo stakeholders have successfully reached out to neighbouring countries to provide mutual support for guideline development and implementation (recommendation 9). Participants identified several future directions for building partnerships with other countries.

Participants indicated that limited progress had been made on four recommendations. Although some workshops had been offered, participants saw a need for additional capacity building and support after the training (recommendation 7). Participants perceived lack of funds for training as the primary barrier to capacity building and believed that there were opportunities for participants to support each other after attending workshops. For the recommendations that involve developing working groups (recommendations 5, 6, and 8), participants thought little progress had been made over the past two years, primarily due to system-level barriers.

Two recommendations describe a cultural shift in perspectives that may enable guideline implementation; however, these recommendations are not as readily actionable as are some of the other recommendations. Specifically, there is a need to create motivational strategies, such as incentives for health care staff, to encourage guideline adherence and enhance communication across stakeholder groups, including clinicians, managers, and policymakers through participation in such activities as guideline development committees (recommendations 3 and 4). Stakeholders made less progress in both of these areas and identified several barriers that may have prevented progress.

Discussion

Updated recommendations

In October 2012, we conducted activities to plan the process of implementing the WHO guideline on the prevention and treatment of PPH in Kosovo. Nine key recommendations and a supporting report on findings were developed as a result of the activities [11]. Two years after the workshop, we completed a realist process evaluation using FGs and key informant interviews to assess participants’ perspectives on progress on the nine recommendations to date, barriers and facilitators experienced related to implementing each recommendation, and possible future directions. The realist approach to the process evaluation allowed us to understand how the characteristics of the implementation setting affected the outputs and progress reported by participants in relation to operationalization of the nine recommendations [15]. One of the lessons learned from the first workshop in Kosovo, and from subsequent similar activities conducted in other LMICs [18–20], was to develop concrete and actionable recommendations for implementation that are grounded in behaviour change theory [21]. Therefore, we developed five updated recommendations: a) develop a guideline development/adaptations working group to create or adapt maternal and perinatal health guidelines; b) develop a guideline implementation working group to pilot test an evidence-informed implementation strategy to support guideline implementation, then scale up the initiative; c) develop guideline policy working groups to work collaboratively with MOH and clinician stakeholders to identify and support opportunities to engage in policy changes related to guideline implementation; d) continue to develop and operationalize a centralized system for data collection; and e) expand guideline development and implementation capacity building and networking activities. Table 3 presents five recommendations adapted from the original recommendations based on progress to date, barriers and facilitators, and suggestions made by participants for improvement. To ground the work in theory, we have also presented elements of the behaviour change wheel [21] that are relevant to each stakeholder group.

Table 3 New recommendations proposed by study team

These recommendations reflect the planning phase of implementation, preparing the system and organizations for guideline implementation, also referred to as readiness, "the extent to which organizational members are both psychologically and behaviorally prepared to implement change" [22]. Although groups may wish to jump into implementation, process models explicitly describe readiness as a step in implementing evidence in practice [12, 23, 24], given that readiness predicts implementation and outcomes [25–27]. In addition to understanding organizational and individual readiness, in LMICs there is a need to also focus on system readiness [26]. Infrastructure (or lack of infrastructure) can facilitate or hinder implementation and outcomes; therefore building a strong infrastructure can have long-term benefits [28, 29]. There are tools to address system readiness in LMICs which can help in this process, such as the Intervention and Research Readiness Engagement and Assessment of Community Health Care (I-RREACH) [30].

Cross-cutting principles

In reviewing the progress to date on the recommendations, we noted that two of the recommendations involve more of a cultural/system-level shift and are, therefore, more difficult to operationalize. Specifically, the in-country work in 2012 identified two significant barriers to implementation: lack of communication between key stakeholder groups and a focus on punitive interventions (rather than taking a more positive, motivational approach) [11]. Both of these barriers are likely influenced by the socio-political climate in Kosovo since the disintegration of Yugoslavia [31], and therefore are not likely to change quickly, nor are they entirely within the sphere of influence of the key stakeholders involved in guideline implementation. It is important to consider the multi-faceted and interactive effects of personal and environmental factors that determine behaviours [32]. Given this context, we have added two cross-cutting principles that all working groups should consider when undertaking guideline development and implementation work:

• 1) Establish a mechanism for continuous communication, collaboration, and experience-sharing between key stakeholder groups (e.g., maternal and child health forum); and

• 2) Foster teamwork culture and collective responsibility rather than focusing on individualization and personalization; this shift could focus on building motivation and using incentivized strategies, rather than focusing on punishment.

Application in other contexts

The findings of this process evaluation are transferrable to other areas of maternal health work and implementation of evidence-based guidelines, particularly in LMIC settings. Many of the barriers and facilitators experienced by health care system stakeholders in Kosovo also exist in other contexts [33], and the need for specific and actionable recommendations to operationalize implementation efforts is crucial. Recommendations should clearly define the “who”, “what”, “when”, and “how” of an implementation strategy so that stakeholders understand their roles, the objectives, the mechanisms of change, and the anticipated outcomes.

Enabling systems change is challenging [34], and the addition of system-level conflict and instability, which are a reality in many LMIC contexts, exacerbates these issues. Given this context, it is imperative to build strategic partnerships across sectors and levels of government, advocating for policy change, and building the capacity of health care providers and other health care system stakeholders. Existing KT platforms that provide guidance and support on policy engagement can help support these efforts, such as EVIPNet and the Supporting Use of Research Evidence (SURE) [35]. These can be particularly helpful in thinking through how best to communicate with stakeholders (e.g., policymakers) and how to rapidly respond to policy requests [36, 37, 3]. Additionally, ongoing implementation and technical support for working groups in LMICs (i.e., training and coaching) can be provided by groups like the GREAT Network to aid in-country stakeholders in achieving and sustaining implementation goals [38].

Limitations

Given that research and implementation activities largely stem from time-limited grants, little research is done to evaluate the long-term activities and impacts of implementation efforts. Therefore, this study presents a unique perspective on implementation activities by following-up on activities two years after the initial workshop. However, some limitations of the study should be noted. Although we were able to collect data from a large number of participants, the size of the focus group was larger than ideal [39]. We had the opportunity to include a wide range of stakeholders in the focus group by combining the focus group with an existing meeting; however, this resulted in a large number of participants, all of whom may not have been able to express their views. Consequently, we decided to conduct interviews with a few key stakeholders to make sure that we captured their perspectives. In addition, communication challenges between groups were identified as a major barrier in 2012 and continue to pose challenges. We had limited MOH representation and involvement. Ideally, we wanted to collect data from the MOH perspective as they are a driving force or barrier behind many of the activities but they declined participation. Finally, our understanding and interpretation of the data were limited by cultural barriers and local contextual factors. To reduce the impact of this limitation, an in-country expert was consulted throughout the process to enhance comprehension of the data and its relevance to the local context; this individual is included as a co-author on this paper.

Conclusion

Using a realist evaluation approach, we assessed activities and progress over a two-year period on the nine recommendations designed to support Kosovo in implementing the WHO guideline on the prevention and treatment of PPH. The evaluation findings provided rich insights into the barriers and opportunities experienced to date. They have also led to the development of five additional recommendations, which were adapted from the original recommendations that can be operationalized by health care system stakeholders in Kosovo. While developed for implementation in Kosovo, these recommendations are transferrable and can be adapted and applied in other LMIC settings because the barriers and facilitators to implementation are shared [33]. Reporting on process evaluation findings, particularly in the context of implementation initiatives, is important for providing insight into realistic targets and results for changes in behavioural outcomes. It is beneficial to share these lessons learned throughout the implementation process for use in other settings. Next steps will involve evaluating how operationalizing the five recommendations has affected outcomes related to use of the WHO prevention and treatment of PPH guideline in Kosovo. Once data monitoring systems are established, future evaluations will examine the degree of WHO maternal health guideline implementation.