Global health systems partnerships: A mixed methods analysis of HPV vaccine delivery network actors in Mozambique

Background Global health partnerships have expanded exponentially in the last two decades with Gavi, the Vaccine Alliance considered the model’s pioneer and leader because of its vaccination programs’ implementation mechanism. Gavi, relies on diverse domestic and international partners to carry out its programs in low- and middle-income countries under a Partnership Engagement Framework (PEF). We harnessed an established partnership evaluation framework, to conduct an in-depth examination of Mozambique’s Gavi driven partnership network which delivered the human papillomavirus (HPV) vaccine during the demonstration phase. Methods We utilized mixed methods to assess five partnership dimensions. Qualitative tools gauged contextual factors, prerequisites, partner performance and practices while a social network analysis (SNA) survey measured the partnership structure and perceived added value in terms of effectiveness, efficiency and country ownership. The 40 key informants who were interviewed included frontline ministry of health workers, ministry of education staff and supporting partner organization members, of whom 34 participated in the social network analysis survey. Results SNA measures for partnership structure revealed a partnership network characterized by high overall connectivity. Reachability 100% and average distance 2.5 scores were high, features that are favorable for rapid and widespread diffusion of information that is needed for engaging and handling multiple implementation scales. High SNA effectiveness and efficiency measures for structural holes (85%) and low redundancy (30%) coupled with high mean perceived effectiveness (97.6%) and efficiency (79.5%) outcome scores were observed. Additionally, the tie strength average score of 4.1 on a scale of 5 denoted high professional trust. These are all markers of a partnership where disparate institutions and organizations worked in a collaborative environment in

which each entity's comparative advantage was leveraged. Lower perceived outcome scores for country ownership (24%) were found with the challenges of working with out-ofcountry partners being prominently cited by study participants as reasons.
Conclusions While there is room for improvement on the country ownership aspects of the partnership, the expanded, diverse and inclusive collaboration of institutions and organizations that implemented the Mozambique HPV vaccine demonstration project was effective and efficient and we recommend that the country adapt a similar model during the national scale up.

Background
Health program implementation especially in low-and middle-income countries (LMICs), requires the involvement of different types of players from different sectors (1). Disparate individuals from different types of organizations and institutions, spanning multiple and hierarchical health system levels, such as community, health facility, regional, national and international all have a role to play (2). This diversity in actors relies on various relationships and interactions amongst all players involved in program operationalization.
Partnership, which is defined as a collaboration with the mission of accomplishing a common goal either contractually or non-contractually, is a key type of such relationships (3). Partnerships, have traditionally existed at the country level amongst different implementing entities, however the global health partnerships (GHPs) phenomenon, is more recent, with most such partnerships being created in just the last two decades (4). Despite their novelty, GHPs have expanded exponentially and gained broader relevance to the extent that they are now a salient feature of health program implementation in LMICs (5). Consequently, partnership has become a significant health program implementation determinant in these settings (6) and the broader implementation science (IS) field formally recognizes the study and measurement of partnerships as a core component of implementation research(7) (8) (9). Gavi, the Vaccine Alliance (Gavi), is a GHP that was set up with the agenda of increasing the availability and accessibility of new and underutilized vaccines in LMICs (10). Founded in 1999, the public private partnership brings together four founding organizations, the Bill and Melinda Gates Foundation (BMGF), World Bank, World Health Organization (WHO) and the United Nations Children's Fund (UNICEF) in a set up designed to leverage each organization's unique technical expertise in the global arena. Additionally, pharmaceutical companies that manufacture vaccines, donor and recipient governments, civil society organizations, research and technical institutes are also members of the alliance (11). In this partnership model, Gavi does not place implementation support teams for its sponsored programs in countries but instead relies on both in-and out-of country partners under its Partnership Engagement Framework (PEF) mechanism. WHO and UNICEF are the principal in country implementation partners undertaking majority of PEF activities, while other organizations' involvement depends on countries' immunization programs' capacity and needs (12).
Gavi financial support for new vaccine introductions (NVIs) began in 2000, with availability of funds for the combined diphtheria, pertussis (whooping cough), tetanus and Hepatitis B (DPT-HepB) vaccine. Subsequently the number of supported vaccines increased to the current eleven; one of which is the Human Papillomavirus (HPV) vaccine (13). Its inclusion in 2011, was however marked by an unprecedented requirement for countries to first implement a demonstration project prior to national scale-up, in order to gauge effectiveness of possible delivery models. The demonstration project was deemed necessary due to the novel target age group of 9 -13 years that falls out of the routine country national immunization program (NIP) target age group of 9 -24 months. Most LMICs do not have an adolescent specific health care service and therefore lack a health service delivery system for this target group (14). HPV vaccine introductions were therefore being conducted in two phases, the demonstration project and the national roll out, each with its own set of decision making, planning and execution processes (15). While Gavi has had a record of success in accelerating the national adoption of the other new vaccines (16), HPV vaccine introduction encountered a challenge of slow adoption.
Twenty three countries completed demonstration projects by December 2016, but only 6 had advanced to national scale up phase by Dec 2017, a stark contrast to Gavi's planned target of 8 nationwide HPV vaccine introductions by 2015 (17). Despite Gavi's effort to accelerate country uptake by removing the required demonstration project phase (18), only 18 LMICs countries have introduced HPV vaccines into their NIPs to date. The lagged adoption has been attributed to multiple factors, however an important barrier is the dearth in published country specific HPV vaccine demonstration project evaluation findings in the literature (19).
Social Network Analysis (SNA) is a modern sociology technique, that provides theory and tools for the visualization and measurement of interactive relationships amongst people or items. SNA is widely applicable to relational patterns created by any tangible or nontangible content (20). Because such patterns form network structures that may act as facilitators or constraints to the achievement of a networks' objectives, SNA findings can be utilized to address a network's structure bottlenecks or leverage its strengths (21).
Health services delivery researchers are increasingly harnessing SNA tools to understand diverse aspects of health systems. Organizational governance has proved to be an opportune topic for this type of inquiry which makes inconspicuous communication and collaboration patterns visible, thus enabling the study of key strategic and managerial decision-making information flows amongst important actors within a health system (22) (23).Despite this expansion in SNA utilization, its use in LMIC health systems governance research is still nascent (24) (25). One aim of our study is to contribute to the expanding health systems SNA body of knowledge, through studying a complex vaccine delivery collaboration. The second objective is to fill the existing gap in documentation of country specific HPV demonstration project research findings in the literature. We present a case study conducted on the Mozambique HPV vaccine delivery demonstration project. We leveraged the Gavi Full Country Evaluation (FCE) partnership framework ( Figure 1) (25) to test the contribution of network structure on the added value of the project's partnership.

Study Setting
The 2014 to 2018 Gavi FCE, was a prospective in-depth assessment of inputs, outputs, outcomes and impact of its financial support to low-and middle-income country immunization programs. One key FCE research question explored the role of Gavi's partnership model on decision making, planning, implementation and technical support in delivery of country immunization services. This paper presents findings from the case study undertaken in Mozambique in response to this partnership research question.

The case
The Mozambique HPV vaccine demonstration project, whose objective was to test a model for HPV vaccine delivery to girls aged between 9 and 13 years, was undertaken during two years, 2014 and 2015. It was implemented in three districts that were chosen to represent the three diverse geographical regions of the country. These were Manhiça, Manica and Mocímboa da Praia in Southern, Central and Northern regions respectively. Funding was obtained from Gavi for Manhiça district while the Government of Mozambique (GoM) funded the project in the other two districts. The tested delivery model was primarily school based whereby health workers administered the vaccine to girls in schools. Periods of one week were predetermined and utilized for the delivery of each dose. In the first year (2014) three doses were delivered in May, June and November in alignment with the then WHO recommended HPV vaccine administration schedule of 0, 1, and 6 months intervals. In the second year (2015) the WHO guidelines had changed to two doses to be given at 0 and 6 months and the vaccine was administered in June and November(26).

The Gavi FCE Partnership Framework
Our study was oriented by the Gavi FCE partnership framework, which was developed through integration of concepts from public administration, organizational science and network analysis, resulting in five causally related partnership evaluation dimensions ( Figure 1). The framework differs from predecessor partnership evaluation frameworks, by placing the role of a partnership's network structure at the core of its effectiveness. It postulates that effective partnerships require appropriate partnership network structures and proposes the empirical measurement of network structure, using social network analysis (SNA) metrics, as a tool to inform partnership formation (25). It hypothesizes that the framework's structure dimension will be highly predictive of a partnership's performance. This assumption is based on the social network diffusion theory (20) which explains how network structures act as either facilitators or barriers for adoption of novel practices. According to the theory, network-level structure outcomes for connectivity, efficiency and effectiveness, are determined by relationships between network actors.
These relationships usually involve a relational content (20), such as information or resources, that can be measured using SNA metrics (20). In this study technical assistance information, was the relational content measured, in order to determine the structure of collaboration relationships amongst Mozambique's HPV vaccine demonstration project implementation partners.

Study design
Mixed methods were used to study the partnership evaluation framework's five dimensions  Contextual factors and prerequisites: Identification of the governmental and non-governmental institutions and organiza well as the individuals that are involved in the implementation of the Mozambique HPV vaccine demonstration project. The ori partnership and how the broader political environment and the global Gavi Alliance partnership has influenced this. Other Moz contextual factors that have facilitated or blocked the successful formation and performance of this partnership.

2.
Partnership structure: Understanding the composition of actors and their relationships with each other as well as the con between the implementers of the HPV vaccine demonstration project in the country.

3.
Partner performance: Exploration of how effectively partners meet their deliverables and the mechanisms for accountab 4.
Partnership practices: Investigation of the functionality of the whole partnership in terms of professional trust. Additiona formal and informal rules and processes that govern the partnership; partners' competencies, capacities, roles and responsibi partnership coordination mechanisms and management of the partnership.

5.
Partnership's added value: Determination of partners' perceptions of outcomes of working together, namely efficiency, effectiveness and country ownership;

Data Collection
Institutional Review Board (IRB) approvals were obtained from University of Washington and Mozambique Ministry of Health prior to data collection. A team of Gavi FCE researchers, collected data at central level and in all three demonstration districts, between during the two years that the HPV vaccine demonstration project was implemented.

Qualitative data
Three methods were used to gather qualitative data i) document review ii) direct Direct observations were conducted through participation in all HPV vaccine demonstration project meetings ranging from NIP technical working group meetings; subcommittee meetings focusing on, for example, cold chain management; national immunization technical advisory group meetings (NITAG); and Inter Agency Coordination Committees (ICC) meetings. Additionally, events such as trainings, supervision visits, the official launch and all implementation activities for HPV vaccine demonstration project in the three demonstration districts were tracked. Information gathered from direct observations that respond to the evaluation aspects of the five dimensions of the partnership framework were used to further refine the questions in the KII topic guide. KII respondents were identified through a two-pronged sampling approach. First, was the comprehensive approach targeting all known NIP stakeholders and second the chain referral (snow ball) approach targeting other key informants (KIs) as they were referred by those already sampled (27). Known stakeholders refer to institutions and organizations that had been distinguished during document review and direct observations. Samples were drawn from the following groups within these identified entities: health facility immunization staff, district immunization staff, district education staff, provincial immunization staff, provincial medical heads, central MOH level immunization program staff, staff from research institutions, non-Governmental organizations (NGOs), bilateral and multilateral institutions that supported the NIP during all the HPV vaccine demonstration project phases. A semi-structured open-ended interview guide was developed and used for all respondents with probing techniques being applied whenever the need arose. Interviews were conducted in the place of convenience to the key informant, usually in their office. Appointments were arranged through both email and phone. Consent forms and topic guides were shared with respondents prior to interviews which were documented through note taking and audio recording.

Social Network and Perceived Outcomes data
Two sets of hypotheses, listed in table 2, guided SNA measurement of the partnership network structure. The first set relate network structure connectivity to WHO health system governance characteristics, an approach originally described by Blanchet et al (28).
The second set stem from SNA theory of diffusion and relate network structure to network efficiency and effectiveness (21).
Both SNA and perceived outcomes data were collected through a single two-section structured questionnaire. The positional strategy (20) was used to specify the network boundary with actor identification being based on their positions in institutions and organizations that were involved in HPV Table 2: Social Network Analyses hypotheses and metrics vaccine demonstration project implementation. Each actor was asked: "With whom did you work with on the HPV vaccine demonstration project?" This process is referred to as a 'name generator' in social network analysis (20). For documentation of the relational content, survey respondents were asked whether they had received or provided technical assistance. In order to measure tie strength respondents were asked to rate their level of

Data Analysis
The units of analysis for the qualitative segment of our study, were the individuals, institutions and organizations directly involved in the implementation of the Mozambique HPV vaccine demonstration project. Audio recorded data was transcribed into texts by known professional transcribers with experience in transcribing other audio recorded data for the Mozambique Gavi FCE. Nvivo software was used to facilitate a theme-based analysis as per the five dimensions of the Gavi FCE partnership conceptual framework. A preliminary codebook was developed and responses to the partnership framework dimension's research questions sought and given their own code. Following coding, a report of each code was produced, read, and re-interpreted resulting in a synthesis of findings related to each dimension.
The individual was the unit of analysis for the social network and perceived outcome analyses. The data collected from the network survey was entered into a spreadsheet matrix in MS Excel. Each reported working relationship, or "tie," was weighted according to the key informant's rating of relationship trust. Technical assistance exchange tie was entered as binary. The resulting spreadsheet contained three asymmetrical, undirected network matrices which were imported into UCINet software(30) for analysis. Descriptive network maps were created using the NetDraw(31) application within UCINet software with nodes being color coded according to node attributes from the survey. Tests were run for each SNA metric associated with the hypotheses behind our SNA (Table 2). Five metrics namely reachability, distance, centralization, betweenness and density measured network structure's connectivity while three metrics, structural holes, redundancy and homophily measured its efficiency and effectiveness. SNA metrics that measure the networks structure efficiency and effectiveness are referred to as network outcome measures.

levels and sectors of the health system)
Reachability: Networks with a high level of reachability have the ability to access various sources of information. Distance: The shorter the distance between the actors the faster the diffusion of information. Networks with short average pat are more likely to facilitate the widespread diffusion of information Characteristic 2: Capacity to anticipate and cope with uncertainties and surprises Centralization: A centralized structure has a higher capacity to coordinate actors and provide rapid response Betweenness: Rapid response occurs when the key actors have the ability to reach all the players in the network Characteristic 3: Capacity to combine and integrate different forms of knowledge Reachability: The diversity of technical knowledge can be achieved through relationships with actors that belong to other sphe other subnetworks. Density: Dense networks are more likely to facilitate the transfer of information however actors in a dense network have diffic accessing diverse forms of knowledge

Theory of network diffusion hypotheses and measures for network efficiency and effectiveness
Structural holes: Network with more structural holes is more efficient for the diffusion of information Redundancy: Less redundancy means a more efficient network for the relaying of information Homophily: Novel information is more likely to enter heterogeneous networks while homophily in a network can be a barrier to new information Data from the perceived benefits and drawbacks survey section was entered in the same excel spreadsheet as the network data and proportional scores calculated for each survey response. In addition, the benefits and drawbacks were stratified into effectiveness, efficiency and country ownership and a mean score calculated for each stratum.
We utilized an iterative triangulation process, in which all the three types of data were analyzed and reanalyzed, to construct patterns and facilitate interpretation.

Descriptive statistics
A total of 40 KIIs were conducted at national, provincial, district and health facility levels in the three HPV vaccine demonstration district sites in the country (Table 3). Of these study participants, 34 responded to the social network and perceived outcomes surveys.
Majority of those interviewed were from the Ministry of Health (MOH) (55%) and the Ministry of Education (MOE) (15%) with a larger proportion (35%) being from the district level. Detailed results are presented below within the five Gavi FCE partnership framework dimensions.

Contextual factors and prerequisites
We found that, Gavi plays a key role in driving Mozambique's national immunization program and in turn the HPV vaccination delivery partnership, however other important contextual factors also emerged as key drivers of the partnership. These were; the country's first lady's involvement as a champion, Gavi's requirement for the NIP to carry out assessments to demonstrate feasibility of the vaccination delivery model and delivery of HPV vaccines in schools which appertain to a separate sector, the MOE.
Gavi's partnership model, is the main driver of the country's national immunization collaboration which according to interviewed informants historically stemmed from the country becoming a beneficiary of Gavi grants in 2001. The two core Gavi partners WHO and UNICEF, whose task within the Alliance, are to provide in-country implementation support, were referred to by NIP staff as the "traditional partners". Each organization is known for specific roles, WHO for technical guidance and UNICEF for logistics and supplies, a fact that we learned has influenced the NIP to adopt the practice of allocating specific roles to potential partner organizations that express interest to collaborate with it.
"We are used to working with WHO for technical guidance. UNICEF they usually support us for vaccine logistics and supply" NIP Two other organizations, VillageReach an international non-governmental organization (NGO) and Fundação para o Desenvolvimento da Comunidade (FDC) a local organization were also considered as "usual" NIP partners and had supported immunization activities for more than five years prior to the HPV vaccine demonstration project launch. Further investigation revealed that that these two organizations had strong ties to Gavi.
VillageReach is an American based organization whose funding comes from the BMGF, a founding partner of Gavi

Institute)
In personnel had to carry out specific HPV vaccination activities at all levels from national to provinces, districts and schools.

Partnership Structure
The SNA results for reachability, distance, centralization, betweenness, density, structural holes, redundancy, homophily and average trust are shown in table 4. The network structure was found to contain a total of 50 actors (nodes) and 164 ties.
The reachability score is 100% meaning that there is at least one path connecting all actors in the network and each can be reached from whichever point one starts from (figure 2). The distance score, which is defined as the average number of edges in the shortest path between pairs, at 2.52 is short. Shorter distance in SNA is commensurate with faster and more accurate information flow. Combining the two SNA metrics scores with our first WHO health system governance characteristic hypotheses (Table 2) we can infer a partnership that has the capacity to effectively engage with and handle multiple scales. The perceived outcome survey scores corroborate this finding (Table 5). At 97.6%, the effectiveness average score was the highest of all outcome survey mean scores.
Furthermore, we noted 100% partners' agreement to three perceived outcome questions; 1) HPV vaccination partnerships ability for better execution, 2) better quality and improved response to challenges when they worked with multiple types of entities and 3) organizational hierarchical levels.  For the second WHO health system governance characteristic hypotheses, the centralization score was found to be neither low nor high at 48%, and average betweenness was 37.24 with a large standard deviation of 110.1. These scores are consistent with the network's outdegree statistics that revealed three outlier actors numbers 2, 5 and 24 (figure 2), around which the network is centralized. While actor number 2 had the highest outdegree score, this centralization value, means that the partnership is not highly centralized around 1 focal actor (e.g. EPI program). In addition, the network structure connectivity scores support the existence of effective relationships between these three key actors, indicating that the partnership could coordinate and respond rapidly to challenges. When triangulated with perceived outcomes survey results, we found a concordance as 100% of respondents, answered affirmatively for the question on their perception of HPV vaccination partnership's capacity to respond to challenges which had arisen during project implementation processes (Table 5). Qualitative data further supported the finding (see quotation below).
"The involvement of many organizations was very advantageous because we as the district directorate of health would not have been able to undertake all the activities within the short time that we had to prepare. The partners and their support helped us to reach where we would not have reached, for example sometimes they gave us fuel when we didn't have and even one hired a boat to reach some islands" (District Health Directorate) The third WHO health system governance characteristic is the capacity to combine and integrate different forms of knowledge whose hypotheses (Table 2)  The last three dimensions are interpreted here jointly because our conceptual framework postulates that a partnership's structure determines its performance, practices and consequently the outcomes. SNA measures have so far revealed a network characterized by high overall connectivity that is favorable for rapid and widespread diffusion of information. In addition, trust within the partnership is very high as evidenced by the high tie strength average score of 4.056 which is very close to the upper limit of five on the scale that respondents had been asked to rate professional trust on. Network outcome SNA measures (described in table 2) unveiled an efficient network structure characterized by a high number of structural holes and less redundancy. Majority (85%) of the nodes in the network require only 2 or 3 paths to reach them and 70% have only one way to connect with other nodes meaning that redundancy is only 30%. The network was also found to be heterogenous with a homophily E-I index scores of 0.195.
On triangulation with qualitative and perceived outcomes data so as to facilitate interpretation we found that these topological features of the network influenced Remarkably each partner organization was known for a specific role in this partnership. Correspondingly, the highest perceived outcomes drawbacks score was that of country ownership at 24%. Several issues were mentioned regarding country ownership but featuring prominently was the preference for a country based partner organization.
Respondents expressed their unhappiness on the inclusion of a non country-based partner in the HPV implementation partnership. Language barrier and lack of contextual knowledge were mentioned as some of the problems of the specific technical assistance that was provided by the particular partner who was considered foreign. The lack of participation in regular technical working group meetings was also noted as a hindrance to individuals based outside the country contributing effectively to the partnership. This is because TWG meetings was the forum where most of the partnership communication took place with updates on progress on processes being made, important discussions taking place and key decisions being made during the meetings. Short visits to the country to provide technical assistance were said to be ineffective by survey respondents and were even blamed for having largely contributed to the delay in the accomplishment of some HPV vaccination pilot implementation phase deliverables.

Discussion
Delving into the Mozambique HPV vaccination implementation stakeholder relationships in this case study, has revealed the importance of balance between donor and government influence, the necessity for adaptability to changing needs and the value of country ownership for an effective vaccine delivery partnership.
We observed a partnership that was equally driven by the MOH as well as the influence of the global Gavi  The third and final outcome measure of our study was the country ownership for which perceived outcome survey score at 79.5% at was among the lower scores. A salient factor that was elucidated was how the location, in or out of country, of a technical assistance provider affects the outcome of their support. Specific barriers that were encountered by the out of country international partner and which we have highlighted in our results section, are particularly informative and relevant for Gavi to consider when assessing potential technical assistance providers for countries in the current era of the partnership engagement framework (PEF).
In applying the Gavi FCE partnership framework we have found it to be a particularly useful tool for conducting a comprehensive partnership evaluation. This is because unlike previous predecessor partnership evaluation frameworks, namely the traditional and Brinkerhoff(34), the FCE framework approach is more wholistic resulting in consideration of an expanded set of partnership characteristics which influence partnership performance outcomes. While all three frameworks invariably describe the causal chain nature of the partnership characteristic from the contextual factors and prerequisites to the outcomes, the inclusion of the partnership structure together with the provision for the use of SNA tools to study it, highly enhanced the framework. These components of the FCE framework offer the possibility of unmasking partnership attributes that may not be overt.
The utilization of SNA tools in our study enabled us to visualize the otherwise invisible constellation of relationships that make up Mozambique's HPV vaccine delivery partnership network. The image of the network structure facilitated a rapid identification of key actors within the implementation collaboration. In addition, SNA measurements demonstrated the partnership network structure's effectiveness and efficiency in information flow and innovation adoption. Such information can only be attained using SNA methods thus making it a powerful methodology to inform other vaccine delivery endeavors and health program partnerships. Furthermore, the FCE framework guided our utilization of mixed methods that allowed for a broader analysis and exploration of the partnership. We therefore recommend the use of the Gavi FCE framework for evaluation of global health partnerships.

Strengths And Weaknesses
Our study's participation rate was quite high with 40 individuals from different entities and different health system levels responding to both the network and perceived outcomes survey. This is a relatively large sample size especially for the qualitative data component; however, the accuracy of network data rely on a full census of a network and this study's response rate may lead to underestimations of network density in particular.
Ties beyond the set boundary were not explored and there might be missing data that was not captured. Mixed methods captured different types of data that were triangulated, and this facilitated interpretation. The generalizability of our findings may be limited to low and middle-income countries whose socioeconomic features are similar to those of Mozambique.

Conclusion
While there is room for improvement on the country ownership aspects of the partnership,  Mozambique HPV vaccine demonstration project partnership network structure