Effectiveness of educational outreach in infectious diseases management: a cluster randomized trial in Uganda

Background Integrated Infectious Diseases Capacity Building Evaluation (IDCAP) teams designed and implemented two health worker in-service training approaches: 1) an off-site classroom-based integrated management of infectious diseases (IMID) course with distance learning aspects, and 2) on-site support (OSS), an educational outreach intervention. We tested the effects of OSS on workload and 12 facility performance indicators for emergency triage assessment and treatment, HIV testing, and malaria and pneumonia case management among outpatients by two subgroups: 1) mid-level practitioners (MLP) who attended IMID training (IMID-MLP) and 2) health workers who did not (No-IMID). Methods Thirty-six health facilities participated in the IDCAP trial, with 18 randomly assigned to Arm A and 18 to Arm B. Two MLP in both arms received IMID. All providers at Arm A facilities received nine monthly OSS visits from April to December 2010 while Arm B did not. From November 2009 to December 2010, 777,667 outpatient visits occurred. We analyzed 669,580 (86.1 %) outpatient visits, where provider cadre was reported. Treatment was provided by 64 IMID-MLP and 1,515 No-IMID providers. The effect of OSS was measured by the difference in pre/post changes across arms after controlling for covariates (adjusted ratio of relative risks = a RRR). Results The effect of OSS on patients-per-provider-per-day (workload) among IMID-MLP (aRRR = 1.21; p = 0.48) and No-IMID (aRRR = 0.90; p = 0.44) was not statistically significant. Among IMID-MLP, OSS was effective for three indicators: malaria cases receiving an appropriate antimalarial (aRRR = 1.26, 99 % CI = 1.02-1.56), patients with negative malaria test result prescribed an antimalarial (aRRR = 0.49, 99 % CI = 0.26-0.92), and patients with acid-fast bacilli smear negative result receiving empiric treatment for acute respiratory infection (aRRR = 2.04, 99 % CI = 1.06-3.94). Among No-IMID, OSS was effective for two indicators: emergency and priority patients admitted, detained or referred (aRRR = 2.12, 99 % CI = 1.05-4.28) and emergency patients receiving at least one appropriate treatment (aRRR = 1.98, 99 % CI = 1.21-3.24). Conclusion Effects of OSS on workload were not statistically significant. Significant OSS effects on facility performance across subgroups were heterogeneous. OSS supported MLP who diagnosed and treated patients to apply IMID knowledge. For other providers, OSS supported team work to manage emergency patients. This evidence on OSS effectiveness could inform interventions to improve health workers’ capacity to deliver better quality infectious diseases care. Electronic supplementary material The online version of this article 10.1186/s12889-016-3375-4) contains supplementary material, which is available to authorized users.


Background
Sub Saharan Africa experiences 37 % of the total deaths attributable to infectious diseases globally [1], but contains only 3 % of the world's health workers [2]. The shortage of physicians in sub Saharan Africa is at critical levels, with only 2 physicians per 10,000 population compared to 28 in high-income regions [3]. This shortage of physicians has led to many physician roles, such as initiation of antiretroviral therapy shifting to mid-level practitioners (MLP) with fewer years of medical training [4], formally [5][6][7] and informally [8,9]. Yet as reported in one Ugandan study, many of these MLP have not been trained to take on these tasks [10]. They therefore require capacity building to meet the demands of changing health policies and guidelines [11]. Some studies have shown that with additional training MLP can competently execute physicians' roles and, in some instances, match their performance [12][13][14]. However, most trainings for MLP are classroom-based, often taking them away from their workstations [15,16], disrupting workflows and limiting patients' access to care. Several systematic reviews show that classroom-based training may not equip the attendants with sufficient skills to ensure adherence to clinical guidelines [17][18][19].
There is increasing interest from experts and donors in favor of educational outreach and continuous quality improvement (CQI) approaches [20][21][22][23][24] that integrate across diseases [25][26][27][28]. Facility-based educational outreach trainings increase opportunities for team-based interactions during MLP learning [20,29] and reduces MLP time away from health facilities. Several studies have shown that educational outreach both with and without CQI activities can improve the quality of patient care [5,[30][31][32][33][34][35]. It may also build capacity in patient care for almost all providers, without increasing workload on the MLP who do not attend trainings.
Between 2009 and 2010, the curriculum development and mobile teams of the Integrated Infectious Diseases Capacity Building Evaluation (IDCAP) developed and delivered two novel interventions to build capacity for treatment and prevention of infectious diseases: 1) the Integrated Management of Infectious Disease (IMID) course with distance learning aspects, taught at the Infectious Diseases Institute in Kampala, Uganda, and 2) facility-based on-site support (OSS) visits, an educational outreach intervention with CQI activities [29]. Implemented as a mixed design with pre/post and cluster randomized trial components, IDCAP tested the effect of IMID training and OSS on individual clinician competence [36], practice [37], mortality among children less than five years of age [20,38], and facility performance [33,39]. IMID was associated with a statistically significant 9.8 % absolute increase in clinical competence, whereas OSS was not associated with an incremental improvement in the same outcomes. [36]. For clinical practice, IMID was associated with statistically significant improvements in taking a patient's history and physical examination, and OSS was associated with an incremental improvement in these tasks [37]. Child mortality increased in both arms, but the incremental effect of OSS delayed the increase during the intervention [38].
In the main trial analysis of facility performance, [33] Weaver et al. reported the effects of IDCAP's IMID and OSS on 23 facility performance indicators. Facility performance refers to the process of care, and may reflect the contributions of more than one provider at a facility. For example, the proportion of malaria suspects with a malaria test result recorded (Indicator 4), reflects whether or not the clinician ordered the test, and the laboratory professional performed the test and recorded the result. Of the 23 indicators, 12 were based on a data surveillance system with data on each outpatient visit. The other 11 indicators were based on Ministry of Health Registers for HIV care, maternal, child and newborn health and tuberculosis. The 12 indicators based on the data surveillance system are presented in Table 1, and include three indicators for emergency triage assessment and treatment, four indicators for malaria case management, four indicators for case management of respiratory illness, and one indicator for HIV testing.
Among those 12 indicators, Weaver et al.'s reported that IMID alone was associated with statistically significant pre/post improvements in three indicators: outpatients triaged (Indicator 1), emergency and priority patients who were admitted, detained or referred (Indicator 2), and pneumonia suspects under age five assessed for pneumonia (Indicator 8). A combination of IMID and OSS was associated with statistically significant improvements in four indicators: outpatients triaged (Indicator 1), emergency and priority patients who were admitted, detained or referred (Indicator 2), 3) estimated malaria cases who received appropriate antimalarial treatment (Indicator 5), and patients with a negative malaria test result who were prescribed an antimalarial (Indicator 6), for which a decrease was an improvement. The incremental effect of OSS was not statistically significant for any of the 12 indicators, but many of the effect sizes were large.
We hypothesized that OSS improved facility performance more among the providers who attended IMID training (IMID-MLP) compared to the ones who did not (No-IMID). Weaver et al.'s analysis included all patients, and we suspected that the No-IMID subgroup could have diluted the effects of OSS in the IMID-MLP subgroup. We therefore sought to test the effect of OSS on the 12 indicators based on the data surveillance system in the two subgroups of health providers: IMID-MLP and No-IMID. The analysis was limited to only outpatient visits with data to identify provider cadres,  [16,39,53] Case management of respiratory illness 8 Proportion of pneumonia suspects aged under 5 years assessed for pneumonia Numerator: Number of child pneumonia suspects with at least one of the three following assessment results recorded: 1) abnormal chest sounds, 2) chest in-drawing, and 3) rapid breaths per minute. A pneumonia suspect was defined as any child aged under five years presenting with cough or who received a diagnosis of "pneumonia" or "cough/cold no pneumonia". Denominator: Number of child pneumonia suspects. Note: The definition of suspect focused on children with cough; difficulty in breathing was inadvertently omitted from the form. [54] 9 Estimated proportion of patients aged under 5 years diagnosed with pneumonia who received appropriate antibiotic treatment Numerator: Number of children diagnosed with pneumonia treated with appropriate antibiotic, where appropriate antibiotic treatment referred to six drugs on the revised Medical Form 5: amoxicillin, benzyl penicillin, erythromycin, chloramphenicol, gentamicin, cotrimoxazole, and 11 other drugs that were specified: ampicillin, azithromycin, cefixime, ceftriaxone, cefuroxime, co-amoxiclav, gatifloxacin, levofloxacin, penicillin, phenoxymethyl penicillin, ampiclox (amoxicillin and cloxacillin). For patients who were prescribed an antibiotic and data on drug availability were missing, we applied the "in-stock" rate for patients with those data. Denominator: Number of children diagnosed with pneumonia [54] and whether or not the provider attended IMID training. In this article, the term "indicator" refers to the facility performance indicators. Also, during the ethical review of the IDCAP protocol, the School of Medicine Ethics and Research Committee of Makerere University expressed concern that the interventions would affect the distribution of the workload. The data surveillance system data provided an opportunity to examine whether workload increased among the IMID-MLP with OSS, and among the No-IMID health workers with OSS. We therefore tested the effect of OSS on changes in workload in each subgroup of providers.

Study design
We conducted a secondary analysis of IDCAP data. As noted above, IDCAP was a mixed design study consisting of pre/post and cluster randomized trial components. It was implemented at 36 health center IVs (HCIV) or comparable health facilities across Uganda. Eighteen of the 36 health facilities were randomized to Arm A and the other 18 to Arm B. In both arms, two MLP from each of the 36 health facilities received IMID training. From April to December 2010, all providers at Arm A facilities received nine monthly OSS visits while Arm B did not. The pre/ post component measured the effects of the combination of IMID training and OSS in Arm A, and IMID training alone in Arm B. The cluster randomized trial component measured the incremental effect of OSS as the difference between the pre/post effects in the two arms. Figure 1a presents the design of the main test of facility performance indicators, which is reported as the overall result below. Note that the overall results presented in Table 5 differ from the results of Weaver et al.'s analysis, because only outpatient records with data on provider cadre and whether or not the provider attended IMID training were analyzed. Figure 1b presents the design of the subgroup analysis.
The process of randomization, blinding, sample size calculation, and recruitment for the cluster-randomized trial were previously reported in Weaver et al. [33]. A The IDCAP protocol is summarized by Naikoba et al. [20]. The full protocol can also be accessed as supplementary file in both Mbonye et al. [39], Weaver et al. [33] and is available as supporting information to this paper in Protocol 1. The CONSORT checklist for this subgroup analysis is available as Additional file 1.

Sites, participants, and inclusion criteria
The 36 study health facilities included 31 health center IV (HCIV) and five small hospitals comparable in size to HCIV. A HCIV serves a population of about 100,000 people providing basic preventive and curative care, and is at the highest referral point for a health sub-district [40]. Two MLP, defined as clinical officers, registered Note: This indicator included anyone who said they knew their HIV status in the protocol, and was revised as described above.
Abbreviations: AFB Acid-fast bacilli, CQI Continuous Quality Improvement, HIV Human Immunodeficiency Syndrome, MOH Ministry or Health nurses or midwives, from each health facility participated in the IMID training. Inclusion criteria for facilities and IMID participants have been described elsewhere [20,29]. Briefly, the key inclusion criteria for selecting IMID participants was involvement in daily management of patients in the outpatient clinics, and spending over 80 % of the time seeing patients at the participating health facilities [20]. All health workers including IMID participants were invited to participate in the OSS sessions. All outpatients participated as part of the routine healthcare seeking process. For the subgroup analyses, as noted above, only records of outpatient visits with data on the health provider's cadre and whether or not the health provider attended IMID training were included.

Interventions
IMID was a three-week core course followed by two one-week booster courses at 12 and 24 weeks after the core course and distance learning [29]. The core course was taught in four sessions at the Infectious Diseases OSS was a two-day visit to the health facilities in Arm A delivered by mobile teams for nine consecutive months from April to December 2010 [20,29]. Each mobile team consisted of four members: a medical officer with CQI experience, a clinical officer, a laboratory technologist, and a registered nurse. Each visit focused on one OSS topic as well as follow-up from the previous sessions and was characterized by four distinct activities: the multidisciplinary team training, one-on-one mentorship, breakout sessions, and CQI activities.

Outcomes
The outcomes were patients per provider used to measure changes in workload, and 12 facility performance indicators based on the data surveillance system whose definitions by numerator and denominator are reported in Table 1, as well as sources of the indicators. Indicator 16, proportion of patients with an HIV test recorded, differs from the earlier report [33], because it excludes registry data from antenatal and tuberculosis clinics.

Data collection
The data source was the revised Uganda Ministry of Health outpatient medical form (MF5) [41]. Using the MF5, health workers prospectively collected data on every outpatient visit from November 2009 to December 2010. From March 2010, data entry assistants were stationed at each health facility to capture these data electronically in an Epi Info® Version 3.2 (U.S. Centers for Disease Control and Prevention, Atlanta, GA) database. At the end of each month, the data entry assistants transmitted electronic data to the Infectious Diseases Institute where data for all health facilities were merged using Microsoft Excel® 2010 (Microsoft Corporation, Redmond, WA, USA), and further cleaned.

Data analysis
We used frequencies to describe provider, patients, and outcomes across arms, subgroups, and time periods. The effect of OSS was estimated as a difference in pre/post (Time 0/Time 1) changes between Arm A and Arm B. Using a generalized linear latent and mixed model [42], data on the number of patients per provider were analyzed as continuous dependent variables with main effects for arm, subgroup, time period and their interaction to estimate pre/post differences within each subgroup in each arm and ratios of relative risk (RRR) for the effect of OSS within each subgroup. The unit of analysis was a subgroup-facility-month, and regressions were estimated with clusters for provider and facility with robust standard errors.
The RRR for a given performance indicator was estimated with main effects for arm, time period and their interaction, using data for the total sample and for each subgroup and the generalized linear model with a Poisson family and log link as described earlier [33,39]. The unit of analysis was facility month, and all regression analyses were clustered on the health facility. To adjust for over dispersion and using Poisson rather than binomial family, the regressions were estimated with robust standard errors. Multiple comparisons were addressed by basing the statistical tests on α = 0.01 and all results were presented with 99 % confidence intervals (CI).
All regression analyses were adjusted for facility type (public or private not-for-profit), facility level (HCIV or small hospital), data entry assistant on-site, Baylor and previous participation in the national HIV CQI program (see description of strata in [33,39]). Indicators 1-7 and 10-11 were also adjusted for staffing and patient age (<5, > = 5 years), indicators 8-9 for staffing, and indicator 16 for staffing, TB suspect and patient age group (2-14 months, 18 months -13 years and ≥14 years) to reflect HIV testing guidelines. All data analyses were conducted using Stata® version 12 (StataCorp, College Station, Texas, USA).

Participant flow
The flow of the participating health facilities, and IMID training and OSS participants is presented in Fig. 2 [33]. All health facilities participated up to the end of the study. Of the 36 individuals who participated in IMID training, one person in Arm A and two in Arm B did not attend one or more boost courses. Attendance of OSS sessions was also not consistent. Out of 513 clinical staff expected to attend multi-disciplinary team sessions, only 440 (86 %) attended at least once; out of 499 clinical staff expected to attend the breakout session, only 344 (69 %) attended at least once; and out of 351 clinical staff expected to attend the clinical mentoring and coaching session, only 186 (53 %) attended at least once. Among those who attended at least one OSS session, the mean number of sessions attended per clinical staff was 4.93 for multi-disciplinary team sessions, 3.64 for breakout sessions and 4.01 for clinical mentoring and coaching sessions.
A total of 777,667 outpatients visited the 36 health facilities during the study period. Of these visits, 669,580 (86.1 %) had data on the provider's name and cadre, which allowed us to determine whether or not the provider attended IMID training, and were included in this analysis ( Table 2). During these visits, patients were seen by 64 IMID-MLP and 1,515 No-IMID providers. Data on 108,087 (13.9 %) visits with missing (91,006), unknown (14,686) or indeterminate (2,395) data on the provider were omitted from the analysis.

Patient volume
In Table 3, baseline data for the number of providers, the number of patients seen, and the estimated number of patients per provider per calendar day is reported by time period, arm and subgroup. The number of patients per provider per calendar day was higher in Arm B with more hospitals than in Arm A in the IMID-MLP subgroup (4.3 vs. 3.7) and similar across arms in the No-IMID subgroup (1.1 vs. 1.1).

Patient care
In Table 4, baseline data for each indicator are reported by both arm and subgroup. Performance in Arm B was generally higher than in Arm A within both IMID-MLP and No-IMID subgroups for eight out of the 12 indicators. For indicators 6 (Proportion of patients with a negative malaria test result who were prescribed an antimalarial), and 7 (Proportion of patients with a positive malaria test result who were prescribed an antibiotic) a lower percentage indicated better performance, and Arm A had lower percentages than Arm B for both indicators. Performance was highest for indicator 5 (estimated proportion of malaria cases who received an appropriate antimalarial). For the remaining indicators, performance was generally low, did not exceed 64 % in both arms and subgroups, and was as low as 3.1 % for indicator 8 (Proportion of pneumonia suspects aged under 5 years assessed for pneumonia) in the IMID-MLP subgroup of Arm A.

Patient volume
As shown in Table 3  Denotes that the effect of OSS was significant at the .01 level and * Denotes that the effect of OSS was significant at the .05 level The 99% confidence intervals (CI) are based on the .01 level of significance Abbreviations: aRR adjusted relative risk, aRRR adjusted ratio of relative risks, IMID Integrated Management of Infectious Disease, OSS On-site support, MLP Midlevel practitioners, RR Relative Risk measured the comparisons in pre/post change from Time 0 to Time 1; RRR Ratio of Relative Risk Measured the effect of OSS in each subgroup Estimates were adjusted for: whether the facility received the on-site intervention from Baylor International Pediatric AIDS Initiative or not, facility was implementing continuous quality improvement prior to IDCAP trial, level of health facility (small hospital or health center IV), facility ownership (public or private-not-for profit) and data entry assistant on-site

Facility performance
The effect of OSS on 12 facility performance indicators is reported in

Discussion
There have been limited rigorous randomized education trials that demonstrate the effects of health worker training approaches on patient care outcomes in the resource constrained world. In this study, we have evaluated the effect of an educational outreach intervention involving nine monthly on-site support (OSS) sessions on health facility performance in two groups of trainees those who were provided with five weeks of off-site classroom education and those who did not have this opportunity. The OSS intervention did not significantly increase patient load within the subgroup of health providers who did not participate in the IMID training. Among the mid-level practitioners (MLP) who received both IMID training and OSS, the number of patients per provider per calendar day increased; but the changes in workload attributable to OSS were not statistically significant. The effects of OSS on facility performance indicators were heterogeneous. Patients treated by health providers who attended the IMID training benefited from improved case management of malaria and respiratory infections. Patients treated by providers without the IMID training benefited from improvements in emergency triage, assessment and treatment.
The number of patients per provider per calendar day was much lower than we expected. On average, the health providers with IMID training saw 4.64 patients per calendar day, and the other providers 0.83 indicating that the MLP saw 5 times more patients than other providers. This was expected since MLP involved in daily management of patients in the outpatient clinics and spent over 80 % of their time at the health facility seeing patients were prioritized during the selection process [20]. These results for the MLP with IMID training are consistent with those of the Institute for Health Metrics and Evaluation's report that showed health providers saw an average of three to five patients per day [43]. These results also indicate that health workers in Uganda treat very few patients per day, but this finding contrasts with other reports in which complaints of heavy workload have been cited among health workers in Uganda [44] and other sub-Saharan Africa countries [45,46]. The reported Denotes that the effect of OSS was significant at the .01 level and * Denotes that the effect of OSS was significant at the .05 level The 99% confidence intervals (CI) are based on the .01 level of significance Abbreviations: aRRR adjusted ratio of relative risk, IMID Integrated Management of Infectious Disease, OSS On-site support, MLP Mid-level practitioners, AFB Acidfast bacilli, HIV HumanImmunodeficiency Syndrome, TB Tuberculosis aRRR adjusted Ratio of Relative Risk measured the effect of OSS in each subgroup Estimates were adjusted for: whether the facility received the on-site intervention from Baylor International Pediatric AIDS Initiative or not, facility was implementing continuous quality improvement prior to IDCAP trial, level of health facility (small hospital or health center IV), facility ownership (public or private-not-for profit) and data entry assistant on-site number of patients seen per provider is an average across the estimated working days for each of the study time periods. It is possible that providers did not work on all estimated days and thus treated a higher number of patients on days when they worked [47]. In a review of studies in six countries including Uganda, it was found that 35 % of the health care providers were absent during their assigned shifts [48]. Among many factors, the absence usually arises because health workers are on leave, attending trainings off-site, tending to their gardens, or doing work in private clinics to supplement their income [47]. If health workers are only on duty an average of 65 % of the time, then the average number of patients per provider per day would increase by one-third. It is also possible that the number of patients' visits per day was higher on some days than others. Adding in the fluctuations in patient visits and it possible that some clinicians would manage much higher number of patients on some days. Further analysis is required to determine how the number of patients per day seen by clinicians varies by day and across facilities.
Among the health workers without the IMID training, this analysis showed no statistically significant effect of OSS on changes in the volume of patients they managed. In fact, the number of patients seen per provider per calendar day among these health providers remained the same before and during the OSS. Yet a slight increase in volume of patients seen per provider per calendar day among the MLP with IMID training was observed during OSS, although this increase was not statistically significant. This increase in patient load among IMID trainees coupled with no increase among providers who did not attend IMID training, shows a general increase in patient volume overall during OSS, perhaps pointing to an appreciation of better quality services being offered at the health facilities. Rather than shift tasks and workload among cadres, OSS sought to improve facility performance within each cadre's scope of practice. This could explain why there was no significant increase in patient volume among health workers without IMID training and decrease among health workers without the IMID training.
Overall, OSS was effective for three of the 12 facility performance indicators in this subgroup analysis, compared to none in the main IDCAP trial results [33]. In this subgroup analysis, only records with complete data on the provider's cadre and IMID attendance were analyzed unlike in Weaver et al.'s [33] analysis that included all patient records. It is plausible that providers with incomplete data also attended fewer OSS or benefited less from them and diluted the effects. We saw evidence of the no-IMID subgroup diluting the OSS effects of the MLP with IMID especially for indicators 11 (patients with AFB smear negative results who received empiric treatment for acute respiratory infection) and 16 (patients with an HIV test result recorded). On the other hand, evidence of the MLP with IMID diluting the effects of the no-IMID subgroup was observed for indicator 2 (emergency and priority patients who were admitted, detained or referred).
The effects of OSS among the MLP with IMID tended to be statistically significant for indicators that reflected clinical decision-making, such as prescribing antimalarials and antibiotics. In contrast, the effects of OSS among the no-IMID subgroup were for emergency, triage and assessment indicators that required a broader team effort to identify and manage patients appropriately. Although OSS did not seek to change a cadre's scope of practice, it did seek to empower all providers to triage patients and organize immediate care for emergency and priority patients.
The effects of OSS observed in this analysis were heterogeneous across indicators as well as across subgroups. In the overall sample, the average effect size was 34 % and higher than estimates elsewhere. For instance, reviews of continuous medical education and effect of educational outreach visits on health workers' training reported median improvements ranging from 6.9 to 13.9 % [19] and from 5.6 to 21 % [49] respectively. The effect sizes for the emergency, triage and assessment and malaria case management indicators were generally larger than the effect sizes for case management of respiratory infections and HIV testing. The indicators for the first two coincide with the first two OSS. The OSS sessions for the latter two were delivered later in the intervention, an indication that perhaps, the expectation of immediate improvements over a shorter period of time may have been unrealistic. There is need for monitoring these indicators over more months to observe whether facility performance will improve over time.
Also future researchers should consider designing their trials for a subgroup analysis. Within resource-limited settings in many sub-Saharan African countries, clinicians and other higher level cadres are the ones usually selected to attend continuous medical education trainings to support them improve the quality of diagnosis and treatment. Allied health workers, nurses and other lower level providers often miss out on such opportunities since they mainly play supporting roles to clinicians in providing patient care. OSS would therefore be a great opportunity to improve capacity of these non-clinician health workers to dispense better quality patient care in the entire patient care process. For clinicians and higher level cadres, OSS would be important in supporting them to apply knowledge and skills in diagnosis and treatment acquired from classroom training.
To our knowledge, the IDCAP trial is one of the first randomized trials that evaluated the effects of an integrated educational outreach and quality improvement intervention in sub-Saharan Africa. The results of this study therefore provide important evidence that could inform capacity building interventions and policies for health workers in resource limited settings.

Limitations
As noted in Weaver et al. [33] sample size calculations for the IDCAP trial were based on comparison across arms and an α = 0.05. Consequently, this subgroup analysis may have been underpowered to detect differencein-difference at α = 0.01 when comparing effect of OSS on facility performance between arms. OSS sessions were delivered gradually for nine months, and its effect may have been underestimated by measuring it during its implementation rather than after.

Generalizability
This study was conducted in a sample of sub-district level referral HCIVs or comparable small hospitals that met the inclusion criteria. These results can thus be generalized for similar health facilities in Uganda and other countries, and for indicators for which OSS was effective.

Conclusions
The MLP with IMID training managed thrice as many patients per day compared to other providers. Out of the 12 facility performance indicators, OSS improved three indicators across all patients and among patients managed by MLP with IMID training, but only two indicators among patients managed by providers without IMID training. These results show that OSS supported MLP who diagnosed and treated patients to apply knowledge from IMID. For other providers, OSS supported team work to manage emergency patients. Since provision of patient care is a team effort with clear roles for every health worker, health workers engaged in clinical diagnosis and treatment may need OSS in addition to classroom training, while the nurses and other providers who play a supporting role to the clinicians will need OSS to effectively contribute more to patient care.