What is actually measured in process evaluations for worksite health promotion programs: a systematic review
BMC Public Health volume 13, Article number: 1190 (2013)
Numerous worksite health promotion program (WHPPs) have been implemented the past years to improve employees’ health and lifestyle (i.e., physical activity, nutrition, smoking, alcohol use and relaxation). Research primarily focused on the effectiveness of these WHPPs. Whereas process evaluations provide essential information necessary to improve large scale implementation across other settings. Therefore, this review aims to: (1) further our understanding of the quality of process evaluations alongside effect evaluations for WHPPs, (2) identify barriers/facilitators affecting implementation, and (3) explore the relationship between effectiveness and the implementation process.
Pubmed, EMBASE, PsycINFO, and Cochrane (controlled trials) were searched from 2000 to July 2012 for peer-reviewed (randomized) controlled trials published in English reporting on both the effectiveness and the implementation process of a WHPP focusing on physical activity, smoking cessation, alcohol use, healthy diet and/or relaxation at work, targeting employees aged 18-65 years.
Of the 307 effect evaluations identified, twenty-two (7.2%) published an additional process evaluation and were included in this review. The results showed that eight of those studies based their process evaluation on a theoretical framework. The methodological quality of nine process evaluations was good. The most frequently reported process components were dose delivered and dose received. Over 50 different implementation barriers/facilitators were identified. The most frequently reported facilitator was strong management support. Lack of resources was the most frequently reported barrier. Seven studies examined the link between implementation and effectiveness. In general a positive association was found between fidelity, dose and the primary outcome of the program.
Process evaluations are not systematically performed alongside effectiveness studies for WHPPs. The quality of the process evaluations is mostly poor to average, resulting in a lack of systematically measured barriers/facilitators. The narrow focus on implementation makes it difficult to explore the relationship between effectiveness and implementation. Furthermore, the operationalisation of process components varied between studies, indicating a need for consensus about defining and operationalising process components.
Employees with unhealthy lifestyle behaviors and overweight or obese employees are less productive at work, show a decreased work ability and take more sick days compared to employees with a healthy lifestyle [1–4]. An unhealthy lifestyle can be characterized by one or more of the following behaviors; low physical activity levels, an unhealthy diet, smoking, frequent alcohol use and poor levels of relaxation (i.e. mental health and vitality). Because employed adults spend about half of their workday waking hours at their workplace, the worksite may be an effective setting to increase employees health and productivity [5–8]. Therefore, in the past decade, numerous worksite health promotion programs (WHPPs) have been conducted to improve employees’ health and lifestyle. However, not all of these programs were successful [5–7].
In order to change employees’ health and lifestyle, programs need to work as intended (and therefore avoid theory failure). Additionally, they must be effectively transferred from research to practice and be maintained over time . Programs often fail to reach potential participants adequately due to a lack of adoption, communication, or program sustainability which could lead to low participation levels (in other words, program failure) . Furthermore, research starts to recognize the importance of evaluating different implementation outcomes (such as recruitment, dose delivered, dose received, fidelity, satisfaction, and maintenance) and the contextual factors that hinder or facilitate the implementation of a WHPP [11–14]. These contextual factors are related to characteristics of the context, organization, implementer, program and participants [11, 12, 15]. Since WHPPs are often comprehensive interventions with multiple components, it is difficult to determine the overall level of implementation and investigate which specific intervention components have been successful [14, 16].
A review by Durlak and Dupre (2008) showed that higher levels of implementation improve program outcomes, suggesting that there should be an adequate focus on the implementation process . However, most research focuses primarily on measuring the effects of a WHPP, with an occasional process evaluation performed after implementation. However, systematic process evaluations can produce valuable insights into the interpretation of the (lack of) effects of an intervention by identifying successful and unsuccessful program components, thereby allowing researchers to optimize their program [16–19]. Furthermore, process evaluations can help to identify barriers and/or facilitators influencing the implementation process, while taking into account the different actor levels at which these factors play a role [16, 18]. These are valuable outcomes that can be used to improve program implementation in the future and across other settings. Hence, effect evaluations should be accompanied by systematic and real-time process evaluations [20, 21].
Murta et al. (2007) looked at the quality of process evaluations accompanying controlled trials studying individual based stress management interventions. They showed that process evaluations are often incomplete and not systematically conducted, meaning that they lacked a theoretical framework and were not planned prior to implementation . However, the quality of the process evaluations that were included in this review was insufficient to make sound conclusions in terms of best predictors for successful interventions. Additionally, Murta’s review only focused on evaluations of occupational stress management interventions published between 1977 to 2003. So their conclusion cannot directly be transferred to recent WHPP focusing on other lifestyle behaviors. In their review, Murta also concluded that the framework of Steckler and Linnan proved to be a useful tool to conduct process evaluations. But in this framework little attention has been given to the great variety of contextual factors that can either hinder or facilitate the implementation process. We therefore recently proposed a framework for a systematic and comprehensive process evaluation based on several theoretical frameworks to gain insight into the implementation process (Figure 1) [11, 12, 14, 15, 23]. The four main aspects of this framework relate to determinants of implementation that may influence the implementation process, and the implementation process itself (i.e. adoption, implementation and continuation). These four main aspects are operationalized using a combination of the framework of Steckler and Linnan for process evaluations and the RE-AIM framework [14, 23]. So, in order to gain insight in the implementation process, eight different process components at three different actor levels (macro-level: organization and management; meso-level: implementer; micro-level: participant) need to be evaluated using a mixed methods approach . Of these eight components, six components focus on implementation (reach, recruitment, dose delivered, dose received, fidelity, and satisfaction), one component (maintenance) on continuation and the eighth component (context) refers to the determinants of implementation. Since many implementation determinants can be identified, context is further defined by categorizing the barriers and/or facilitators into five main categories: 1) characteristics of the socio-political context, 2) characteristics of the organization, 3) characteristics of the implementer, 4) characteristics of the intervention program, 5) characteristics of the participant [11, 12, 15].
Some reviews have focused on process evaluations and relevant implementation barriers and/or facilitators [10, 12, 13, 22, 24, 25]. However, these reviews focused on domains other than WHPPs [12, 13, 24, 25], limited the identification of barriers and/or facilitators to those affecting WHPP participation levels , or focused only on process evaluations accompanying stress management interventions . To our knowledge, there has been no systematic review that has examined process evaluations for worksite health promotion programs targeting lifestyle change among employees. The aim of this review was therefore to: (1) further our understanding of the quality of process evaluations alongside effect evaluations for worksite health promotion programs (WHPPs), (2) identify barriers/facilitators affecting implementation, and (3) explore the relationship between effectiveness and the implementation process.
Literature search and study selection
For this systematic review, peer-reviewed studies were eligible for inclusion when they reported an effect evaluation as well as a process evaluation for a worksite health promotion intervention focusing on stimulating a healthy lifestyle (physical activity, nutrition, smoking, alcohol use and relaxation) published in English between 2000 and July 2012. The literature search was conducted in two steps.
Step 1: a literature search was in the online databases Pubmed, EMBASE, PsycINFO, and the Cochrane Central Register of Controlled trials for peer-reviewed WHPP effect evaluations published in English from 2000 to July 2012. Searches included the following combination of keywords: (Randomized controlled trial OR controlled trial) AND (worker* OR employee* OR worksite OR work environment) AND (worksite health promotion OR lifestyle intervention) AND (absenteeism OR sickness absence OR body mass index OR lifestyle OR health behavior OR cholesterol level). To ensure no studies were overlooked this search string was repeated in each database with the addition of the following keywords for each lifestyle behavior separately: ‘physical activity’, ‘smoking’, ‘alcohol use’, ‘healthy diet’ and ‘relaxation’. Since we were only interested in studies focusing on change in actual behavior rather than change in attitude, social norm or self-efficacy keywords were added on outcome measures related to actual behavior change. All the keywords were specified for each database using Mesh or Thesaurus terms. The complete search strategy for each database and lifestyle behavior are presented in Additional file 1. The inclusion criteria for the first stage of the selection process were: (1) a randomized controlled (RCT) or controlled trial (CT), (2) an evaluation of the effects of worksite health promotion interventions focusing on physical activity, smoking cessation, alcohol use, healthy diet and/or relaxation at work, and (3) targeting employees aged 18-65 years. With respect to the second inclusion criteria it should be noted that studies investigating interventions that primarily aim to promote a healthy lifestyle as well as interventions that primarily aim to prevent musculoskeletal disorders of which the intervention contains one of the lifestyle components mentioned above were included. The first author (DW) performed an initial selection based on the titles and abstracts of all papers reporting on effect evaluation. The abstracts of the effect evaluations were presented to the fourth author (SD), who was blinded for authors, affiliations, journal and year of publication. In a consensus meeting between both authors (DW and SD), a final selection of effects evaluations was made. When an abstract contained insufficient information or when the authors disagreed, the full paper was retrieved and read. When disagreement persisted, the second author (LE) was asked to decide about eligibility.
Step 2: DW checked whether the selected effect evaluations were accompanied by a peer-reviewed process evaluation. The inclusion criteria for the second stage of the selection procedure were: (1) published in English between 2000 and July 2012, and (2) reporting on implementation/process outcomes. DW first contacted the corresponding authors by e-mail to ask whether the effect evaluation had been accompanied by a paper on implementation/process outcomes. If the corresponding author did not respond or could not provide the requested information, DW first checked the full text paper of the effect evaluation for references to a related process evaluation (in the reference list and/or text). If nothing was found, DW searched all four databases using the title of the study (if known), trial registration number and the names of all authors for an additional formative, implementation or process evaluation. If based on title and abstract it was unclear whether the paper included information on the implementation process of the intervention, the full text paper was retrieved and read.
The full papers were retrieved for all eligible effect evaluation papers paired with a published process evaluation paper. These complete studies were independently assessed for eligibility by the first three authors (DW, LE and PE) before being included in the review. During this final round, LE and PE were blinded for authors, affiliations, journal and year of publication. All references used in eligible effect evaluations and process evaluations were checked by the first author (DW) for other relevant publications that might have been missed in the electronic search (‘snow ball’ procedure).
Methodological quality assessment
In order to answer the first research question, a methodological quality assessment was performed. The methodological quality of all papers included in the review was independently assessed by the first three authors (DW, LE and PE) using a checklist (Table 1). Disagreements between reviewers were discussed and resolved during consensus meetings.
The criteria for the assessment of the included effect evaluations were based on the methodological guidelines for systematic reviews developed by the Cochrane Back Review Group . These guidelines were developed for RCTs studying low back pain. Some Cochrane criteria were therefore omitted or adapted to fit the studies included in this review. Other reviews focusing on worksite interventions have used an adapted version of this guideline [5, 27]. The final criteria list consisted of three main categories: internal validity (n = 8), descriptive criteria (n = 5) and analysis (n = 3). Items were scored as positive (+), unsatisfactory (+/-) or negative (-). If an item was not applicable, this was stated. The quality of studies was considered to be ‘above average’ if the overall validity score (V) was above 50%, and a score above 75% was considered to represent relatively good quality [26, 28]. The overall validity score (V) was based on the eight internal validity criteria (V1-V8). When items were not applicable (N/A), they were not included in the percentages.
Given the absence of a standardized assessment form, we defined our own criteria for the methodological quality assessment of process evaluations on the basis of our proposed conceptual framework, shortly explained in the introduction and published elsewhere . These criteria are described in detail in Table 1. The final criteria list included nine items. Items were scored as positive (+), unsatisfactory (+/-) or negative (-). If an item was not applicable, this was stated. The quality of studies was considered to be ‘above average’ if the overall validity score (T) was above 50%, and a score above 75% was considered to represent relatively good quality. This validity score (T) was based on the nine internal validity criteria (T1-T9). Non-applicable items (N/A) were not included in the percentages.
The first author (DW) extracted the data using a predefined format. For the effect evaluations, information was extracted about study design, company type, study population, intervention content, and intervention goal. In addition, although not the main aim of this review, the proportion of affected primary outcomes was extracted to provide some information about the effectiveness of the program. For the process evaluations information was extracted about data collection, methods, timing of measurements, level of evaluation, type of evaluation, linking effect to implementation outcomes, model for process evaluation, reported process components and reported barriers and/or facilitators. During data extraction, the process components reported in the studies were classified under the eight process components (recruitment, reach, dose delivered, dose received, fidelity, satisfaction and maintenance, and context) on the basis of the definitions proposed by Wierenga et al. [11, 14]. If a study used another framework/model with additional components, these were classified separately. The barriers and/or facilitators found in the studies were assigned to five categories based on the review by Fleuren et al. (2004): socio-political context, organization, implementer, program and participants [12, 15]. When something was unclear, advise was asked from the second (LE) and third author (PE). The complete data extraction form can be obtained from the corresponding author.
The initial computerized search identified 9112 articles (see flow chart in Figure 2). Initial screening of the titles and abstracts produced 704 potentially relevant articles looking at all lifestyle behaviors. 8408 articles were excluded because they were not (R) CT related to a WHPP with the aim of changing employee lifestyles. We excluded 397 of the 704 potentially relevant articles on the grounds that articles were found in multiple databases (Pubmed, EMBASE, PsycINFO, Cochrane Central Register of Controlled Trials), or in the different searches based on lifestyle behavior. Interventions not related to any of the five health behaviors or not targeting employees were also excluded.
A total of 307 full-text articles were retrieved and checked for related process evaluation articles, resulting in the exclusion of another 277 articles. The main reasons for exclusion were: no additional published process evaluation for the same study, not targeting employees, or no lifestyle component. Thirty studies reporting on both the effects and the implementation process of the same WHPP were checked for eligibility. After reading the papers, eight studies were excluded because, contrary to expectations, they did not perform a process evaluation. No articles were added after snowballing the references in the selected studies. Finally, twenty-two studies met the selection criteria and were included in this review (Additional file 2).
Intervention and study characteristics
The general study characteristics are presented in Additional file 2; second column, in order to give an overview of the type of studies included. Half of the studies were conducted in the USA [29–39], and the other half in Europe [40–50], mostly in the Netherlands [40, 41, 45, 47–49]. Nineteen (86%) studies used a randomized controlled trial design [29, 30, 33–36, 38–50] and three (14%) a controlled trial design [31, 32, 37]. Fourteen (63%) studies reported on interventions targeting physical activity (PA) [29–34, 40–44, 48–50]. Two of these studies focused mainly on preventing musculoskeletal disorders [40, 44], but the intervention included a PA component and so these studies were included in the review. Twelve studies (55%) reported on the effect and process evaluation of interventions targeting healthy nutrition [29, 32–37, 41, 45, 47–49]. Furthermore, five (23%) studies reported interventions for smoking and tobacco use [36–39, 41], and two (9%) studies focused on relaxation [46, 48]. None of the twenty-two studies targeted alcohol use.
Methodological quality assessment
Table 2 shows the methodological quality scores of the included studies. Initial disagreement between the reviewers about 50 of the 448 effect evaluation criteria (11%) and 46 of the 207 process evaluation criteria (22%) were mainly attributable to the interpretation of the items. The differences concentrated primarily on drop-out (V4), company characteristics (D3), intention to treat (A3), data collection for process evaluation (T5), and the timing of data collection for process evaluation (T6). Full agreement was reached after two discussion sessions, thereby completing the scoring process.
The methodological quality scores for the effect evaluations of the studies ranged from 37.5% to 100%. Fifteen (68%) effect evaluations were considered to be ‘above average’ (>50%) [29–33, 37, 39–41, 43, 44, 46, 48–50]. Only six of these effect evaluations (27%) were relatively good (quality score >75%) [31, 40, 41, 43, 48, 49]. The quality scores for the process evaluations ranged from 12.5% to 100%. Eight process evaluations (36%) were relatively good (quality score >75%) [51–58].
Only three (14%) studies scored relatively good (quality score >75%) on both the process as well as the effect evaluation with respect to their methodological quality [51, 57, 58]. In four (18.2%) studies the methodological quality of both the effect as well as the process evaluation was poor (quality score 50% or less) [36, 38, 42, 45]. No relation was observed between the quality of effect evaluations and process evaluations.
Process evaluation design
The characteristics of the included process evaluations are presented in Additional file 2; third and fourth column. Most studies (50%) used a mixed methods approach (qualitative and quantitative) to look at the implementation process [51–56, 58–62]. With regard to the number of measurements, ten (45%) studies conducted a post-process evaluation only [29, 55, 59, 61–67]. Three (14%) studies collected process evaluation information at three points in time: before, during and after the intervention [54, 56, 60].
Only two (9%) studies evaluated the implementation process at all three actor levels (the macro-, meso- and micro-levels, see above) [53, 60]. The majority (50%) of the studies evaluated the implementation process at both the meso- and micro-levels [51, 52, 54–59, 61, 67, 68]. A minority of the studies (n = 8; 36%) reported using a theoretical framework to guide their process evaluation [47, 51–53, 57, 58, 60, 69]. Of these eight studies, four used the framework of Steckler and Linnan (or an adapted version of that framework) [51, 57, 58, 69], and the remaining four studies applied either the integrative model , the RE-AIM model , a model based on Durlak and Dupre , or a model based on Baranowski and Stables, in combination with Rogers’ framework for the diffusion of innovations . It should be noted that the integrative model is originally a model for the development of workplace environmental interventions and not specifically for process evaluations .
Reporting of process evaluation components
The predetermined process components that were measured are presented in Additional file 2; fourth column. Seven of these components (reach, recruitment, dose delivered, dose received, fidelity, satisfaction and maintenance) measure the degree of implementation. The eighth component, context, maps the barriers and/or facilitators that affect implementation. The average number of process evaluation components found in the studies was 3.9, ranging from 1 to 8. Ten (45%) studies evaluated fewer than four process components [29, 53, 62–64, 66–68, 70, 71]. Six (27.5%) studies reported five or more process components [51, 56–58, 60, 69]. The studies focused mainly on dose received (82%) [29, 51, 52, 54–63, 65, 67–69, 71], dose delivered (68%) [29, 51, 52, 54, 55, 57–62, 65, 67],, and context (68%) [52, 54–57, 60, 61, 64, 66, 69–74]. Fewer than half of the studies (41%) looked at fidelity [51, 52, 56–60, 65]. Nine (41%) studies measured satisfaction [51, 54, 57–59, 63–66]. Relatively few studies reported on reach (n = 7; 32%) [51, 55, 57, 58, 60, 61], or recruitment (n = 5; 23%) [56, 58, 60, 64, 69]. In addition, two (9%) studies looked at maintenance [56, 60].
Implementation barriers and/or facilitators
Next, we identified barriers and/or facilitators which can affect implementation. However, the barriers and/or facilitators were often not systematically measured by means of questionnaires or interviews and often only observed and documented on the basis of the researchers’ experience. Nevertheless, fifty-four different barriers and/or facilitators were obtained from nineteen (86%) studies (Table 3) [52, 54–57, 59–64, 66, 68–74]. On average, the studies described 6.5 [range 1-22] barriers and/or facilitators. None of the studies reported a context analysis prior to implementation. The majority of barriers and facilitators in the five categories as a whole were reported only in a single study.
Fewest barriers and/or facilitators were reported in the category ‘characteristics of the socio-political context’. Only two (9%) studies reported a barrier or facilitator in this category (these studies are listed in Table 3). The largest number of barriers and/or facilitators (n = 17) were found in the category ‘characteristics of the organization’. Management support was found most frequently, having been mentioned in eight studies (36%). Strong management support was described as a facilitator, whereas unbalanced or lack of management support was found to be a barrier. The definition of management support varied widely. Another frequently reported barrier was the lack of financial, staffing or material resources (n = 5 studies; 26%). In the category ‘characteristics of the intervention’ fifteen barriers and/or facilitators were identified, with ‘compatibility of intervention with the organization’ being the most commonly reported facilitator in eight studies (42%). The facilitator ‘relative advantage of the intervention’ was reported in five (26%) studies. In the category ‘characteristics of the implementer’ twelve barriers and/or facilitators were identified; the factor ‘available time of the implementer’ was most commonly reported (n = 5; 26%). In the category ‘characteristics of the user’, eight barriers and/or facilitators were identified with ‘time constraints’ was frequently reported as a barrier to participation. Furthermore, we found that high work demands and a high workload were also a barrier to participation.
Degree of implementation and program effectiveness
Only seven (31.8%) studies evaluated the association between implementation and program outcomes [52, 58–62, 67]. These studies generally found that the level of implementation in terms of high fidelity and dose was positively associated with a positive change in their primary outcome measures (body weight, waist circumference, Body Mass Index, fruit and vegetable intake, physical activity levels, smoking cessation). This was analyzed by means of linear or logistic regression analysis [52, 58–62, 67], latent growth modeling , dose or as-treated analysis [52, 58–62, 67], analysis of variance [52, 58–62, 67], mixed model logistic regression analysis or linear mixed model regression analysis [52, 58–62, 67], chi square tests [52, 58–62, 67], and multilevel linear regression analysis [52, 58–62, 67]. Of the seven studies, three studies found that higher participation levels (dose received) significantly and positively impacted their primary outcome measures (body weight, waist circumference, physical activity levels and smoking cessation) [52, 58, 62].
In the study on personal and team goal-setting with the aim of increasing leisure-time physical activity, Dishman and Wilson found that high implementation groups had a greater increase in vigorous physical activity over the three time points than did the low implementation sites. For dose received no significant change over time was observed for any of the outcome measures (walking, moderate physical activity, vigorous physical activity) . The results of the study of Volpp and Kim on improving smoking cessation rates by offering financial incentives for smoking cessation, the results showed that the attendance levels to the cessation programs were higher among quitters than among non-quitters in the intervention group. Additionally, quitters in the intervention group attended more than two times as many sessions as non-quitters in the intervention group [52, 58–62, 67]. Finally, in the study of Verweij on preventing weight gain by implementing an occupational health guideline, it was found that employees with higher attendance and satisfaction levels significantly reduced their waist circumference and body weight compared to employees with lower attendance and satisfaction levels [52, 58–62, 67].
The aims of this review were to (1) further our understanding of the quality of process evaluations alongside effect evaluations for WHPPs, (2) identify barriers/facilitators affecting implementation, and (3) explore the relationship between effectiveness and the implementation process.
Prior to discussing the main findings, it should be noted that only a small number of studies that evaluated the effectiveness of a WHPP included a process evaluation relating to the implementation of that WHPP. Of the 307 effect evaluations identified in this review, only twenty-two (7.2%) published an additional process evaluation. With respect to the first aim we can conclude that the quality of process evaluations alongside effect evaluations of WHPPs was generally poor to average and a systematic approach was lacking. This makes it difficult to draw firm conclusion about reliably identifying implementation barriers and/or facilitators (second aim) or about the relation between effectiveness and implementation (third aim). Murta et al. (2007) found the same difficulties in a systematic review. They concluded that process evaluations alongside workplace stress-management interventions were mainly poor to average, also making it difficult to identify reliable determinants of effective intervention implementation .
The process evaluations covered by our review lacked a theoretical basis, and the most frequently measured process components were dose delivered and dose received. This reflects the primary interest of researchers in actual intervention delivery and in participation levels (in other words, quantitative outcomes), rather than in how an intervention is delivered, the quality of delivery (fidelity), maintenance, or the reasons respondents participated or not. Possible explanations may be that researchers are more trained in quantitative methods of research and less in qualitative methods and analysis. Furthermore, good qualitative research takes a lot of time and energy and is therefore more expensive than quantitative research .
The included process evaluations tended to operationalise the measured process components in different ways, even when using the same framework for evaluation. For instance, four studies used Steckler and Linnan’s framework as a guideline for their process evaluation [51, 57, 58, 69]. However, in some of these studies, actual attendance levels were placed under reach [51, 57, 58], whereas another study included attendance levels as part of the dose received . The definitions for ‘reach’ and ‘dose received’ as defined by Steckler and Linnan are: ‘proportion of intended target audience that participates in an intervention’ and ‘the extent to which participants actively engage with the intervention’ respectively . The different approaches to operationalisation in studies of these components could be explained by these somewhat ambiguous descriptions in Steckler and Linnan’s framework. What is the difference between ‘participating’ and ‘actively engaging with’ an intervention? We can assume that ‘participating in an intervention’ involves a focus on participation and non-participation regardless of the frequency, duration and intensity of participation and that these aspects may actually be taken into account when measuring active engagement with the intervention. Since several studies had different approaches to the operationalisation of process components, it was a challenge to translate the definitions used into general terms. We therefore adopted a working definition for each process component based on a previously published framework in order to interpret our findings . For example, we defined reach as ‘the proportion of the target audience that is aware of the intervention’ and dose received as ‘the proportion of participants that actually participates in the intervention’, including frequency, duration and intensity . We therefore distinguished between awareness and actual participation, the latter including level of participation. However, due to the operationalisation ambiguity in Steckler and Linnan’s framework, others might define these components, and therefore interpret the findings, differently. This limitation was also mentioned in a comparable review by Durlak and Dupre (2008), revealing that researchers still do not take the time and effort necessary to develop consensus about the best way to perform process evaluations in different settings .
Our review of the studies provided evidence of various barriers and/or facilitators that could influence the implementation process. However, partly as a result of the relatively low quality of the process evaluations, no systematic examination of barriers and/or facilitators affecting implementation was possible. Barriers and/or facilitators were often not systemically measured and only observed and documented on the basis of the researchers’ experience. This raises the question of how reliable the results of this review are with respect to the identified barriers and/or facilitators (Table 3). Since the barriers and/or facilitators identified in this review clearly overlap with other reviews focusing on these factors, we can conclude that our results should be reliable [10, 12, 13, 22, 24, 25]. For example, Sangster-Gormley also identified ‘active management participation’ as a facilitator for implementation .
Another notable finding was that researchers evaluated barriers and/or facilitators mostly after implementation and only a few studied them during implementation. However, interventions are more likely to be successful if potential barriers and/or facilitators are assessed beforehand so they can be anticipated, facilitating implementation. Although most studies failed to observe barriers and/or facilitators systematically, some barriers and/or facilitators were more frequently reported in several studies. This suggests that these are important factors which researchers, practitioners and implementers need to take into account during a WHPP. In the category ‘characteristics of the socio-political context’, only one barrier (‘competitive business environment’) and one facilitator (‘compatibility of program with societal developments’) were reported in all twenty-two studies. This could suggest that this category is virtually disregarded, despite its importance for a thorough understanding of these socio-political factors prior to implementation, since the intended user of the intervention is part of an organization, which in turn is part of a wider environment . However, it is also possible that intervention developers and researchers already anticipate on socio-political issues and try to take these into account throughout the development and implementation process.
Most barriers and/or facilitators were reported within the category ‘characteristics of the organization’. This is not surprising since most implementation research focusses at this level as this is essential for continuation. Moreover, the studies in this review are selected because they focus on organizations. Frequently mentioned implementation barriers were: ‘lack of resources’, ‘no fit of the intervention with organizational culture’ and ‘unbalanced or lack of manager support’. For example, when the intervention program requires a fitness center but there is no center nearby, this intervention is not the best fit with the company and other options, such as sports activities (possibly outdoors) in the immediate vicinity of the company, should be explored. Frequently mentioned organizational facilitators for WHPP implementation were: ‘organizations awareness of perceived benefits and relevance’; ‘good collaboration with all persons involved’; ‘strong and formal management support’; and ‘active management participation and engagement’. The researcher or implementer could work on these factors with the aim of optimizing implementation, for example by organizing manager meetings before the start of the intervention so that everyone is informed. However, another frequently mentioned facilitator for implementation was: ‘an organizational culture that emphasizes goal setting and supports social interaction’. Nevertheless, it is unclear what researchers and implementers can do when the organization culture does not fulfill this criterion before the start of the study, especially since organizational culture is a complex phenomenon which is not easily changed . We suggest that the best way to deal with this ‘problem’ is to adapt the intervention where possible to ensure an optimal fit with the current organizational culture. In addition we would advise researchers and implementers to take the time to assess the organizational culture and include organizational determinants in their process evaluation .
Turning to the category ‘characteristics of the implementer’, the results of this review indicate that the most important facilitators for implementers are: ‘sufficient time’, ‘skills’, ‘knowledge’ and ‘competence’. When implementers experienced ‘planning difficulties’ or a ‘heavy workload’, this represented a barrier to implementation. However, these barriers can be overcome by making sure that the implementer receives enough support from stakeholders within the organization and enough administrative support. In the category ‘characteristics of the intervention program’, fifteen barriers and/or facilitators were reported. The three most frequently reported facilitators in this category were: ‘relative advantage’, ‘compatibility’ and ‘complexity’. These three factors are part of Rogers’ Diffusion of Innovations theory, and they are therefore known facilitating factors which could explain why researchers focused on these factors . However, we also identified barriers relating to the intervention characteristics, including: ‘conflicting interests between worksite and interventions’ and ‘technical problems’ which are not part of a known theory. Fewest barriers and/or facilitators were reported in the category ‘characteristics of the participant’. The most commonly reported barrier to participation was ‘time constraints of participants’, suggesting that employers should give employees time off to participate in the interventions. This facilitator is also a reflection of the facilitator ‘management support and engagement’ which was mentioned earlier. Two frequently reported perceived barriers for participants were ‘work demands’ and ‘current workload’. However, as for many of the other found barriers, it is difficult to address these two factors. A possible suggestion to overcome these barriers could be that at onset of the study, employers should stress out that employees are allowed to participate under working hours and that if they experience difficulties in participating due to work demands and workload they should discuss this with their manager.
The reported barriers and/or facilitators in this review are comparable with the results of other reviews mentioned in this paper [10, 12, 24, 25]. This suggests that the findings of this review are generalisable to other settings. It would be beneficial to explore all the barriers and/or facilitators listed here before implementation so that researchers and implementers can anticipate possible barriers, incorporate possible facilitators and perhaps adjust their program accordingly. A new Dutch instrument was proposed recently that makes it possible to map barriers and/or facilitators beforehand, and therefore possibly help to facilitate the actual implementation of WHPPs. It also serves as a monitoring instrument enabling program adjustments before and during implementation (TNO R10625; Fleuren et al. 2012).
Unfortunately, it was not possible to rank the 54 reported barriers and/or facilitators in order of importance because some determinants were only identified once. This could suggest that the factor was specific to the type of intervention. In addition, the relation between the 54 reported barriers and/or facilitators is unclear. This limitation was also experienced by Fleuren et al. (2004), showing that more research is needed into barriers and/or facilitators affecting the implementation process .
Despite the low quality of most included process evaluations, the findings of this review do suggest that higher levels of implementation are associated with better program outcomes. However, the few studies that investigated this association showed a narrow focus on implementation, mainly addressing the link between dose received and/or fidelity in relation to program outcomes. Whereas it is so important to use all aspects of process evaluations for the interpretation of the (lack of) effects of a program [18, 19].
A limitation, which is inherent to writing a systematic review, is publication bias or studies being overlooked. In other words, our overview of the literature may not be complete. We tried to avoid this pitfall by selecting four different databases, both medical and psychological, using broad search terms and checking each lifestyle (i.e. physical activity, nutrition, smoking, alcohol use and relaxation) separately and by checking the references in the studies we included. Clearly, our decision to restrict our search to process evaluations in combination with effect evaluations in a (randomized) controlled trial design for WHPP has caused a number of process evaluations to fall out of scope for this review. Although we do acknowledge this, our aim was to gain insight into the relationship between implementation quality and effectiveness. Importantly, this is also a tendency that has been suggested by implementation journals, such as Implementation Science.
A best-evidence synthesis falls outside the scope of this review. Moreover, it could not be performed since we included only the studies which could be paired with a process evaluation. This means that not all known and relevant controlled trials in the field of WHPPs focusing on healthy lifestyle are included in this review.
Given the fact that relatively few process evaluations were found by comparison with the high number of RCTs, and taking into account that not all process evaluations are published, it is safe to conclude that process evaluations are still not as high on the research agenda as effect evaluations. The importance of conducting an effect evaluation as well as a process evaluation is increasingly being advocated [22, 79]. It does appear that lately more process evaluations are being conducted, but this review shows that they are in general of low to average quality due to the lack of a systematic approach.
Durlak and Dupre (2008) noted that “implementation matters” and “science cannot study what it cannot measure accurately and cannot measure what is does not define” . Our findings support this observation, and moreover, suggest that we should be asking: what are we measuring, when we measure implementation at all? Without a standardized approach to the operationalisation of process evaluation components, it is difficult to faithfully replicate studies, identify implementation barriers and/or facilitators, or assess the methodological quality of process evaluations. This indicates that a general framework for process evaluations is required that researchers, implementers, reviewers and practitioners can use to evaluate and assess the quality of the implementation of a WHPP [80, 81]. In order to create a general framework, it is essential that all relevant stakeholders subscribe to a consensus about the terminology and operationalisation of relevant process components by developing a taxonomy for process evaluations. This taxonomy could constitute a fundamental first step towards the standardization of process evaluations in multiple fields, and lead to program reproducibility . Furthermore, it will allow researchers to compare and assess the quality of process evaluations, systematically identify implementation barriers and/or facilitators and possibly link implementation to program outcomes.
In short, this review demonstrates the need for a systematic approach to process evaluation as a way of improving WHPP implementation. In order to set the first steps into this direction, we suggest that future process evaluations need to apply the framework Wierenga et al. 2012 we proposed in the introduction of this review and should also use this framework to assess the methodological quality of the studies. This framework allows mapping the complete implementation process and takes into account determinants of implementation and provides the necessary explicit operationalization of each component .
Worksite health promotion program.
Kirsten W: Making the link between health and productivity at the workplace–a global perspective. Ind Health. 2010, 48 (3): 251-255. 10.2486/indhealth.48.251.
Jans MP, van den Heuvel SG, Hildebrandt VH, Bongers PM: Overweight and Obesity as Predictors of Absenteeism in the Working Population of the Netherlands. J Occup Environ Med. 2007, 49 (9): 975-980. 10.1097/JOM.0b013e31814b2eb7.
Robroek SJ, van den Berg TI, Plat JF, Burdorf A: The role of obesity and lifestyle behaviours in a productive workforce. Occup Environ Med. 2011, 68 (2): 134-139. 10.1136/oem.2010.055962.
Alavinia SM, Molenaar D, Burdorf A: Productivity loss in the workforce: Associations with health, work demands, and individual characteristics. Am J Ind Med. 2009, 52 (1): 49-56. 10.1002/ajim.20648.
Engbers LH, Van Poppel MN, Paw MJ CA, Van Mechelen W: Worksite health promotion programs with environmental changes: a systematic review. Am J Prev Med. 2005, 29 (1): 61-70. 10.1016/j.amepre.2005.03.001.
Ni Mhurchu C, Aston LM, Jebb SA: Effects of worksite health promotion interventions on employee diets: a systematic review. BMC Public Health. 2010, 10 (10): 62-
Jepson RG, Harris FM, Platt S, Tannahill C: The effectiveness of interventions to change six health behaviors: a review of reviews. BMC Public Health. 2010, 8 (10): 538-
Proper KI, Koning M, van der Beek AJ, Hildebrandt VH, Bosscher RJ, van Mechelen W: The effectiveness of worksite physical activity programs on physical activity, physical fitness, and health. Clin J Sport Med. 2003, 13 (2): 106-117. 10.1097/00042752-200303000-00008.
Kristensen TS: Intervention studies in occupational epidemiology. Occup Environ Med. 2005, 62: 205-210. 10.1136/oem.2004.016097.
Robroek SJ, van Lenthe FJ, van Empelen P, Burdorf A: Determinants of participation in worksite health promotion programmes: a systematic review. Int J Behav Nutr Phys Act. 2009, 6: 26-10.1186/1479-5868-6-26.
Wierenga D, Engbers LH, van Empelen P, Hildebrandt VH, van Mechelen W: The design of a real-time formative evaluation of the implementation process of lifestyle interventions at two worksites using a 7-step strategy (BRAVO@Work). BMC Public Health. 2012, 12 (1): 619-10.1186/1471-2458-12-619.
Fleuren M, Wiefferink K, Paulussen T: Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care. 2004, 16 (2): 107-123. 10.1093/intqhc/mzh030.
Durlak JA, DuPre EP: Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008, 41: 327-350. 10.1007/s10464-008-9165-0.
Steckler A, Linnan L: Proces Evaluation for Public Health Interventions and Research. 2002, San Francisco: Jossey-Bass
Paulussen TH, Wiefferink K, Mesters I: Invoering van effectief gebleken interventies. Gezondheidsvoorlichting en gedragsverandering. Edited by: Brug J, Van Asseman P, Lechner L. 2007, Assen: Van Gorcum
Craig P, Dieppe P, Macintyre S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008, 337 (1655): 979-983.
Saunders RP, Evans MH, Joshi P: Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promotion Pract. 2005, 6 (2): 134-147. 10.1177/1524839904273387.
Oakley A, Strange V, Bonell C, Allen E, Stephenson J: RIPPLE Study Team: Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006, 332 (7538): 413-416. 10.1136/bmj.332.7538.413.
Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks PA: The future of health behavior change research: what is needed to improve translation of research into health promotion practice?. Ann Behav Med. 2004, 27 (1): 3-12. 10.1207/s15324796abm2701_2.
Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp N, Smith JL: The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006, 21 (2): 1-8.
Mittman BS: Creating the evidence base for quality improvement collaboratives. Ann Int Med. 2004, 140 (11): 897-901. 10.7326/0003-4819-140-11-200406010-00011.
Murta SG, Sanderson K, Oldenburg B: Process evaluation in occupational stress management programs: a systematic review. Am J Health Promot. 2007, 21: 248-254. 10.4278/0890-1171-21.4.248. 0890-1171; 0890-1171; 4
Glasgow RE, Vogt TM, Boles SM: Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999, 89: 9-1322.
Koppelaar E, Knibbe JJ, Miedema HS, Burdorf A: Determinants of implementation of primary preventive interventions on patient handling in healthcare: a systematic review. Occup Environ Med. 2009, 66 (6): 353-360. 10.1136/oem.2008.042481.
Sangster-Gormley E, Martin-Misener R, Downe-Wamboldt B, Dicenso A: Factors affecting nurse practitioner role implementation in Canadian practice settings: an integrative review. J Adv Nurs. 2011, 67 (6): 1178-1190. 10.1111/j.1365-2648.2010.05571.x.
Furlan AD, Pennick V, Bombardier C, Van Tulder M, Editorial B: Cochrane Back Review Group: 2009 updated method guidelines for systematic reviews in the Cochrane Back Review Group. Spine (Phila Pa 1976). 2009, 34 (18): 1929-1941. 10.1097/BRS.0b013e3181b1c99f.
Proper KI, Staal BJ, Hildebrandt VH, van der Beek AJ, van Mechelen W: Effectiveness of physical activity programs at worksites with respect to work-related outcomes. Scand J Work Environ Health. 2002, 28 (2): 75-84. 10.5271/sjweh.651.
Van Tulder M, Furlan A, Bombardier C, Bouter L: Editorial Board of the Cochrane Collaboration Back Review Group: Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976). 2003, 28 (12): 1290-1299.
French SA, Harnack LJ, Hannan PJ, Mitchell NR, Gerlach AF, Toomey TL: Worksite environment intervention to prevent obesity among metropolitan transit workers. Prev Med. 2010, 50 (4): 180-185. 10.1016/j.ypmed.2010.01.002.
Dishman RK, DeJoy DM, Wilson MG, Vandenberg RJ: Move to Improve: a randomized workplace trial to increase physical activity. Am J Prev Med. 2009, 36 (2): 133-141. 10.1016/j.amepre.2008.09.038.
Yap TL, Davis LS, Gates DM, Hemmings AB, Pan W: The effect of tailored E-mails in the workplace. Part I. Stage movement toward increased physical activity levels. AAOHN J. 2009, 57 (7): 267-273. 10.3928/08910162-20090617-02.
Goetzel RZ, Baker KM, Short ME, Pei X, Ozminkowski RJ, Wang S, Bowen JD, Roemer EC, Craun BA, Tully KJ, Baase CM, DeJoy DM, Wilson MG: First-year results of an obesity prevention program at The Dow Chemical Company. J Occup Environ Med. 2009, 51 (2): 125-138. 10.1097/JOM.0b013e3181954b03.
Lemon SC, Zapka J, Li W, Estabrook B, Rosal M, Magner R, Andersen V, Borg A, Hale J: Step ahead a worksite obesity prevention trial among hospital employees. Am J Prev Med. 2010, 38 (1): 27-38. 10.1016/j.amepre.2009.08.028.
Sorensen G, Barbeau E, Stoddard AM, Hunt MK, Kaphingst K, Wallace L: Promoting behavior change among working-class, multiethnic workers: results of the healthy directions–small business study. Am J Public Health. 2005, 95 (8): 1389-1395. 10.2105/AJPH.2004.038745.
Beresford SA, Thompson B, Feng Z, Christianson A, McLerran D, Patrick DL: Seattle 5 a Day worksite program to increase fruit and vegetable consumption. Prev Med. 2001, 32 (3): 230-238. 10.1006/pmed.2000.0806.
Sorensen G, Barbeau EM, Stoddard AM, Hunt MK, Goldman R, Smith A, Brennan AA, Wallace L: Tools for health: the efficacy of a tailored intervention targeted for construction laborers. Cancer Causes Control. 2007, 18 (1): 51-59. 10.1007/s10552-006-0076-9.
Sorensen G, Stoddard A, Quintiliani L, Ebbeling C, Nagler E, Yang M, Pereira L, Wallace L: Tobacco use cessation and weight management among motor freight workers: results of the gear up for health study. Cancer Causes Control. 2010, 21 (12): 2113-2122. 10.1007/s10552-010-9630-6.
Stoddard AM, Fagan P, Sorensen G, Hunt MK, Frazier L, Girod K: Reducing cigarette smoking among working adolescents: results from the SMART study. Cancer Causes Control. 2005, 16 (10): 1159-1164. 10.1007/s10552-005-0353-z.
Volpp KG, Troxel AB, Pauly MV, Glick HA, Puig A, Asch DA, Galvin R, Zhu J, Wan F, DeGuzman J, Corbett E, Weiner J, Audrain-McGovern J: A randomized, controlled trial of financial incentives for smoking cessation. N Engl J Med. 2009, 360 (7): 699-709. 10.1056/NEJMsa0806819.
Driessen MT, Proper KI, Anema JR, Knol DL, Bongers PM, van der Beek AJ: Participatory ergonomics to reduce exposure to psychosocial and physical risk factors for low back pain and neck pain: results of a cluster randomised controlled trial. Occup Environ Med. 2011, 68 (9): 674-681. 10.1136/oem.2010.056739.
Groeneveld IF, Proper KI, van der Beek AJ, van Mechelen W: Sustained body weight reduction by an individual-based lifestyle intervention for workers in the construction industry at risk for cardiovascular disease: results of a randomized controlled trial. Prev Med. 2010, 51 (3–4): 240-246.
Gilson N, McKenna J, Cooke C, Brown W: Walking towards health in a university community: a feasibility study. Prev Med. 2007, 44 (2): 167-169. 10.1016/j.ypmed.2006.09.012.
Andersen LL, Saervoll CA, Mortensen OS, Poulsen OM, Hannerz H, Zebis MK: Effectiveness of small daily amounts of progressive resistance training for frequent neck/shoulder pain: randomised controlled trial. Pain. 2011, 152 (2): 440-446. 10.1016/j.pain.2010.11.016.
Haukka E, Pehkonen I, Leino-Arjas P, Viikari-Juntura E, Takala EP, Malmivaara A, Hopsu L, Mutanen P, Ketola R, Virtanen T, Holtari-Leino M, Nykanen J, Stenholm S, Ojajarvi A, Riihimaki H: Effect of a participatory ergonomics intervention on psychosocial factors at work in a randomised controlled trial. Occup Environ Med. 2010, 67 (3): 170-177. 10.1136/oem.2008.043786.
Steenhuis I, Van Assema P, Van Breukelen G, Glanz K, Kok G, De Vries H: The impact of educational and environmental interventions in Dutch worksite cafeterias. Health Promot Int. 2004, 19 (3): 335-343. 10.1093/heapro/dah307.
Hasson D, Anderberg UM, Theorell T, Arnetz BB: Psychophysiological effects of a web-based stress management system: a prospective, randomized controlled intervention study of IT and media workers [ISRCTN54254861. BMC Public Health. 2005, 5: 78-10.1186/1471-2458-5-78.
Vermeer WM, Steenhuis IH, Leeuwis FH, Heymans MW, Seidell JC: Small portion sizes in worksite cafeterias: do they help consumers to reduce their food intake?. Int J Obes (Lond). 2011, 35 (9): 1200-1207. 10.1038/ijo.2010.271.
Strijk JE, Proper KI, van der Beek AJ, van Mechelen W: A worksite vitality intervention to improve older workers’ lifestyle and vitality-related outcomes: results of a randomised controlled trial. J Epidemiol Community Health. 2012, 66 (11): 1071-1078. 10.1136/jech-2011-200626.
Verweij LM, Proper KI, Weel AN, Hulshof CT, van Mechelen W: The application of an occupational health guideline reduces sedentary behaviour and increases fruit intake at work: results from an RCT. Occup Environ Med. 2012, 69 (7): 500-507. 10.1136/oemed-2011-100377.
Jorgensen MB, Faber A, Hansen JV, Holtermann A, Sogaard K: Effects on musculoskeletal pain, work ability and sickness absence in a 1-year randomised controlled trial among cleaners. BMC Public Health. 2011, 11: 840-10.1186/1471-2458-11-840.
Driessen MT, Proper KI, Anema JR, Bongers PM, van der Beek AJ: Process evaluation of a participatory ergonomics programme to prevent low back pain and neck pain among workers. Implement Sci. 2010, 5: 65-10.1186/1748-5908-5-65.
Wilson MG, Basta TB, Bynum BH, DeJoy DM, Vandenberg RJ, Dishman RK: Do intervention fidelity and dose influence outcomes? Results from the move to improve worksite physical activity program. Health Educ Res. 2010, 25 (2): 294-305. 10.1093/her/cyn065.
Wilson MG, Goetzel RZ, Ozminkowski RJ, DeJoy DM, Della L, Roemer EC, Schneider J, Tully KJ, White JM, Baase CM: Using formative research to develop environmental and ecological interventions to address overweight and obesity. Obesity (Silver Spring). 2007, 15 (Suppl 1): 37S-47S.
Pehkonen I, Takala EP, Ketola R, Viikari-Juntura E, Leino-Arjas P, Hopsu L, Virtanen T, Haukka E, Holtari-Leino M, Nykyri E, Riihimaki H: Evaluation of a participatory ergonomic intervention process in kitchen work. Appl Ergon. 2009, 40 (1): 115-123. 10.1016/j.apergo.2008.01.006.
Hunt MK, Barbeau EM, Lederman R, Stoddard AM, Chetkovich C, Goldman R, Wallace L, Sorensen G: Process evaluation results from the Healthy Directions-Small Business study. Health Educ Behav. 2007, 34 (1): 90-107.
Vermeer WM, Leeuwis FH, Koprulu S, Zouitni O, Seidell JC, Steenhuis IH: The process evaluation of two interventions aimed at portion size in worksite cafeterias. J Hum Nutr Diet. 2012, 25 (2): 180-188. 10.1111/j.1365-277X.2011.01219.x.
Strijk JE, Proper KI, van der Beek AJ, van Mechelen W: A process evaluation of a worksite vitality intervention among ageing hospital workers. Int J Behav Nutr Phys Act. 2011, 8: 58-10.1186/1479-5868-8-58.
Verweij LM, Proper KI, Hulshof CT, van Mechelen W: Process evaluation of an occupational health guideline aimed at preventing weight gain among employees. J Occup Environ Med. 2011, 53 (7): 722-729. 10.1097/JOM.0b013e318222af9b.
Groeneveld IF, Proper KI, Absalah S, van der Beek AJ, van Mechelen W: An individually based lifestyle intervention for workers at risk for cardiovascular disease: a process evaluation. Am J Health Promot. 2011, 25 (6): 396-401. 10.4278/ajhp.091001-QUAN-319.
Estabrook B, Zapka J, Lemon SC: Evaluating the implementation of a hospital work-site obesity prevention intervention: applying the RE-AIM framework. Health Promot Pract. 2012, 13 (2): 190-197. 10.1177/1524839910385897.
Beresford SA, Shannon J, McLerran D, Thompson B: Seattle 5-a-Day Work-Site Project: process evaluation. Health Educ Behav. 2000, 27 (2): 213-222. 10.1177/109019810002700207.
Kim A, Kamyab K, Zhu J, Volpp K: Why are financial incentives not effective at influencing some smokers to quit? Results of a process evaluation of a worksite trial assessing the efficacy of financial incentives for smoking cessation. J Occup Environ Med. 2011, 53 (1): 62-67. 10.1097/JOM.0b013e31820061d7.
Yap TL, Busch James DM: Tailored e-mails in the workplace. AAOHN J. 2010, 58 (10): 425-432. 10.3928/08910162-20100928-05.
Gilson N, McKenna J, Cooke C: Experiences of route and task-based walking in a university community: qualitative perspectives in a randomized control trial. J Phys Act Health. 2008, 5 (Suppl 1): S176-S182.
Hunt MK, Harley AE, Stoddard AM, Lederman RI, MacArthur MJ, Sorensen G: Elements of external validity of tools for health: an intervention for construction laborers. Am J Health Promot. 2010, 24 (5): e11-e20. 10.4278/ajhp.080721-QUAN-130.
Steenhuis I, van Assema P, Reubsaet A, Kok G: Process evaluation of two environmental nutrition programmes and an educational nutrition programme conducted at supermarkets and worksite cafeterias in the Netherlands. J Hum Nutr Diet. 2004, 17 (2): 107-115. 10.1111/j.1365-277X.2004.00507.x.
Quintiliani L, Yang M, Sorensen G: A process evaluation of tobacco-related outcomes from a telephone and print-delivered intervention for motor freight workers. Addict Behav. 2010, 35 (11): 1036-1039. 10.1016/j.addbeh.2010.06.003.
Hunt MK, Fagan P, Lederman R, Stoddard A, Frazier L, Girod K, Sorensen G: Feasibility of implementing intervention methods in an adolescent worksite tobacco control study. Tob Control. 2003, 12 (Suppl 4): IV40-IV45.
Jorgensen MB, Faber A, Jespersen T, Hansen K, Ektor-Andersen J, Hansen JV, Holtermann A, Sogaard K: Implementation of physical coordination training and cognitive behavioural training interventions at cleaning workplaces–secondary analyses of a randomised controlled trial. Ergonomics. 2012, 55 (7): 762-772. 10.1080/00140139.2012.665946.
Andersen LL: Influence of psychosocial work environment on adherence to workplace exercise. J Occup Environ Med. 2011, 53 (2): 182-184. 10.1097/JOM.0b013e3181207a01f.
Hasson H, Brown C, Hasson D: Factors associated with high use of a workplace web-based stress management program in a randomized controlled intervention study. Health Educ Res. 2010, 25 (4): 596-607. 10.1093/her/cyq005.
Driessen MT, Groenewoud K, Proper KI, Anema JR, Bongers PM, van der Beek AJ: What are possible barriers and facilitators to implementation of a Participatory Ergonomics programme?. Implement Sci. 2010, 5: 64-10.1186/1748-5908-5-64.
DeJoy DM, Bowen HM, Baker KM, Bynum BH, Wilson MG, Goetzel RZ, Dishman RK: Management support and worksite health promotion program effectiveness. Ergon Health Aspects. 2009, 5624: 13-22.
Verweij LM, Proper KI, Leffelaar ER, Weel AN, Nauta AP, Hulshof CT, van Mechelen W: Barriers and Facilitators to Implementation of an Occupational Health Guideline Aimed at Preventing Weight Gain Among Employees in the Netherlands. J Occup Environ Med. 2012, 54 (8): 954-960. 10.1097/JOM.0b013e3182511c9f.
Pope C, Ziebland S, Mays N: Qualitative research in health care. Analysing qualitative data. BMJ. 2000, 320: 114-116. 10.1136/bmj.320.7227.114.
Cameron KS, Quinn RE, Robert E: Diagnosing and changing organizational culture: Based on the Competing Values Framework. 2006, San Fransico: Jossey-Bass
Weiner BJ, Lewis MA, Linnan LA: Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res. 2009, 24 (2): 292-305.
Rogers EM: Diffusion of innovations. 2003, New York: Free Press
Nielsen K, Randall R, Holten A, Gonzálezc ER: Conducting organizational-level occupation health interventions: What works?. Work & Stress. 2010, 24 (3): 234-259. 10.1080/02678373.2010.515393.
Abraham C, Michie S: A taxonomy of behavior change techniques used in interventions. Health Psychol. 2008, 27 (3): 379-387.
Pedersen LM, Nielsen KJ, Kines P: Realistic evaluation as a new way to design and evaluate occupational safety interventions. Saf Sci. 2012, 50 (1): 48-54. 10.1016/j.ssci.2011.06.010.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/13/1190/prepub
This study was funded by: The Netherlands Organization for Health Research and Development (ZonMw, project number 50-51405-98-019).
The author’s declare that they have no competing interests.
DW carried out the design, literature search, data extraction, quality analysis, data analysis and drafted the manuscript. LE and PvE participated in its design, data extraction, quality analysis, discussing the paper, providing methodological input, and helped to draft the manuscript. LE and SD were involved in the literature search. WvM and VH were involved in the design of the review and provided feedback of draft versions. All authors commented on the draft versions and read and approved the final manuscript.
About this article
Cite this article
Wierenga, D., Engbers, L.H., Van Empelen, P. et al. What is actually measured in process evaluations for worksite health promotion programs: a systematic review. BMC Public Health 13, 1190 (2013). https://doi.org/10.1186/1471-2458-13-1190