Using the RE-AIM framework to evaluate physical activity public health programs in México

This manuscript presents the evaluation of twelve public programs that promoted physical activity in Mexico, described the behavioral targets and outcomes of the programs, and determined the degree to which programs reported on individual and organizational factors using the RE-AIM framework. We also explored whether reporting differed by funding support for the program and found no difference in reporting between government versus privately supported programs. Perhaps the most striking finding was the lack of measurement of outcomes across programs. Although all programs had a behavioral target of promoting physical activity, only three had a measurement outcome of the program of increasing physical activity, suggesting that there was little acknowledgement of the relationship between promoting a behavior and measuring whether it was done. Four programs focused on body composition changes as outcomes, which may reflect the recent acknowledgement of the significant burden of overweight and obesity in Mexico.

Echoing the recognition of the burden of overweight and obesity, half of the programs also included behavioral targets focused on diet or nutrition, either dietary modification in general, or eating more fruits and vegetables or reducing sugar sweetened beverages in specific. One included reducing sedentary time, and none had any emphasis on sleep, despite the role of these factors in their relationship to overweight and obesity [25-27]. In addition to the body composition outcomes noted above, two programs had smoking reduction as an outcome, and one program also included outcomes focused on decreasing alcohol consumption and increasing water consumption. The lack of focus of programs on increasing water consumption may be related to the history of unsafe drinking water in Mexico [28,29]. This history endures even today, where it is socially questionable to invite people to drink water, and, instead, other beverages, often high in calories and low in nutrients, are offered [18,30].

The lack of measurement of outcomes was also echoed by the measurement of efficacy/effectiveness, with over half of the programs reporting none or one of the indicators in this domain. Half of the programs had no evaluation plan in place, and nine did not have a clear indicator or definition of how program success would be described. Perhaps the lack of efficacy/effectiveness measurement is driven by the inability of public programs to measure behavioral or health outcome changes among such large segments of the population. Although national surveys of population health in Mexico have been conducted since 1995, questions measuring physical activity, rather than merely sports participation, had not been measured until recently [31]. It may also be that measuring efficacy is simply perceived as too big a challenge by public health practitioners. In a survey of public health practitioners implementing the National Physical Activity Plan in the US, most reported that changes from the plan were difficult to implement and impact of changes was hard to observe [32]. In another investigation of implementation of policy and programs in the USA impact was inconsistently reported, only about half of the time [33].

In contrast to the lack of efficacy/effectiveness reporting, the reporting of program reach was more consistently available, echoing studies done in other countries [33]. Perhaps defining a target population, and showing that the population was reached is more important for marketing and reporting to program supporters. Large corporations that support programming need venues for promoting their name and products, so clear and careful definition of the population is very important. It is possible that programs that are funded via private sources may have a stronger need for evaluation to document effectiveness in order to justify corporate funds allocated to programs and promotion of their good corporate citizenship. Government agencies that rely on voter satisfaction to ensure political stability may also need to reach a carefully selected segment of the population, although it is impossible to document whether this is the case.

Factors related to adoption were the most commonly reported of all the RE-AIM indicators. Nearly all programs reported the location of initiatives and programming and who would deliver programming. Presumably these would be important for consumers of the programs. Nevertheless, few programs reported indicators related to the proportion of sites or program delivery agents who were offered the program and actually delivered the program. In some cases, this was not feasible, if the program were delivered online; however, in other cases, it simply wasn’t information that programs presumably collected.

Nearly all the programs described the core program components, including information on for whom the program was intended, where they could do the program, and what the program featured. Most other indicators of implementation were not included. Indicators of maintenance were largely absent, either because the program was primarily over, without reporting maintenance, or there was no maintenance plan in place. Only one program showed evidence of having been institutionalized, by having staff roles and responsibilities center on program goals.

Strengths of this study include a carefully constructed coding protocol and thorough search in a virtually unexplored area of research. Limitations include a small sample size of programs, limiting the investigations of relationships among variables. Coding was dependent on whether programs had websites available, which may have excluded programs without websites, and the content of the websites. When information was not available on the websites, research staff made attempts to contact program staff to determine whether information was available and not posted. Future research is needed to determine other channels through which public health programs might be disseminated, and to determine better measurement strategies in order to evaluate their effectiveness.

The information collected using the RE-AIM framework has important implications for future research, policy, and practice. This study identified indicators within the RE-AIM framework where reporting from public PA programs might be improved. This information must be used to help guide the development and design of future PA programs in order to be able to include reporting indicators so that it is possible to measure the impact of these programs and how they can be replicated in other settings. Perhaps the biggest area for improvement is in the reporting of indicators for measurement and evaluation of program outcomes. The lack of indicators available in these arenas may be due in part to the overwhelming number of participants in national level programs. In another study evaluating the implementation of programming in the US, program implementers often believed that adding RE-AIM indicators to evaluation plans required special training or expertise, suggesting that simple educational strategies about the importance and ease of measurement might improve reporting in the future [34]. In order to make measurement efforts more feasible, program officials should foster partnerships and collaboration with university researchers who can apply scientific understanding to developing suitable strategies for evaluating program outcomes or physicians in community clinics to develop strategies for overcoming measurement barriers. Most people (95%) in Mexico receive health care in public clinics which are often seen as “one-stop shops” where patients can see a physician, complete necessary lab work, and receive health information [35]. Clinics are under the jurisdiction of the office of the state Secretary of Health (Secretaría de Salud), which is also responsible for public health programs, making this an ideal partnership. Public clinics are accessible in the community and have the equipment and trained staffing to coordinate evaluation efforts in conjunction with public health programs.

The RE-AIM framework can be used as an evaluation tool, it can also be used at the program’s conception to help guide the planning of the program to ensure adoption, successful implementation, and evaluation. Using the RE-AIM framework to guide the design and evaluation of public health programs can lead to the development of policies and standards that can increase the execution and reporting of RE-AIM indicators. This may produce important information on the effectiveness and replicability of these programs, by improving smaller program details like the allocation of program funding and training of program staff.