Program overview
The community-based physical activity program was available to children and adolescents aged 7–14 years residing within the local council district area. The program was promoted via on-site flyers and posters, through the program website, leaflets in community organisations (such as local supermarkets) and via word of mouth. The program set four core aims: to create ‘participation pathways’ (i.e., opportunities within the organisation that allow for continued sports participation and skill progression from early childhood into adolescence and adulthood) from ages 4–18 years in all sports included within the program, engage with the community using students as positive role models, encourage participation in sport and physical activity, and provide a fun and safe environment for young people to enjoy sport and maintain an active lifestyle. Attendees could participate in multiple sports and there was no minimum attendance period. During 2009–2010 when this evaluation took place, the program received partial financial support indirectly via National Lottery funding allocated by Sport England to the host site’s sports centre. As this funding stream was finite, a secondary aim was to generate revenue from the sports sessions to maintain its sustainability in the community. The program is unique in that it operates as part of a broader participation pathway which provides routes into performance-specific and participation-only pathways interchangeably. The participation pathway provides individuals with access to sporting facilities throughout their life with the long-term goal of contributing to a healthier lifestyle.
At the time of this evaluation, the program offered athletics, badminton, football, hockey, judo, multi-skills, netball, swimming, tennis and trampolining sessions. These activities were offered at a central hub (leisure facility), and as part of an outreach arm that delivers sporting opportunities in local schools, and provides expertise and support to local sports clubs. Hub-based sports are delivered through daily after-school sport sessions on a term-time basis and holiday sports camps during the school half-term breaks, and six week summer holiday. The aim was to mimic the school term times to maximise the program’s reach. The program had existed within the community for 7 years prior to the evaluation taking place.
The evaluation was based on a single mixed-method case study using a concurrent triangulation design [19]; triangulation is a method of comparing and contrasting multiple data sources, research methods or inferences to strengthen the validity of the interpretations [20]. The core premise is that all methods have inherent biases and limitations, therefore the use of only one method to assess a given phenomenon will inevitably yield biased and limited results [21]. As there are often multiple dimensions within a single case study, triangulation is a valuable method of corroborating evidence of the same phenomenon by viewing it from different perspectives, rather than converging on a single consistent account of the event [22]. Between-method triangulation was used within this case study to confirm the findings generated through one particular method by another. Integrating different methodologies in this way can improve the study’s validity, and overcome the biases inherent with quantitative and qualitative methodologies alone.
Due to the complexity and multiple components within the program, a mixed method approach was chosen to facilitate greater validity of inferences and more comprehensive and insightful evaluation [23]. Four embedded units of analysis were specified across two levels within the case. At the organisational level, units of analysis included the program managers (with overarching responsibility for the program) and the program coaches (delivering the sports sessions). At the individual level, units of analysis included parents (whose children attended the program) and children and adolescents who directly participated. Five sources of mixed method data were collected over 12 months which included archival records; program-related documentation; field notes from direct observations; interviews with the program managers, coaches and parents; and questionnaires from the children and adolescents attending. Qualitative and quantitative data were collected concurrently and analysed independently in short succession to avoid major developments or changes to the case.
Participant recruitment
Ethical approval for this study was granted by the University of Bath Ethics Committee and organisational consent was obtained from the program managers.
Interview participants
The three program managers and all 14 program coaches (head and senior coaches) were specifically targeted for recruitment due to their seniority level within the program. This strategy ensured key informants with experience of all 10 sports were included and individuals would have sufficient knowledge of the program to contribute evidence across all five RE-AIM dimensions. Staff were invited via email to participate in a 1-hour semi-structured interview to discuss their experiences and perspectives of the program. Parents of children attending the program were recruited opportunistically via letters of invitation distributed at the main program reception desk and at the end of sports sessions. Based on initial responses, the participants of sports without parent representation were contacted directly via on-site visits, and parents asked verbally to participate. The objective was to maximise parent representation across the program sports and minimise the potential effects of volunteer bias. Signed consent was obtained prior to interview commencement.
Child and adolescent participants
All children and adolescents attending the program during the data collection phase were eligible to take part. Letters were sent home to parents seeking passive consent for their child’s participation 4 weeks ahead of data collection. Questionnaires were completed at the end of sports sessions over a 2-week period in February 2010. Program attendees could participate in multiple sports and therefore complete multiple questionnaires unique to each activity. Questionnaires were anonymous and participants were able to opt out on the day of data collection.
Measurement tools
Interview schedule
The interview schedule contained 45 questions based on criteria within the five RE-AIM dimensions. The questions were framed in context of the program and the terminology tailored for the context of the managers, coaches or parents involved.
Questionnaires
A questionnaire lasting approximately 15 min was designed to assess the determinants of children’s and adolescents’ participation in the program, and factors influencing their sustained engagement. Each scale contained 21-items rated on a 5-point Likert scale ranging from 1 “unimportant” to 5 “very important”. Questions were based on environmental/organisational factors (i.e. facilities), social factors (i.e. family, peers), intrapersonal factors (i.e. goals, progression) and interpersonal factors (i.e. beliefs) as they map to the specified elements of the socioecological model [24, 25]. The reliability of the scales was established based on alpha coefficients, mean inter-item correlations and participation to item ratios.
Documentation
Internal program documentation describing the history and development of the program was requested monthly from the program managers and coaches. Attendance records were provided for one specific time point during data collection; February 2010. As program participation rates were transient throughout the year, the February attendance records were requested to correspond with the associated questionnaire data collection among the children and adolescents. External program documentation (i.e. online promotional material) was searched for weekly via the internet. Field notes were recorded throughout the 12-months data collection, i.e. during the interview and questionnaire data collection phases and impromptu site visits to gain a more reflective account of implementation in the natural context in which it occurred.
Additional evidence included 7 program-related documents: program advertising leaflets, posters and holiday camp brochures (N = 4), email correspondence with program managers outlining the program’s history and development (N = 1) and participant attendance records for February 2010 (N = 2). Archival records included local council census data containing local population figures and school statistics. Field notes were taken following 20 informal observations during the delivery of program sessions, data collection phases and from personal reflections of the program.
Data analysis
Interviews were transcribed verbatim, transcripts entered into NVivo8 and analysed using a framework approach [26]. Framework analysis uses a hierarchical thematic framework to classify and organise data based on key themes, concepts and emergent categories [27]. Following the preliminary stage of familiarisation, transcripts were systematically indexed using codes which mapped against the RE-AIM criteria for the five dimensions. The data were then charted individually for each participant and interpreted within-and-between sub-groups until a consensus on themes was reached. To increase transferability [28, 29] of the interview data, questions were framed around the RE-AIM criteria for all five dimensions and refined following expert review of content validity.
Questionnaire data was analysed using SPSSv14 and descriptive statistics reported. Mean item scores were produced for each sport independently, stratified by participant gender and age. As the data was non-independent, figures reported relate to the number of completed questionnaires by age and gender, not the number of participants in the sample.
Data synthesis
The interview data provided the greatest coverage of all five RE-AIM dimensions therefore initial synthesis was undertaken using this source. Firstly, the interview data was triangulated within each participant sub-group (program managers, coaches and parents). The strength of convergence was determined based on the frequency and extensiveness of overlapping themes. Themes were ranked across the participant group to identify those with the greatest vs the least convergence. It was expected that within-sub-groups, participants would have a similar experience and understanding of the case due to their position and level within it, therefore a similar emergence of themes was expected. Triangulation of interviews was then repeated using these emergent themes between participant sub-groups to identify differences across the case. Convergence between-groups was established when at least 2 of the 3 participant sub-groups referred to a theme. As the program managers, coaches and parents represented different levels within the case, they were not expected to have a similar experience of the program. Instances of evidence divergence were therefore expected and reported.
Evidence from the remaining four data sources was then additionally integrated into each dimension where applicable and triangulated with the existing interview themes. Questionnaire data contributed to assessing Reach, Effectiveness and Implementation. Documentation assisted with the assessment of Reach, Effectiveness, Adoption and organisational-level Maintenance. Population census data was used to assess Reach and Adoption, and field notes provided evidence for all five RE-AIM dimensions. (Additional file 1: Table S1) presents a summary of the evidence used to assess each RE-AIM dimension.
Dimension assessment score
Following triangulation of all five data sources within each RE-AIM dimension, an individual ‘success score’ was allocated to each dimension (1 = less successful, 2 = moderately successful and 3 = highly successful). The success score was based on (i) Data applicability (i.e. the extent that the available data could address the dimension criteria), and (ii) Dimension outcome (i.e. the positive/negative outcome based on assessment of the dimension criteria using the available data).
For example Reach (What is the absolute number, proportion and representativeness of individuals willing to participate?): the assessment was based on whether the data available (i.e. program attendance records and interview data on uptake) could assist in quantifying the number of individuals willing to participate (success criterion i) and secondly whether the program had attained a positive outcome in terms of its Reach, defined in this case as whether the individuals included within the evaluation were representative or not of the target population based on the triangulation of participant accounts (success criterion ii). A single score was given to each dimension combining these considerations, as the interaction of data applicability and outcome meant that a high score in one would be meaningless (in terms of data quality) without a high score in the other.