Predictors of validity and reliability of a physical activity record in adolescents

Participants

A convenience sample of 302 school-going adolescents, aged 11–15 years, was recruited from a rural (Nabón; N = 70) and an urban (Cuenca; N = 232) area in the Azuay province of the Ecuadorian Andes region (Figure 1). Seven mixed gender schools, four urban and three rural, were selected for the study. Except for one private urban school, all were public schools. All children aged 11–15 years (from grade 7 to 11) in the schools were invited to participate. The exclusion criteria used were: (i) having a medical condition that hampered physical activity or (ii) being pregnant. Parental informed consent forms were distributed to the children that were eligible for the study. Those who returned signed parental consent forms (N = 302) and completed individual assent forms (N = 302) were included in the study. The study took place from April to July in 2008 and was approved by the Ethics Committee of the Ghent University Hospital (B67020084010).

Design and procedures

The overall validity of the PA record was assessed by (i) comparing it with the accelerometer recordings (validity) and (ii) comparing the two administrations of the record (test-retest reliability). In addition, we used a socio-demographic and Likert scale questionnaire to explore whether overall validity was affected by factors at individual (age, sex, Body Mass Index (BMI) and perceived convenience to complete the PA record) and environmental (setting) level.

Data collection was organized during school hours on three occasions, i.e. (i) on the first day of the study, (ii) after one week of accelerometer measurement, and (iii) after three weeks. On the first day, we provided classroom demonstrations and instructions on how to wear the accelerometer and complete the PA record to both the participants and their teachers. All participants were instructed to complete the PA record for seven consecutive days and wear the accelerometer during the same time period, i.e. both instruments were temporally matched. During the measurement period, teachers and researchers regularly reminded the participants in class of the importance of completing the record as soon as possible after activities had ended. In addition, socio-demographic data (age, sex and setting) and anthropometry were collected for each participant on the first day. On the second visit, i.e. after one week, both accelerometers and the completed PA record were collected from the participants. The data from those students who (i) were absent, (ii) were not wearing the accelerometer or (iii) did not have their PA record with them on this day, were collected during additional visit(s) to the schools. Finally, three weeks after the first visit, the PA record was administered a second time to assess reliability. To maximize comparability of both PA record administrations, the same procedures were applied (i.e. use of the same measuring instrument, the same researcher explaining the PA record) under the same conditions at school (e.g. no holidays or special activities). During this second administration, a Likert scale questionnaire was administered to examine perceived convenience as a predictor of validity and reliability.

Anthropometric measurements

Anthropometric measurements were carried out in duplicate by two independently trained researchers while ensuring optimal privacy. Adolescents wore light clothing but no shoes during the measurements. Body height was measured and recorded to the nearest 0.1 cm with a portable stadiometer and body weight to the nearest 0.1 kg using a digital calibrated balance (model SECA 803, Seca GmbH & CO, Hamburg, Germany). Adolescents were classified, based on age and sex, into BMI categories for 11–15 year old adolescents (underweight, normal weight, overweight and obese) [23, 24].

PA record

To assess PA levels, a simplified version of a previously validated PA record was used [25]. The list of pre-defined common activities and related numerical codes present in the original PA record were omitted in our study. On the form, each day was divided into 96 intervals of 15 min with space available to report performed activities. The instrument and related instructions were provided in Spanish. To limit recall bias, participants were instructed to record all types of activities performed during 24 hours for 7 consecutive days either directly after finalizing the activities or as soon as possible after the activities had ended.

For the analysis, each 15 min activity interval of the PA record was converted to a MET-value using the compendium of energy expenditure values for young people [26], and classified into durations by multiplying the estimated MET value of the activity by the time engaged in it. Due to logistical constraints at the schools, the first and the last measurement day started and ended at 12:00 noon and were therefore omitted from analysis. To ensure comparability of total estimated time of the record and the accelerometer registrations, only the daily active time reported by the record was included. In addition, all days with less than nine hours (540 min) were omitted from analysis. Time (MET-min/day) spent on sedentary PA (SED) (≤ 1.5 METs), low intensity PA (LPA) (≥1.5 and < 3 METs), moderate intensity PA (MPA) (≥ 3 and < 6 METs) and vigorous intensity PA (VPA) (≥ 6 METs) were computed as outcome variables [27, 28].

Accelerometers

An objective assessment of PA levels was obtained using the uniaxial GT-256 and GT1M ActiGraph accelerometers (Actigraph Manufacturing Technology Incorporated, Fort Walton Beach FL, USA). These accelerometers are appropriate to measure PA behavior in adolescents and are considered comparable for the evaluation of intensity levels [29]. On the first day of the study and after measuring anthropometry, pre-initialized accelerometers were distributed and placed on the right side of the hip using an adjustable elastic belt. Participants received a demonstration from a trained researcher on how to wear the accelerometer. Instructions provided were for example “accelerometers could only be removed when sleeping, showering or engaging in other water activities”, “do not clean the accelerometer with a solvent”, “always wear the accelerometer on the same place on your waist”. In accordance with the protocol of the PA record, the accelerometers were programmed to initialize at 12:00 noon on the first measurement day and were set to register 1-minute epoch cycles.

Possible predictors of validity and reliability estimates

A Likert scale questionnaire was developed based on two focus groups (results not included in this study). Focus groups were conducted with participants that were different from those who participated in this study. These focus groups explored the factors that might compromise or promote the validity of any self-reported PA assessment. The focus groups allowed us to use appropriate language and tailor the questionnaire to this specific age group. This final questionnaire assessed the degree of difficulty (i.e. convenience) for the adolescents to complete the record by including questions on: (i) their time perception, (ii) recall bias and (iii) social norms/desirability. We used the following Likert scale response categories: 1 “strongly disagree”, 2 “disagree”, 3 “neutral”, 4 “agree”, 5 “strongly agree” and 6 “I don’t know”. A convenience scale (Cronbach’s α = 0.59) combining these questions was used to provide a comprehensive assessment of the degree of difficulty of completing the PA record. Children with a higher score on this scale were those who provided answers reflecting the most favorable conditions for completing the PA record. To explore the association of convenience with reliability and validity, tertiles from this scale were created. Finally, tertiles of participants who found it respectively the least, more or less, and the most convenient to complete the PA record were compared with one another.

Accelerometer data reduction

The Actilife Software (Actigraph Manufacturing Technology Incorporated, Fort Walton Beach FL, USA) was used to process the accelerometer data and computed both total registered time and time spent in different intensity levels of PA. Non-wearing time for the accelerometers was defined as 60 min of continuous zero values, allowing for 1–2 min registrations of less than 10 counts [30]. Accelerometers were considered malfunctioning if no counts were registered or when a constant number of counts were recorded during the whole day (N = 62) (Figure 1). As with the PA record, both the first and the last day of the administration period were excluded and all days with less than nine hours (540 min) were omitted from analysis (N = 2). The following cut-off points were adopted from other studies in this age-group to determine the time spent on different intensity levels of PA: SED (≤ 100 counts) [31, 32], LPA (101–759 counts) [32], MPA (760–4011 counts) [32, 33] and VPA (≥ 4012 counts) [33]. The cut-off point for MPA was chosen to detect moderate intensity non-ambulatory and ambulatory activities with high sensitivity and specificity [32]. The cut-off point for VPA was chosen to detect ambulatory vigorous intensity activity with high specificity for the age-group [33].

Data analysis

Data from the PA record were entered in double using Epi data (Version 3.14, Odense Denmark) and analyzed using Stata (Intercooled Stata version 12 Statacorp, college station, TX, USA). PA registrations were included if subjects had at least two days of corresponding accelerometer and PA record recordings. This inclusion criterion was chosen as the study aimed to estimate PA on a population level with as many subjects as possible. In this case, our criterion is acceptable even though this means having fewer valid days and higher individual variance [21]. As measurement agreement might vary with the total time captured, the measured time for each intensity level was standardized to 900 min, which corresponds to the assumption of nine hours of sleep a day. All analyses were performed using these standardized outcome measures for both the accelerometers and PA record. Descriptive data were reported as mean and SD. We tested differences in means between methods (validity) and repeated measures (reliability) using linear mixed effects models with the levels school and individual. Standard errors were estimated using the Huber-White sandwich estimator that relaxes distributional assumptions of homoscedasticity of the model residuals [34]. Statistical significance was set at an alpha level of 0.05 and all tests were two-sided.

The measurement agreement between the duration of each PA intensity level was examined for both validity and reliability using the Bland Altman diagnostic plots. The plots visualize the difference between the PA measurements (validity: PA record 1 – accelerometer; reliability: PA record 1 – PA record 2) against their average values. In case plots showed a tendency for the differences to increase as the magnitude of measurement increased, data were log-transformed and re-plotted [35]. In the latter case, the mean difference and Limits Of Agreement (LOA) were back transformed by taking the antilog and values were presented as percentages. In case of a linear trend between the differences and the mean of two measurements, the differences were regressed over the means to obtain LOA that are a linear combination of the mean of the two measurements (A) [35]. Lowess curves were used to visualize any group differences in validity and reliability. Classification agreement for both validity and reliability was further examined using linear weighted Kappa statistics and its 95% CI, based on groups defined by tertiles of SED, LPA, MPA and VPA. Strength of agreement for the kappa coefficient was evaluated using the standards as proposed by Landis and Koch [36]. To account for prevalence and bias effects the prevalence-adjusted and bias-adjusted kappa (PABAK) is presented alongside the kappa statistics [37]. Finally, we assessed the association between gender, convenience level (in tertiles), age, setting (urban vs. rural) and BMI with reliability and validity. For this purpose we used the differences (validity: PA record 1 – accelerometer; reliability: PA record 1 – PA record 2) for each PA intensity level between 2 measurements as an outcome variable. These analyses were performed separately for each intensity level using linear mixed effects models with robust estimation for SE to account for clustering of estimates at the school level [34].

Characteristics of participants

A subsample of 113 adolescents (48.7% male) provided data for both PA records (Figure 1). There was no difference between the adolescents that provided data for both PA records and those who had data on the first PA record and accelerometer in terms of mean age (P = 0.54), weight (P = 0.68), height (P = 0.22) and BMI (P = 0.97). Finally, there was no significant difference in the convenience score between those participants included in the final sample and those participants initially recruited (P = 0.84).

Descriptive PA estimates

Validity

Results for validity are provided for those participants with data on both the PA record and accelerometers (N = 140) (Figure 2). On average, the PA record estimated 57 min (95% CI: [36;77] and LOA [-189;303]) more time spent on SED. For LPA, differences between both methods increased when more LPA was reported (β = 1.22; CI [0.91; 1.54]; lower LOA: -426.7 + 0.8A; upper LOA: -278.4 + 1.7A). Log transforming MPA showed that record estimates were on average 17% (CI [-26%; -8%]) lower than the accelerometers, but LOA were wide [-77.5%; 205%]. For lower mean MPA, adolescents tend to under report whilst for higher mean MPA they over report. For VPA the measurement disagreement between the PA record and accelerometer increased with increasing time measured. After log transforming, it is clear that the record reported more time spent on VPA (224%; CI [145%; 327%]) compared to the accelerometer. For VPA, the wide LOA [-84%; 6390%] indicated large discrepancies between both methods for some individuals.

Reliability

Results for reliability are provided for those participants with data on both PA records (N = 113) (Figure 3). After log transformation, the first PA record reported significantly less time spent on SED (mean difference -14%, CI: [-19%, -9%] and LOA [-53%; 58%]) than the second PA record. The SED plot showed fair agreement as both mean difference and LOA were within acceptable limits. For LPA the first record significantly exceeded the second with 25% (CI: [11%, 42%]) with LOA [-66%; 367%]. The plots of MPA and VPA also showed how the mean bias was acceptable for the different intensities of PA. For MPA and VPA the mean differences of the first PA record were respectively 19% (CI: [-3%; 46%] and 12% (CI: [-23%; -52%]) higher than the registrations of the second PA record. Even though these mean differences were acceptable, the LOA for MPA [-87%; 933%] and VPA [-94%; 1889%] indicated large discrepancies between both records in individuals. The differences between the repeated measures decreased with higher mean estimates of MPA.

When comparing both PA records, an overall moderate classification agreement was found. The kappa statistics improved when taking into account the impact of prevalence and bias in determining the magnitude of the kappa coefficient (Table 3).

Predictors of validity

Predictors of reliability