Field-testing of the rapid assessment of disability questionnaire

Ethics approval

Ethics approval was obtained from the University of Melbourne Human Research Ethics Committee (Australia), the Royal Victorian Eye and Ear Hospital Human Research and Ethics Committee (Australia), the International Centre for Diarrhoeal Disease Research, Bangladesh (icddr, b) Ethical Review Committee (Bangladesh), and the Fijian National Research Ethics Review Committee (Fiji). The study was conducted in accordance with the tenets of the Declaration of Helsinki. All participants provided a written or verbal informed consent. In case of participants who were not literate or were unable to provide written consent their verbal consent was obtained, i.e. the consent form was read to participants and their verbal agreement was recorded by the interviewer in front of a witness.

The RAD questionnaire

The RAD questionnaire was developed based on a literature review, expert panel workshops and in-depth interviews with people with disability [10, 12]. The RAD questionnaire is interviewer administered and is comprised of two parts: a household questionnaire administered to the head of the household assessing socio-economic indicators such as source of water, electricity, sanitation, housing materials and assets including durable goods (e.g. television, radio, bicycle), and ownership of house, land and cattle and an individual questionnaire administered to each of the eligible members of the household.

The initial version of the individual questionnaire comprised five sections 1) demographics, 2) self-assessment of functioning, 3) awareness of the rights of people with disability, 4) well-being, and 5) access to the community [12]. Section 1 included 25 items related to demographics such as age, gender, ethnicity, religion, marital status, education, occupation, health conditions and assistive devices used. Section 2 comprised 18 items related to functioning in eight domains: vision (1 item), hearing (1 item), communication (1 item), mobility (2 items), self-care (1 item), gross and fine motor (1 item), cognitive (4 items), appearance (1 item) and psychological distress (6 items). The items related to psychological distress are from the Kessler-6 scale [12, 13]. Each item asked the participants to report whether they experienced difficulty in functioning in the last six months even when using assistive devices available to them (e.g. difficulty seeing even if wearing glasses). Those who answered ‘yes’ to an item were then asked to rate the frequency of difficulty as ‘some of the time’, ‘most of the time’, or ‘all of the time’. Those who self-reported having difficulty ‘most’ or ‘all of the time’ to at least one item from the first seven domains or at least two items from the psychological distress domain were considered as people potentially experiencing disability. The cut off criteria was determined based on the recommendations from the WG for short set questions [14] and also based on the consensus among the research group.

The purpose of section 3 was to obtain information about the awareness of the rights of people with disability. Section 3 comprised 17 items related to awareness of the rights of people with disability such as the right to ‘access information’, ‘live in a safe home environment’, ‘go to school or study’, ‘access health care’, ‘marry’ and ‘have children’. Each question was phrased “To what extent do you have rights to ….?”, and the response scale was rated on ‘none’, ‘some’ and ‘all’ categories.

Section 4 comprised 18 items to assess individuals’ perception of their well-being (Figure 1). The items included ‘good health’, ‘making friends’, ‘being safe in daily life’, ‘opinion counted in family’ and ‘taking care of one’s self’, ‘taking care of household’, ‘making new friends’, ‘maintaining family relationships’, and ‘respected in the community’ with the frequency of experiencing each situation reported on a four-category response scale ranging from ‘never’ to ‘all of the time’.

Section 5 asked respondents whether they had access to 14 community related domains such as access to health services, education and vocational training, work opportunities, social, legal, religious and rehabilitation. Each domain had three questions and the first question was phrased “In the last 6 months, did you have access to…… as much as you needed?” If they responded that they had not accessed services in this domain as much as they needed, they were asked to report reasons/barriers for this. Respondents were then asked which of the reasons given had limited access to the domain the most.

Psychometric evaluation of the RAD questionnaire

The psychometric evaluation of the RAD questionnaire was conducted in two phases. The first phase involved field-testing in Bangladesh to assess the psychometric properties of the questionnaire. The findings of the Bangladesh field-testing study were used to identify poorly performing items, with the goal of removing or revising such items to enhance the psychometric robustness of the RAD. The second phase was conducted in Fiji to assess measurement properties of the revised questionnaire, to assess the sensitivity and specificity of the RAD to identify people with disability, and to assess the performance of the RAD in a cross-cultural setting. The findings from this phase of the study further informed modifications to the questionnaire and the development of the final version of the RAD.

Training of interviewers

Field supervisors and interviewers were recruited based on their skills and previous experience. Some of the interviewers were people with disability. Training was provided for a week that included disability inclusion, study design, recruitment of participants, administration of the RAD questionnaire, ethics in research and collecting survey data, conducting interviews, data storage and referral mechanisms for participants. Supervised field practice sessions were also conducted as part of the training.

Statistical analysis

Household questionnaire data from the Bangladesh study were used to derive a household wealth index using principal component analysis (PCA). Individuals were then classified into one of the three socio-economic groups based on values of the wealth index: the bottom 40% as poor, the next 40% as middle and the top 20% as rich [17, 18]. Descriptive statistical analysis was performed on the socio-demographic variables in section 1. Paired samples t-tests (for continuous variables) and generalised estimating equations (for categorical variables) were used to compare socio-demographic characteristics of people with disability and their matched controls.

Locations of category thresholds [19] for items from section 2 were analysed to examine whether participants could discriminate between response categories. Following this evaluation, responses were converted to binary variables of having a disability (difficulty ‘most’ or ‘all of the time’) or not (no difficulty or ‘some of the time’). Inter-item correlations (point biserial correlation coefficients for dichotomous variables) between section 2 items were examined to identify redundant items. Correlations of 0.75 or higher were interpreted as indicating redundancy.

Sensitivity and specificity of the cut off criteria used in RAD (i.e. self-reported difficulty ‘most’ or ‘all of the time’ to at least one item from the physical/sensory/cognitive domains or at least two items from the psychological distress domain of section 2) was assessed using data from Fijian field study. Sensitivity was calculated as the proportion of participants with disability identified by the RAD questionnaire among cases. Specificity was defined as the proportion of participants not identified to have disability by the RAD questionnaire among controls. Given that the RAD is intended for use as a population-level screening tool, we considered sensitivity and specificity values 60%-69% to be acceptable, 70%-79% good, 80%-89% very good, and ≥90% excellent.

Rasch modelling, a form of item response theory (IRT) that transforms ordinal scores into interval-level estimates [19, 20] (in logit units), was used to test construct validity of the sections 3 and 4 through the assessment of threshold ordering, unidimensionality, and targeting. Only sections 3 and 4 were subjected to Rasch analysis as these are the only sections of the RAD intended to be used for calculating section totals by summing responses across section items to capture participants’ perceived access to rights and well-being, respectively.

The Andrich rating scale model was used to obtain estimates of the level of characteristic (i.e., awareness of rights and well-being) represented by each item (item measure) and perceived level of these characteristics for each participant (person measure). Locations of category thresholds [19] were also estimated and items within each section were assessed for the evidence of disordered thresholds and to examine whether participants could discriminate between response categories. The response categories at least 1.4 logits apart were considered to be clearly differentiated by participants [21].

Measurement precision of the two sections, i.e. the ability to distinguish different levels of underlying characteristic, was assessed by the person separation index (PSI) and person separation reliability (PSR) scores. A PSI of ≥2.0 and a PSR ≥0.8 are considered minimally acceptable levels of separation, indicating that the questionnaire can distinguish at least three different levels of the characteristic of interest [20].

The extent to which each section measured a single underlying concept (i.e., section unidimensionality) was evaluated using item fit statistics and PCA of residual correlations. Item fit was assessed with the information-weighted (infit) mean-square (MnSq) statistics and standardised fit residuals (Zstd), both of which measure the discrepancy between the observed responses and responses predicted from the Rasch model. MnSq values of 0.7 to 1.3 and Zstd values of −2.0 to 2.0 were considered acceptable [22]. Values below the lower limit indicate redundancy and values above the upper limit indicate unacceptable level of variability in the responses (high measurement error). When a scale functions as intended, with all items measuring a single dimension, this single dimension should explain most of the correlations in the item set. The existence of any additional dimensions is inferred through the assessment of residual variance (variance unexplained by the primary dimension) using PCA. Unidimensionality was inferred if 1) the primary dimension explained at least 50% of the variance in the responses to the section items and 2) the first residual component had an eigenvalue <2.0, indicating that variance explained by this residual component carries less information than 2 items, the smallest number of items needed to form a dimension [23].

The capacity of the items to adequately represent the full continuum of a characteristic of interest is called targeting and was assessed by visual inspection of the person-item map and the difference between means of person and item measures. For a perfectly targeting instrument, the difference in means would be 0; an absolute difference of >1.0 logits indicates significant mistargeting and occurs when the amount of characteristics in the sample is substantially higher or lower than the average level of the same characteristic targeted by the items [24].

Sections 3, 4 and 5 were assessed on known-groups validity by comparing the scores of people with disability and controls using paired-samples t-tests (sections 3 and 4) and generalised estimating equations (section 5). It was expected that people with disability would score significantly higher on each section than their matched controls.

Rasch analysis was performed using Winsteps version 3.74 (Winsteps, Chicago, IL). Remaining statistical analyses were performed using PASW Statistics 18 (PASW Statistics for Windows, SPSS Inc., Chicago, IL).

Questionnaire administration

Field testing of the RAD sampling and interview methodology showed that, on average, the full interview (sections 1–4) takes approximately 45 minutes to complete, while the basic version (sections 1–2) takes approximately 20 minutes. Thus, a survey team comprising three interviewers is able to interview 50 people (one segment) in a day.

Participants

A total of 202 adults were recruited in Fiji and of these, 101 were cases (mean ± SD age: 44.2 ± 15.3 years) and 101 were controls (mean ± SD age: 44.1 ± 15.8 years) (Table 1). The numbers of participants who reported difficulties on physical/sensory/cognitive domains and psychological distress domain were 60 (59.4%) and 27 (26.7%) respectively. Cases had significantly lower education levels (no education: 18.8% vs 3.0%, 10 years or more: 28.7% vs 47.5%, p = 0.004) and significantly higher frequency of unemployment (33.7% vs 12.9%, p < 0.001) compared to controls.

Section 2: self-assessment of functioning

The average locations of response categories for the 18 items in section 2 along the disability continuum were analysed (Figure 2). There was substantial separation in the average locations of the two lowest response categories (0 = never and 1 = some of the time) and between the second and third (2 = most of the time and 3 = all of the time) response categories. Response categories 2 and 3 appeared to be very close to each other on the disability continuum, indicating that these response categories could not be distinguished by the respondents and were representing similar level of disability.

Section 3: awareness of the rights of people with disability

Examination of the average locations of response categories of the 17 items in section 3 on the awareness of rights continuum showed that while there was no threshold disordering, the average locations of the two lowest response categories (1 = none and 2 = some) were very close to each other (<1.4 logits separation). This indicated that participants could not discriminate three response categories, but used them as a dichotomous scale (yes/no). Category 1 (none) had large amount of random error (MnSq 1.34) with only 7% of the sample endorsing this response option. The most frequently endorsed response category was category 3 (all), which was endorsed by 48.8% to 82.9% of respondents depending on the item, indicating strong ceiling effects and/or bias towards acquiescent responding. The mean (±SD) person measure was 1.99 (±1.11) in people with disability and 2.03 (±0.10) logits in controls and there was no statistically significant difference between the two groups (p = 0.786).

The precision of the scale for measuring levels of rights awareness was poor, with PSI (1.58) and PSR (0.71) lower than the acceptable range (≥2.0 for PSI and ≥0.8 for PSR), indicating that the item set had low ability to discriminate between people with high and low levels of awareness of their rights. Section 3 also exhibited evidence of multidimensionality. The item ‘rights to have children if you want to’ had high amount of random error (MnSq 1.20, Zstd 3.20) and items ‘rights to make decisions about your own life’ (MnSq 0.82, Zstd −2.30), ‘rights to your opinion counting in family decisions’ (MnSq 0.84, Zstd −2.10) and ‘rights to be treated the same way as everyone else in society’ (MnSq 0.83, Zstd −2.60) showed evidence of redundancy. On PCA of the residuals, total variance explained by the item set was 32.9% and the eigenvalue of the first residual component was 2.1 further suggesting multidimensionality of the items set. However, when the Rasch scores of this section were further explored using exploratory factor analysis no meaningful separation of items into dimensions had emerged.

Section 4: well-being

The categories (1 = never, 2 = some of the time, 3 = most of the time and 4 = all of the time) were reversed for Rasch analysis so that higher values on the logit scale corresponded to higher levels of well-being. The response category thresholds were ordered as expected and the precision of the scale was good, with PSI 2.14 and PSR 0.82, supporting the ability of the scale to discriminate at least three levels of well-being. The most difficult item (corresponding to the highest level of well-being) was ‘able to make new friends’ (2.13 logits) and the least difficult item (the lowest level of well-being) was ‘maintaining family relationships’ (−1.95 logits). There were two misfitting items: ‘able to make new friends’ (MnSq 1.52, Zstd 7.1) and ‘able to maintain friendships’ (MnSq 1.39, Zstd 5.5). However, PCA of the residuals revealed that the variance explained by the items was 58% and the eigenvalue of the first residual component was 1.9 supporting unidimensionality of the scale. The ‘get the help you need’ item was found to be redundant with the ‘opportunity to help’ item (residual correlation of 0.3).

The mean ± SD person measure of 1.18 ± 1.34 indicated that the average level of perceived well-being of the sample was better than that represented by the items, which was unexpected given that people without disability comprised half of the sample. However, the targeting has been satisfactory with reasonably good spread of items along the well-being continuum of the sample (Figure 1) and people with disability had significantly poorer well-being scores compared to the controls (mean difference = 0.97, p < 0.001).

Section 5: access to the community