The concept of health literacy was introduced in China in 2005 by the Chinese government through a manual entitled “Basic Knowledge and Skills of People’s Health Literacy” [1, 2]. That manual used the definition of health literacy of the World Health Organization: the cognitive and social skills which determine the motivation and ability of individuals to gain access to, understand and use information in ways which promote and maintain good health [3]. Under that definition, health literacy goes beyond the narrow concept of health education and individual behavior-oriented communication: it addresses the environmental, political, and social factors that determine health. The US Institute of Medicine defines health literacy in a similar way: health literacy is a set of skills that enable people to participate more fully in society instead of simply functional capabilities [4]. The ability to read and write is the foundation for health literacy, upon which a range of complementary skills can be built [5].

Based on the existing situation in China, health literacy was defined by its government as a set of capabilities in three domains in 2008: conceptual knowledge and attitudes; behavior and lifestyle; and health-related skills. The first nationwide survey on health literacy in China was conducted in 2008, and it focused on health knowledge [6]. The second national survey was conducted in 2012, with an emphasis on basic reading ability, arithmetic, and understanding medical information [7].

Internationally, the most commonly used measure of health literacy is the Rapid Estimate of Adult Literacy in Medicine and its shortened version; these assess an adult patient’s ability to read common medical terms and lay expressions for body parts and illnesses [8–10]. The Test of Functional Health Literacy in Adults and its shortened version are timed tests of reading comprehension of medical information [11, 12]. Other measures of health literacy in clinical settings include the following: the Medical Achievement Reading Test; the Newest Vital Sign; the Set of Brief Screening Questions; Functional, Communicative and Critical Health Literacy; the eHealth Literacy Scale; the Cancer Health Literacy Test; and the Diabetes Numeracy Test [13–20]. These measures focus on a single dimension of health literacy, rather than identifying its multidimensional nature [21].

By contrast, some measures have expanded the scope of medical care-related literacy; they include the following: the Health Activity Literacy Scale; the Demographic Assessment of Health Literacy; the 2003 National Assessment of Adult Literacy; the Adult Literacy and Life Skills Survey; and the Health Literacy Assessment Using Talking Touchscreen Technology [22–27]. These scales and questionnaires are more comprehensive because they involve different health-related competencies. However, they are considered proxy measures owing to the lack of an explicit definition of the concept of health literacy [28].

In China, health literacy in clinical settings has been measured using translated versions of scales used overseas as well as original Chinese scales among certain populations [29–38]. For example, health literacy among older adults has been measured using the Chinese version of the Rapid Estimate of Adult Literacy in Medicine [33]; the translated version of the Test of Functional Health Literacy in Adults was employed to measure health literacy among adolescents aged 12–16 years in Nanning; the eHealth Literacy Scale was translated and used on a sample of senior high school students [34]; the Chinese version of the Diabetes Numeracy Test was used in a cluster-randomized trial in patients with diabetes [35]; and a three-question measure of health literacy derived from a systematic review was applied among cataract patients [36, 39]. However, all these studies investigated cross-sectional health literacy without evaluating the instruments employed [40–42]. In the present study, we assessed the reliability and construct validity of the Chinese Resident Health Literacy Scale based on item response theory (IRT) and classical test theory using a population-based sample from Hunan Province in 2012. We also examined the association between health literacy and sociodemographic factors.

The Spearman–Brown split-half coefficient was 0.94. The overall Cronbach’s alpha was 0.95; Cronbach’s alpha of the three dimensions was as follows: 0.90 (knowledge and attitude, 38 items); 0.83 (behavior and lifestyle, 22 items); and 0.85 (skills, 20 items). The two-parameter logistic model fitted the data well (P >0.05). The difficulty and discrimination parameters from the IRT model appear in Table 3. Most items exhibited good discriminative power and moderate difficulty. As shown in Fig. 2, the test information reached a peak when the participants’ ability was between −1 and 0, which indicates that the measurement was most discriminative among participants with limited to medium-level abilities in health literacy.

		Correct (%)	Item correlation to dimension score	Factor loading	IRT parameters
					Discrimination	Difficulty
Dimension 1: Knowledge and attitudes
A01	Prevention of influenza	66.2	0.44	0.37	0.60	−0.82
A02	Hypertension	85.3	0.40	0.35	0.75	−1.78
A03	Health food	88.3	0.28	0.24	0.51	−2.67a
A04	Transfusion	64.4	0.51	0.47	0.80	−0.63
A05	AIDS detection and consultation	92.1	0.32	0.28	0.74	−2.43a
A06	Pregnancy and hazardous job	91.2	0.31	0.25	0.65	−2.54a
A08	Range of body temperature	76.5	0.27	0.20	0.33	−2.25a
A11	Food label	96.7	0.23	0.19	0.74	−3.21a
A15	Health examination	72.8	0.44	0.39	0.68	−1.11
B01	Definition of health	71.5	0.59	0.55	1.18	−0.79
B02	Improve resident’s health	78.4	0.56	0.50	1.19	−1.06
B03	Blood donation	80.4	0.50	0.46	0.99	−1.24
B04	Transmission of hepatitis B	70.7	0.51	0.48	0.83	−0.89
B05	Self-measured blood pressure	51.7	0.50	0.48	0.74	−0.11
B07	Early signals of cancer	60.4	0.59	0.57	1.06	−0.41
B08	Chronic diseases	56.7	0.50	0.49	0.76	−0.32
B09	Management of gas poisoning	80.4	0.49	0.43	0.93	−1.29
B10	Treatment of tuberculosis	80.9	0.40	0.36	0.66	−1.62
B11	Folic acid supplementation	56.5	0.53	0.53	0.85	−0.30
B12	Occupational protection	69.6	0.53	0.48	0.87	−0.81
B13	Iodine deficiency	71.9	0.53	0.51	0.95	−0.87
B14	Pesticide residues	41.0	0.31	0.31	0.40	0.56a
B18	Children immunization	85.4	0.47	0.42	0.99	−1.53
B23	Health knowledge	71.2	0.44	0.41	0.66	−1.04
B26	Warning icons	71.0	0.57	0.55	1.08	−0.80
B31	Adult pulse	72.7	0.49	0.46	0.81	−0.99
B34	Dog bite	90.4	0.29	0.21	0.57	−2.72a
B38	Expired food	87.0	0.46	0.42	1.01	−1.62
C02	Liver function	27.9	0.48	0.52	0.88	0.85
C03	Transmission of tuberculosis	30.4	0.43	0.47	0.72	0.83
C04	Child fever and rash	51.9	0.58	0.59	1.01	−0.12
C05	Osteoporosis	10.8	0.42	0.48	1.41	1.53
C06	Prevention of chronic diseases	43.4	0.55	0.55	0.91	0.20
C07	Health food selection	54.5	0.60	0.62	1.09	−0.21
C08	Dead livestock	70.5	0.52	0.50	0.86	−0.87
C17	Pesticide custody	61.7	0.34	0.29	0.32	−0.90a
D06	Weight control	70.9	0.50	0.47	0.73	−0.38
D07	Obesity-related disease	75.2	0.53	0.53	0.90	−0.86
Dimension 2: Behavior and lifestyle
A07	Fruit and vegetable	75.2	0.40	0.36	0.53	−1.44
A09	Internet addiction	90.4	0.20	0.16	0.29	−4.74a
A10	Child and adolescent depression	87.6	0.37	0.32	0.68	−2.07a
A12	Lack of sleep	92.4	0.30	0.25	0.68	−2.60a
A13	Obtain health knowledge	93.3	0.30	0.26	0.72	−2.61a
A14	Attitude toward chronic diseases	57.0	0.38	0.32	0.41	−0.45a
B06	Smoking	54.9	0.55	0.63	0.83	−0.21
B15	Sport and electrolytes	70.0	0.48	0.44	0.71	−0.90
B16	Mental health	86.0	0.51	0.50	1.34	−1.34
B17	Public health service	46.3	0.56	0.52	0.88	0.12
B19	Fever symptom	79.3	0.42	0.33	0.64	−1.51
B20	Adverse drug reaction	79.0	0.39	0.31	0.57	−1.62a
B21	Health care card for pregnant women	52.6	0.45	0.40	0.57	−0.14
B27	Medical care procedure	71.8	0.57	0.54	1.14	−0.78
B29	House ventilation	65.7	0.50	0.45	0.79	−0.66
B39	Drug addiction	61.1	0.52	0.52	0.76	−0.48
C01	Mental health promotion	50.8	0.58	0.55	0.99	−0.05
C10	Bean products	23.4	0.52	0.49	1.03	1.00
C11	Sport and health	41.5	0.60	0.58	1.10	0.26
C13	Cough and sneeze	32.4	0.51	0.45	0.81	0.70
C14	Length of hospital stay	59.8	0.56	0.54	0.90	−0.39
C15	Management of type 2 diabetes	26.8	0.55	0.50	1.03	0.84
Dimension 3: Skills
B22	Medical institutions	77.4	0.55	0.52	1.12	−0.88
B24	Infectious diseases	78.2	0.58	0.55	1.35	−0.82
B25	Food poisoning	78.6	0.57	0.54	1.36	−0.85
B28	OTC drugs	37.3	0.54	0.51	0.59	1.05
B30	Reading a thermometer	51.2	0.59	0.56	0.80	0.18
B32	Prenatal examination	41.2	0.44	0.39	0.38	1.05
B33	Condoms and STDs	84.2	0.52	0.51	1.19	−1.13
B35	Slight burns and scalds	70.9	0.49	0.44	0.69	−0.80
B36	Fire escape	77.8	0.46	0.41	0.66	−1.24
B37	Bleeding wound management	22.7	−0.04	−0.16	0.03	0.00a
B40	Electrical injury	70.6	0.43	0.40	0.42	−1.04
C09	Cardiopulmonary resuscitation	53.7	0.65	0.65	1.06	0.09
C12	Diabetes	43.8	0.51	0.46	0.53	0.60
C16	Brest feeding	42.3	0.51	0.50	0.56	0.70
C18	Lightning storms	76.9	0.44	0.40	0.58	−1.23
D01	Indications of amoxicillin	25.0	0.55	0.54	0.86	1.42
D02	Use of amoxicillin	64.4	0.50	0.43	0.55	−0.38
D03	Adverse reaction of amoxicillin	54.2	0.65	0.62	1.07	0.05
D04	Calculation of BMI	41.8	0.62	0.58	0.94	0.60
D05	Classification of BMI	45.7	0.59	0.53	0.77	0.46

^aItems that met two or more criteria for deletion were removed

With the CFA results, the three-factor model showed slightly better fit than the one-factor model. Correlations among the three factors (knowledge and attitudes; behavior and lifestyle; skills) were 0.96–0.98, which indicates good evidence for unidimensionality, i.e., the dominant dimension of health literacy. Factor loading and the correlation coefficient between items and dimensional scores are presented in Table 3.

In all, 16 items were removed from the scale according to the criteria of item selection; 10 of them were true-or-false questions, which showed poor discriminative power and small factor loading. Sixty-four items were selected according to classical and modern test theory standards. The Spearman-Brown coefficient was 0.94. The overall Cronbach’s alpha was 0.95; Cronbach’s alpha of the three dimensions was as follows: 0.90 (knowledge and attitude, 30 items); 0.83 (behavior and lifestyle, 16 items); and 0.86 (skills, 18 items). Goodness-of-fit of the CFA and the IRT models improved slightly compared with the original scale. Factor loading, difficulty parameters, and discrimination parameters of all the items met the criteria.

A shorter version of the scale, comprising 19 items with discrimination parameters ≥1.0, was also created. The shorter version consisted of eight items in the knowledge and attitude dimension, five items in the behavior and lifestyle dimension, and six items in the health-related skill dimension. The overall Cronbach’s alpha was 0.88; Cronbach’s alpha of the three dimensions was 0.76, 0.64, and 0.77 respectively. The split-half coefficient was 0.87. The correlation coefficients and factor loadings of all the items were above 0.4 (mostly >0.5), and the discrimination parameters of all the items were 0.5–2.0 (mostly 1.0–2.0).

Differential item functioning in the IRT model was used to examine measurement invariance. The chi-square tests showed significant measurement invariance in both gender and race (P <0.05); however, the slope and threshold parameters were very close between male and female as well as between urban and rural groups.

To validate the scale used in the 2012 National Health Literacy Survey, we performed this study using a population-based sample in Hunan Province. Classical test theory (Cronbach’s alpha, split-half coefficient, and factor analysis) and modern test theory (IRT) were used in validating the scale. We found that the 2012 scale of health literacy meets psychometric standards. The overall Cronbach’s alpha was 0.95. The assumption that the scale measures a unidimensional construct was supported by the three-factor model fit being approximately that of the one-factor model fit and the three factors (knowledge and attitudes; behavior and lifestyle; skills) being highly correlated. Among the 80 items tested, 16 performed poorly and were removed. The remaining 64 items yielded a reliable estimate of health literacy, especially among participants with moderate and limited health literacy. The short version of the scale, which comprises 19 items with discrimination parameters ≥1.0, did not meet the standards for individual measurement (reliability ≥0.9). Nevertheless, the short version may still be effective for group comparisons [52].

In IRT, an item is useful only when it has good discrimination and its difficulty corresponds to a certain range in the ability scale: questions that are too hard or too easy provide little information [53]. However, if the discrimination is too high (i.e., greater than 2.5, as seen in clinical and psychological studies), the measured construct is often conceptually narrow. We limited the discrimination parameters to 0.5–2.0 because health literacy is a relatively broad concept. In this study, we identified items with inappropriate discrimination and difficulty. Most of them also had low factor loadings and correlation to the dimension score. However, the test used in the present study is time consuming. It usually took 30 min for an adult to complete the test; it took even longer for participants with limited literacy. Thus, in the future, it will be necessary to develop computerized adaptive testing and provide participants with short, tailored tests that have scores comparable to those of fixed-length tests.

Differential item functioning showed significant measurement invariance within both gender and race; however, the slope and threshold parameters were extremely close between the male and female as well as between the urban and rural groups. We observed no large differences between gender and race groups. The sample size in our study was sufficiently large to detect such slight differences. Thus, our results suggest that the Chinese Resident Health Literacy Scale may be efficiently applied for Chinese subjects of different genders and races for comparable scores.

The demographic factors associated with health literacy included education level, occupation, and annual income. Participants with higher education and better economic status were more likely to have adequate health literacy. Gender, age, race, and type of residence were found to be insignificant in the regression. The multilevel model identified an obvious intracluster correlation at the city level (primary unit in the sampling frame), with an intracluster correlation coefficient of 34.5 %. Health literacy is the outcome of health promotion, and both health literacy and socioeconomic factors are determinants of health. However, the potential of health education as a tool for promoting the social determinants of health has been neglected [54]. Health education should not focus only on changing personal lifestyles and improving compliance with disease management, but also on raising awareness of the social determinants of health [5].

Some limitations of this study deserve mention. First, we did not assess the content validity since the scale was initially developed by an expert panel from the Ministry of Health. Second, we did not perform repeated measures during the field study. Thus, the test-retest reliability was not determined. Third, as noted above, the test is time consuming: it usually took 30 min for an adult to complete and even longer for participants with limited literacy or other conditions.

Despite these limitations, this study has a number of implications. First, the original scale was found to be appropriate in terms of reliability and validity. We removed 16 items according to factor analysis and IRT, and the scores of the 64-item scale correlated highly with the scores of the original scale. Accordingly, the main conclusions of the 2012 National Health Literacy Survey were unaffected by validation of the scale it employed. Second, a shortened 19-item version was created because applying the original scale was very time consuming. The 19-item version was found to be slightly inferior to the original scale in terms of reliability (Cronbach’s alpha decreased from 0.95 to 0.88); however, it would still be effective for group comparisons and population studies. Third, the instruments used in the National Health Literacy Survey in 2008 and 2012 were different. Therefore, a direct comparison based on raw scores would be inappropriate. In the present study, IRT provided an opportunity for longitudinal comparison.

We evaluated and revised the Chinese Resident Health Literacy Scale based on IRT and classical test theory using a population-based sample in Hunan, China. The revised 64-item scale was found to have strong psychometric properties and be free of obvious measurement invariance within the race and gender groups employed in this study. This is the first investigation to evaluate and revise the instrument used in the 2012 National Health Literacy Survey in China. The findings of this study support use of the new instrument in research into health literacy in public health settings, and this investigation offers useful implications for future studies.