Measuring health literacy in populations: illuminating the design and development process of the European Health Literacy Survey Questionnaire (HLS-EU-Q)
BMC Public Health volume 13, Article number: 948 (2013)
Several measurement tools have been developed to measure health literacy. The tools vary in their approach and design, but few have focused on comprehensive health literacy in populations. This paper describes the design and development of the European Health Literacy Survey Questionnaire (HLS-EU-Q), an innovative, comprehensive tool to measure health literacy in populations.
Based on a conceptual model and definition, the process involved item development, pre-testing, field-testing, external consultation, plain language check, and translation from English to Bulgarian, Dutch, German, Greek, Polish, and Spanish.
The development process resulted in the HLS-EU-Q, which entailed two sections, a core health literacy section and a section on determinants and outcomes associated to health literacy. The health literacy section included 47 items addressing self-reported difficulties in accessing, understanding, appraising and applying information in tasks concerning decisions making in healthcare, disease prevention, and health promotion. The second section included items related to, health behaviour, health status, health service use, community participation, socio-demographic and socio-economic factors.
By illuminating the detailed steps in the design and development process of the HLS-EU-Q, it is the aim to provide a deeper understanding of its purpose, its capability and its limitations for others using the tool. By stimulating a wide application it is the vision that HLS-EU-Q will be validated in more countries to enhance the understanding of health literacy in different populations.
Health literacy is a composite term used to describe the capacities of persons to meet the complex demands related to health in modern society. As an outcome of health education and communication activities, it represents the cognitive and social skills that determine the motivation and ability of individuals to gain access to, understand and use information in ways that promote and maintain good health . The concept has gained increasing attention both in research and practice due to its close association to the social determinants of health , health behavior and health outcomes , health service use  and quality of health systems as well as capacity building for professionals . Along with the increasing interest in empirical work on health literacy, there has been a growing demand for tools to measure health literacy .
The existing tools that purport to measure health literacy vary in their approach and design, as well as in terms of their purpose. Some tools have been developed for the purpose of screening, and serve to divide people into categories with low or high levels of health literacy. Examples of this kind of tool are the Rapid Estimate of Adult Literacy in Medicine (REALM) [7, 8], the Test of Functional Health Literacy (TOFHLA) [9–11] and the Newest Vital Sign (NVS) . As they are often used in clinical settings, these tools are necessarily short and quick and easy to use. Other tools aim at measuring a broader concept of health literacy, with a view to provide an in-depth assessment of the dimensions of health literacy, or to explore its relationships with social determinants, health behavior, health status or healthy service use such as the National Assessment of Adult Literacy survey (NAAL) , the Critical Health Competence Test (CHC) , the Swiss Health Literacy Survey , the Health Literacy Management Scale (HeLMS)  and the Health Literacy Questionnaire (HLQ) . Furthermore, existing health literacy measurement tools differ in terms of their administration style and their focus on specific aspects, such as the recognition and pronunciation of medical terms, numeracy, comprehension, and decision-making competencies. In terms of their technical qualities, the tools differ in terms of scoring and ranges. Accordingly, the time and resources needed for application also vary [18, 19]. Yet, in spite of the wide range of tools that are available, it is recognized that many have substantial weaknesses . Existing tools are far from optimal and show several limitations . The most apparent shortcomings of most tools are that they fail to capture all relevant aspects of health literacy and only focus on one or a few dimensions of the concept; that they have a primary focus on personal attributes at the cost of population aspects; that they have an unclear relationship to current definitions and conceptual frameworks of health literacy; and that they show only weak associations with causes and outcomes of health literacy . According to Pleasant et al. , a comprehensive measure of health literacy should reflect the following attributes: build explicitly on a testable theory or conceptual framework of health literacy; be multi-dimensional in content and methodology, to reflect the emerging theories of health literacy as a construct with multiple conceptual domains and practical components; use multiple methods; distinguish health literacy clearly from communication; treat health literacy as a latent construct, in the sense that the measure should include multiple items that sample from the conceptual domains outlined by the underlying theory or conceptual framework; honor the principle of compatibility in the sense that the measure should not focus exclusively on the clinical setting to research public health behaviors and outcomes; allow comparison and/or be commensurate across contexts including culture, life course, population group, and research setting; and prioritize social research and public health applications versus clinical screening .
To accomplish the European Health Literacy Survey (HLS-EU), which aimed to measure and compare health literacy in populations in selected countries in Europe , the HLS-EU Consortium consisting of nine research institutes from Austria, Bulgaria, Germanya, Greece, Ireland, the Netherlands, Poland and Spain, developed the European Health Literacy Survey Questionnaire (HLS-EU-Q). It embraces the principles outlined by Pleasant and colleagues and captures the essential dimensions of health literacy as outlined in the definition and conceptual model proposed by Sorensen et al. .
The present paper describes the process of developing the HLS-EU-Q. Specifically, it provides a detailed outline of the structured and systematic approach that was taken concerning the item generation, pre-testing, field-testing, external consultation, plain language check, and translation of the tool, with the purpose of creating and testing a concept based, multidimensional, multinational, interdisciplinary and comprehensive measurement of health literacy in populations. As such, the paper provides insight to the extensive development process of designing the HLS-EU-Q, which is useful for its subsequent application and validation. Initially, the methods are described for each step performed in the development process. Then the results for each step are presented. Finally the development process and the attributes of the HLS-EU-Q are discussed in terms of quality and limitations.
Applying a concept validation approach
In line with the principles outlined by Pleasant et al. , the development of the HLS-EU-Q followed a concept validation approach. Therefore, the design process was guided by the conceptual model of health literacy derived from a systematic literature review of existing definitions and conceptualisations of the concept by Sorensen et al. . The model starts from a definition of health literacy which integrates the different aspects of health literacy as identified in the literature, stating that:
“Health literacy is linked to literacy and entails the motivation, knowledge and competencies to access, understand, appraise and apply health information in order to make judgements and take decisions in everyday life concerning healthcare, disease prevention and health promotion to maintain or improve quality of life throughout the course of life”.
Based on this definition, the HLS-EU Consortium developed a conceptual framework outlining the main dimensions of health literacy as mentioned in the literature, and integrating them in a logical model identifying the proximal and distal factors that may impact on health literacy, as well as potential consequences of health literacy in terms of health related behaviors, health outcomes and health service use . The core of the model can be conceived of as a 12 cell matrix positing the key processes off accessing, understanding appraising and applying health-related information within three domains (Table 1) :
The domain of healthcare, where health literacy refers to the ability to access information on medical or clinical issues, to understand medical information, to interpret and evaluate medical information, and to make informed decisions on medical issues and comply with medical advice.
The domain of disease prevention, where health literacy involves the ability to access information on risk factors for health, to understand information on risk factors and derive meaning, to interpret and evaluate information about risk factors, and to make informed decisions to protecting against risk factors for health.
The domain of health promotion, where health literacy refers to the ability to regularly update oneself on determinants of health in the social and physical environment and derive meaning, to interpret and evaluate information on determinants of health in the social and physical environment, and the ability to make informed decisions on health determinants in the social and physical environment and also engage in joint action.
Starting from the conceptual model, a logical, systematic and structured development process was undertaken, which included the following eight steps entailing qualitative as well as quantitative methods:
A Delphi procedure, which is a group facilitation technique including an iterative multistage process designed to transform opinion into group consensus , was applied to generate items to measure each of the 12 cells of the matrix. Hence, through successive Delphi rounds items were gathered, refined and synthesized by ranking and prioritizing opinions to shape the initial version of the HLS-EU-Q. In the first Delphi round all nine research teams in the HLS-EU Consortium were invited to propose items related to the twelve sub-domains in the HLS-EU matrix. The second Delphi exercise dealt with the selection of items proposed in the first round. The Delphi rounds were conducted through email to permit efficient participation and respondents had opportunities to revise their inputs and comment on summary feedback at each stage.
To test the face validity of the draft questionnaire, focus groups were organised by the HLS-EU project partners in Ireland (JF, GD), Greece (BK, DA) and the Netherlands (VS, KS, SvdB). Participants provided informed consent and each group were invited to give feedback on the design, clarity and content of the questionnaire. To keep the costs low recruitment of participants for the focus groups was done via convenience sampling , which entailed involving the most accessible, participants with knowledge on health and preferably health literacy as well as general citizen skills. Hence, the sample included students and academic staff from the three participating universities, respectively. The characteristics of the participants are outlined in Table 2.
The revised version resulting from the focus-group feedback was field-tested in two countries (Ireland and the Netherlands). The field test included 50 computer assisted face-to-face interviews in each country conducted by JF in Ireland and VS and KS in the Netherlands. To recruit participants for this field test, judgement sampling, also known as purposeful sampling was used to guarantee an equal distribution of participants in terms of the parameters age, gender and education . Due to an incorrect saving procedure data from one Dutch interview was lost, leaving a total of 99 interviews. The interview time varied from 25-90 minutes. The profile of the sample is described in Table 3.
The methodological approach concerning data analysis involved both a qualitative and quantitative analysis of the data. For the qualitative analysis, data derived from logbooks and observations made by the interviewers and general comments and feedback from participants were scrutinized using the recommendations to refine the questionnaire. The quantitative analysis involved an item analysis, Principal Component analysis (PCA) and reliability analysis on the scores of the respondents on the questionnaire items. For the item analysis, the distribution of the responses on each item was inspected to eliminate items with a low discriminative power (i.e., 95% or more of the answers in the same category). For the PCA, a separate analysis was performed for each domain (healthcare, disease prevention and health promotion), with the number of components fixed at four related to the four information-processing dimensions outlined in the health literacy matrix derived from the conceptual model and definitions and a VARIMAX rotation to yield maximum discrimination between the components. The resulting factor structures were inspected, and items without sufficient loading (< 0.30) on any of the components or with a small difference in factor loading on any two components were excluded. The remaining items were again entered into PCA. This iterative procedure was repeated until an interpretable component solution was obtained. Subscales were constructed on the basis the highest component loading of an item. The internal consistency of the scales obtained through the PCA was tested by means of the Cronbach’s alpha.
In addition to the field test, consultations were organised with experts in the field of health and health literacy to assess the construct validity as well as the technical qualities (scaling, ordering of items etc.) of the questionnaire. The experts were recruited from the national advisory panels that had been established as part of the HLS-EU Project and among the collaborative partners of the HLS-EU project to gain second opinions supplementing the work carried out by the HLS-EU Consortium [22, 26].
Finalisation of the questionnaire
The results of the pre-test, field test and expert consultations, respectively, were pooled and evaluated by a sub-group within the HLS-EU consortium (KS, JP, SvdB, ZS, GD) supplemented with input from the collaborative partner (RO). Items that did not fit well within the conceptual model and rationale of the questionnaire or which did not have direct or indirect relevance to the twelve sub-domains were eliminated. Items that were only indirectly associated to the rationale of the questionnaire were combined with other items. Proposed objective items such as questions related to concrete knowledge were discharged due to cultural discrepancies among the participating countries. Instead, it was decided only to include self-reporting items, similar to the practice of Chew et al. . Hence, the format of all items was changed from ‘statements’ to ‘questions’, and their formulation standardized so that all would assess the difficulty of a specific health relevant task, i.e.: “On a scale from very difficult to very easy, how easy would you say it is to …followed by the question to be answered on a Likert-type scale ranging from “very easy”, “easy”, “difficult” or “very difficult”. An answer category was added as “I don’t know”, which was only to be used by the interviewer. While it was ensured that the reformulated items stayed true to the original content, some new items were added, although not tested, to replace items that had been eliminated during the ‘culling’. This procedure resulted in a pre-final version of the questionnaire
Plain language check
The pre-final questionnaire was examined for plain language by literacy experts from the National Adult Literacy Agency in Ireland for its compliance with plain language guidelines.
The English version of the final questionnaire served as a master version for translation into the six other languages to be used in the European Health Literacy Survey (Bulgarian, Dutch, German, Greek, Polish and Spanish). Two independent translators translated the questionnaire from English, to the target language. For each language, a panel consisting of national research partners, the European Health Literacy Survey Coordinator, the translators and other relevant health professionals assessed the two translations with the aim of agreeing to a formal national version of the HLS-EU-Q.
For each step, protocols were made to ensure standardization of the procedures across the countries involved.
The results of each of the eight development steps are described in the following and the items included in the final questionnaire are presented in the Additional file 1.
According to the two successive Delphi rounds, the first Delphi resulted in a total of 136 generated items across the 12 sub-domains of health literacy. Most of the items were self-report statements, to be answered on a five-point Likert scale. In addition, two to four items in the form of objective knowledge tests were formulated for each domain (healthcare, disease prevention and health promotion) to test the level of achieved health literacy. The second Delphi dealt with the selection of items proposed in the first round, and resulted in a reduction to 43 health literacy items across the 12 sub-domains. This round was based on partial consensus. The final decisions were made by the coordination team from Maastricht University [KS, SvdB and VS] in the process of creating the draft version of the questionnaire to be tested in focus groups.
The focus group discussions resulted in feedback on the structure, clarity and content of the questionnaire.
With regard to the structure, participants suggested to change the order of items from focusing on the competencies related to information processing to a focus on the domains of healthcare, disease prevention and health promotion. Another comment with regard to the structure of the questionnaire concerned its repetitiveness. With regard to clarity, it was mentioned that the questionnaire was expert biased. Finally, with regard to content, remarks were that (i) the ‘objective’ questions and the ‘knowledge’ questions, were too difficult; (ii) items might be culturally sensitive, as some questions were not generic enough to cover differences in health systems and contexts across the eight countries; and (iii) some items were found to prompt socially/culturally acceptable answers, e.g., related to health beliefs. In addition, there were concerns about privacy and the extent to which respondents would be willing to share their opinion and reply to the health literacy related questions and, especially, to socio-economic status related questions.
Following the incorporation of the comments and suggestions from the focus groups, a revised version of the questionnaire was made, which consisted of 47 items associated to health literacy: 22 for the healthcare domain (of which 19 self-report items and 3 objective items testing the level of achievement); 13 for the disease prevention domain (10 self-report and 3 test items); and 11 for the health promotion domain (9 self-report and 2 test items).
The qualitative analysis of field test data derived from logbooks, observations by the interviewers and general comments from both the Irish and Dutch participants pointed out that although the content of the questions was well-understood and considered as appropriate, the questionnaire was seen as too lengthy, too comprehensive, too repetitive, and too expert biased in terms of language (e.g., use of the word “hypertension” instead of “blood pressure”). It was also considered very time-consuming, with the interview time varying between 30 and 90 minutes, with an average of approximately 60 minutes. As a result of these comments, the design of the questionnaire was slightly changed after the first 10 interviews. Highly similar items were combined into sets of questions, to avoid repetition and facilitate the process of answering the questions.
With regard to the quantitative analysis of the data from the field test (N = 99), the distribution of the responses to the items indicated that all items except from two showed sufficient variation across the response categories (i.e., less than 95% of responses on a single category). These two items were discarded. The PCA on the self-reported items measuring health literacy in healthcare, after three iterations, resulted in a four-component solution explaining 59% of the common variance. The PCA on the self-reported items concerning disease prevention, after four iterations, resulted in a four-component solution explaining 64% of the common variance, and finally the PCA concerning health promotion, after four iterations, resulted in a four-component solution explaining 62% of the common variance. In conclusion, these results indicate that for each of the three domains (healthcare, disease prevention and health promotion) a four-component structure was found which reflected the four dimensions of accessing, understanding, appraising and applying health related information. Alpha Cronbach levels ranged between 0.51 and 0.91. Taking into account that Cronbach’s alpha is sensitive to a low number of items, these values suggest that the obtained scales are reasonably homogenous.
The expert consultations (N = 25) requiring second opinions from members of the health literacy advisory boards and other collaborative partners in the HLS-EU project resulted in a series of critical reflections about the content and format of the questionnaire. The main recommendations by the experts were to: (1) remain true to the aim of the questionnaire (i.e. measuring health literacy in the general population) by looking at people’s competencies to access, understand, appraise and apply information to take decisions in terms of disease management, risk management and health management; (2) keep the focus on people, patients and lay persons; the system efforts are not the main focus of this survey; (3) maintain a generic approach, recognizing that health literacy is content and context specific and that the survey will be applied across many countries and different cultural settings; (4) keep the design simple at all levels – from lay-out to content such as items and response categories; (5) ensure clear language and avoid “expert” terminology; (6) keep the questionnaire easy to administer.
Finalisation of the questionnaire
The final item selection process integrating the results of the focus groups, qualitative and quantitative analysis of the field test data, and expert consultations, yielded a pre-final version of the HLS-EU-Q which differed from the field tested version in several ways. While still closely related to the conceptual model and matrix, it was less repetitive; it included plain language and contained only self-reported items. The intended ‘objective’ items such as the knowledge questions were discarded due to lack of consistency across the eight countries. Literacy items related to e.g. word recognition and text comprehension were also not included. The core questionnaire contained 47 health literacy related items, covering the 12 sub-scales of the HLS-EU matrix with 3-5 items in each scale. The number of items in each scale was a result of the consensus-based item selection process within the sub-group. The exact wording of each item is presented in the Additional file 1.
Examination for use of plain language
The examination of the questionnaire by the National Adult Literacy Agency in Ireland (NALA) resulted in a number of smaller changes to accomplish more simple language in the final version of the questionnaire e.g. “…judge the reliability of illness-related information presented in the media?” was changed to “…judge if the information about illness in the media is reliable”. The review by NALA ensured that the items were easy to read and understand which in turn facilitated that data collection would run more smoothly and quickly, than was experienced in the pre-test and field-test.
The translation of the final version of the questionnaire produced identical versions of the questionnaire in 7 languages ((Bulgarian, Dutch, English, German, Greek, Polish and Spanish). In addition, the English version was adapted in its original version to be applied in Ireland, and the German version was adapted for use in Germany and Austria by the translation panel and the experts involved in the respective countries to ensure its cultural applicability e.g. in terms of translations of specific words that differed or system-related items.
An overview of the characteristics of the design and development process for the HLS-EU-Q is presented in Table 4.
Health literacy has gained increasing attention in research and practice over the past decades . With this increasing interest, there has been a growing demand for tools to measure health literacy . However, existing tools to measure health literacy focus mostly on screening for functional health literacy within clinical settings by e.g. testing word recognition or the understanding of food labels [11, 31]. As HLS-EU-Q was developed for measuring the health literacy of general populations and not of specific patient groups, it does not follow a narrow clinical or medical focus, but captures a broad public health perspective. Grounded in public health, the HLS-EU-Q measures health literacy in terms of three domains where people’s health is of concern and is expressed in terms of accessing, understanding, appraising and applying information to manage disease, manage risks and manage health. In other words as a patient being ill navigating the healthcare system, as a person at risk encountering information on disease prevention, and as a citizen striving for optimal health encountering health promotion offers in the community, the work place, the educational system and the market place as described in detail by Sorensen et al.  As illustrated in the HLS-EU model the assumption is that health literacy is the outcome of informal and formal learning and health education. This means health literacy refers to an evolving set of competencies that do not remain static over time. In this definition, health is linked to quality of life and can be regarded as a means to an end rather than a fixed state, to which a person should aspire: “(Health is) the extent to which an individual or group is able on the one hand, to realize aspirations and satisfy needs; and, on the other hand, to change or cope with the environment. Health is, therefore, seen as a resource for everyday life, not an object of living; it is a positive concept emphasizing social and personal resources, as well as physical capacities” .
Bridging measurement gaps
The HLS-EU-Q addresses many of the shortcomings of existing tools brought forward by e.g. Jordan et al.  and Pleasant et al. . It is explicit build on a definition and a conceptual framework of health literacy. It is multi-dimensional in content and distinguishes health literacy from communication. It treats health literacy as a ‘latent construct’ and follows a principle of compatibility, since different scales can be used for different contexts. It permits comparison in different populations and makes reference to public health rather than just clinical use. The development process of the measurement instrument has been based on a consensus approach involving nine research teams from eight countries as well as a large number of collaborating partners from Europe and abroad, all with a variety of professional educational backgrounds and experiences. Though the cross-national and cross-disciplinary group have been an advantage to ensure a wide range of perspectives in every step of the development process, the multifaceted group also remained a challenge throughout the process since the overall initial demands from all the investigators involved to content and form turned out to be too comprehensive and lengthy and therefore not feasible to administer. By respecting the feedback from users, participants and external stakeholders crucial decisions were taken during the development process, which considerably impacted the design of the final version. Eventually, the new orientation and the accompanying process of scrutinizing every item for its suitability according to agreed quality criteria paved the way for a stringent tool, matching the aim of the study while still staying true to the original ideas presented in the definition, concept and questionnaire matrix of the HLS-EU Consortium. The variation in interview time depended on the responses made by the participants as some were fast and could answer quickly, while others took longer because they found it difficult to deal and manage their health and had to reflect a longer time. The choice of a 4 point-Likert scale differentiated the responses, and for those who wished to skip an answer or could not answer items, the interviewer had the option of the “I don’t know” category. Due to the fine-tuning of the questionnaire the administration time was reduced to an average of 20-30 minutes including an additional section on personal back-ground variables referring to e.g. demographic and socio-economic factors that was also included in the test phases.
Several limitations should be noted in the design and development of the HLS-EU-Q. The Delphi generated items mainly in the domains of healthcare and disease prevention and less in the domain of health promotion. Furthermore, the Delphi resulted in an excess of items, hence the process resulted in partial consensus where the coordination team adjusted the questionnaire by e.g. selecting the most preferred items. In addition, there was limited geographical scope applied in the testing phase. Focus groups were only carried out in three countries and the field test applying face-to-face interviews only in two. Ideally, it would have been better to include all countries, but this was not feasible within the financial constraints of the study. The data analysis regarding the field test revealed a variation of Chronbach’s alpa from 0.51 to 0.91, which warrants further research, since some of the values were considerably low. The overall results of the focus groups, the field test and the expert consultations generated a change in the item design, hence it is suggested that the aspect of Chronbach Alpha is taken specifically into consideration when validating the questionnaire in its application e.g. in the European Health Literacy Survey. Although, the expert consultations involved experts from Ireland, Israel and Australia (see acknowledgement) adding to the multi-national perspectives already represented in the HLS-EU Consortium, an even wider representation may have been beneficial for the questionnaire’s applicability in different cultures. The continuous feedback throughout the development process on problems concerning expert biases in terms of difficult words and wordings emphasized how difficult it was for the wider group of researchers to let go of expert language and underlying paradigms. In response, a plain language check was carried out by a designated Literacy Agency to ensure a final assessment of clear and transparent language in the questionnaire. The translations involved professional translators only in the final step after the plain language check. Ideally professional translations would have been carried out for the pre-test and field test as well. By illuminating the overall research process, as well as the detailed steps in the design and development process undertaken by the HLS-EU Consortium, the necessary transparency is offered for others to apply the HLS-EU-Q with a deeper understanding of its purpose, its capability and its limitations. Yet, it is also evident that further research is needed to enhance its quality and applicability in the future.
This paper has explained the design and development process of the European Health Literacy Questionnaire, the HLS-EU-Q. By illuminating the detailed steps in the design and development process of the HLS-EU-Q, a deeper understanding of its purpose, its capability and its limitations has been provided for others using the tool. Bearing these insights in mind it is the vision that HLS-EU-Q with its conceptual-based, multi-facetted attributes will be validated in more countries to enhance the understanding of health literacy at population level.
aWhile in most participating countries, samples representative for the whole country were sampled, however for feasibility reasons in Germany only the biggest state North Rhine Westphalia took part in the study!
Nutbeam D: Advancing health literacy: a global challenge for the 21st century. Health Promot Int. 2000, 15: 183-184.
Nutbeam D: The evolving concept of health literacy. Soc Sci Med (1982). 2008, 67: 2072-2078.
Paasche-Orlow MK, Parker RM, Gazmararian JA, Nielsen-Bohlman LT, Rudd RR: The prevalence of limited health literacy. J Gen Intern Med. 2005, 20: 175-184.
Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K: Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011, 155: 97-107.
Brach C, Dreyer BP, Schyve P, Hernandez LM, Baur C, Lemerise AJ, Parker RM: Attributes of a Health Literate Organization. Inst Med. 2012, http://www.iom.edu/~/media/Files/Perspectives-Files/2012/Discussion-Papers/BPH_HLit_Attributes.pdf,
Pleasant A, McKinney J, Rikard RV: Health literacy measurement: a proposed research agenda. J Health Commun. 2011, 16 (Suppl 3): 11-21.
Davis TC, Long S, Jackson R, Mayeaux E, George R, Murphy P, Crouch M: Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med. 1993, 25: 391-395.
Davis TC, Gazmararian JA, Kennen EM: Approaches to improving health literacy: lessons from the field. J Health Commun. 2006, 11: 551-554.
Hanson-Divers EC: Developing a medical achievement reading test to evaluate patient literacy skills: a preliminary study. J Health Care Poor Underserved. 1997, 8: 56-69.
Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss J: Development of a brief test to measure functional health literacy. Patient Educ Couns. 1999, 38: 33-42.
Bass PF, Wilson JF, Griffith CH: A shortened instrument for literacy screening. J Gen Intern Med. 2003, 18: 1036-1038.
Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, Mockbee J, Hale FA: Quick Assessment of Literacy in Primary Care: the Newest Vital Sign. Ann Fam Med. 2005, 3: 514-522.
Kutner M, Greenberg E, Jin Y, Paulsen C: The health literacy of America’s adults: results from the 2003 National assessment of adult literacy. Education. 2006, 6: 1-59.
Steckelberg A, Hülfenhaus C, Kasper J, Rost J, Mühlhauser I: How to measure critical health competences: development and validation of the Critical Health Competence Test (CHC Test). Adv Health Sci Educ Theory Pract. 2009, 14: 11-22.
Wang J, Schmid MR, Thombs BD: The swiss health literacy survey: development and psychometric properties of a multidimensional instrument to assess competencies for health. Health Expect. 2012, http://onlinelibrary.wiley.com/doi/10.1111/j.1369-7625.2012.00766.x/pdf,
Jordan JE, Buchbinder R, Briggs AM, Elsworth GR, Busija L, Batterham R, Osborne RH: The health literacy management scale (HeLMS): a measure of an individual’s capacity to seek, understand and use health information within the healthcare setting. Patient Educ Couns. 2013, 91: 228-235.
Osborne RH, Batterham RW, Elsworth GR, Hawkins M, Buchbinder R: The grounded psychometric development and initial validation of the Health Literacy Questionnaire (HLQ). BMC Public Health. 2013, 13: 658-
Haun J, Luther S, Dodd V, Donaldson P: Measurement variation across health literacy assessments: implications for assessment selection in research and practice. J Health Commun. 2012, 17 (Suppl 3): 141-159.
McCormack L, Valerio M, Haun J, Sorensen K: Panel D: Improving coherence between health literacy frameworks and measures. 2012, Bethesda, Washington: Annual Health Literacy Research Conference, Bethesda
Jordan JE, Osborne RH, Buchbinder R: Critical appraisal of health literacy indices revealed variable underlying constructs, narrow content and psychometric weaknesses. J Clin Epidemiol. 2011, 64: 366-379.
Griffin JM, Partin MR, Noorbaloochi S, Grill JP, Saha S, Snyder A, Nugent S, Baines Simon A, Gralnek I, Provenzale D, Van Ryn M: Variation in estimates of limited health literacy by assessment instruments and non-response bias. J Gen Intern Med. 2010, 25: 675-681.
HLS-EU Consortium: The European Health Literacy Project Grant Agreement 2007-113. 2008, :
Sorensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, Brand H: Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012, 12: 80-
Hasson F, Keeney S, McKenna H: Research guidelines for the Delphi survey technique. J Adv Nurs. 2000, 32: 1008-1015.
Marshall M: Sampling for qualitative research. Fam Pract. 1996, 13: 522-525.
The European Health Literacy Project (HLS-EU): Grant agreement from the European Agency of Consumers 2007-113. 2013, http://www.health-literacy.eu,
Chew LD, Bradley KA, Boyko EJ: Brief questions to identify patients with inadequate health literacy. Fam Med. 2004, 36: 588-594.
HLS-EU Consortium: Comparative Report on Health Literacy in Eight EU Member States. 2009, Europe: The European Health Literacy Project
Parker RM, Ratzan SC: Health literacy: a second decade of distinction for Americans. J Health Commun. 2010, 15 (Suppl 2): 20-33.
Pleasant A, McKinney J: Coming to consensus on health literacy measurement: an online discussion and consensus-gauging process. Nurs Outlook. 2011, 59: 95-106.
Osborn CY, Weiss BD, Davis TC, Skripkauskas S, Rodrigue C, Bass PF, Wolf MS: Measuring adult literacy in health care: performance of the newest vital sign. Am J Health Behav. 2007, 31 (3): S36-S46.
World Health Organisation’s European Office: Statements from Global Conferences. 2009, OMS: Health Promotion, 1-42.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/13/948/prepub
The authors thank all partners in the HLS-EU Consortium for contributing to the development of the HLS-EU-Q. The HLS-EU Consortium includes: Maastricht University, the Netherlands: Helmut Brand, Stephan Van den Broucke, Kristine Sorensen; National School of Public Health, Greece: Demosthenes Agrafiodis, Elizabeth Ioannidis, Barbara Kondilis (Hellenic American University). University College Dublin, National University of Ireland: Gerardine Doyle, James Fullam; Kenneth Cafferkey; Ludwig Boltzmann Institute Health Promotion Research: Jürgen M. Pelikan, Florian Röthlin; The Cardinal Wyszyński Institute of Cardiology, Warsaw, Poland: Zofia Slonska; University of Murcia, Spain: Maria Falcon; Medical University, Sofia, Bulgaria: Kancho Tchamov and Alex Zhekov; National Institute of Public Health and the Environment, the Netherlands: Mariële Droomers, Jantine Schuit, Iris van der Heide and Ellen Uiters; and NRW Centre for Health, Germany: Monika Mensing and Gudula Ward. The authors also wish to acknowledge the contribution of all collaborating partners, especially the HLS-EU national advisory body on health literacy and Jennifer Lynch and her team at the National Adult Literacy Agency in Ireland; Diane Levin from Clalit Health Services in Israel; and Rachelle Buchbinder from Monash University in Australia to the development process of the HLS-EU-Q. The development of the HLS-EU-Q is a deliverable of the HLS-EU project, which is supported with grant 2007-113 from the European Commission’s Health Programme.
The authors declare not to have any competing interests.
Background: KS; SVdB; JMP; GD; ZS; BK; HB. Method: KS (Delphi, focus groups, field-test, expert consultations, translations); SVdB (Delphi; field test, expert consultations); JMP (Delphi, expert consultations, translations), JF (Delphi, focus groups, field test, expert consultations); GD (Delphi, focus groups, field test, expert consultations); ZS (Delphi, expert consultations, translations); BK (Delphi, focus groups, expert consultations, translations); RHO (expert consultations), HB (expert consultations). Results: KS; SVdB; JMP; JF; GD; ZS; BK; RHO; HB (collaborative development of the HLS-EU-Q during seminars and telephone conferences on the basis of outcomes of the design and development steps). Discussion: KS; JMP; GD; ZS; BK; RHO; HB. All authors read and approved the final manuscript.
Electronic supplementary material
About this article
Cite this article
Sørensen, K., Van den Broucke, S., Pelikan, J.M. et al. Measuring health literacy in populations: illuminating the design and development process of the European Health Literacy Survey Questionnaire (HLS-EU-Q). BMC Public Health 13, 948 (2013). https://doi.org/10.1186/1471-2458-13-948